ada1: Previously was known as ad6, a novel by FreeNAS-9.3-STABLE-201509022158

[Wade Blackwell]

Good morning all,
I woke up yesterday morning to a down storage device and thus a down Vmware environment. After a short time I realized the NFS volume serving my environment was no longer available. After combing through dmesg and some of the logs I did find what appears to be the root of the issue;

"Jan 17 09:13:52 cre-freenas-01 ada0: Previously was known as ad4"
"Jan 17 09:13:52 cre-freenas-01 ada1: Previously was known as ad6"

I did find some links referencing this but none of them seemed to indicate why the device names would change and they all just wanted to blow the drives away and start over. I need to keep the data on these devices. Upon more looking in the command history I found this (which I did not execute);

20 17:59 dd if=/dev/zero of=/dev/ada0 bs=1m count=1
21 17:59 dd if=/dev/zero of=/dev/ada1 bs=1m count=1

31 17:40 dd if=/dev/zero of=/dev/ada4 bs=512 count=1 && dd if=/dev/zero of=/dev/ada5 bs=512 count=1
32 17:41 dd if=/dev/zero of=/dev/ada0

What is that about?

So I'd like to know if my raid1 volume is recoverable and I'd like to know what caused FreeNas to freak out, rename the devices causing the zpool/volume/NFS share to fail and why on earth would the device issue a dd to destroy the volumes. Debug from the webui and my notes from the last 24 hours attached. the device boots from USB so the system is still functional. Thanks all, greetings from the Central Coast of California.

 

[DrKK]

Could we have a clear list of your hardware? CPU, RAM, Mobo, types of drives, whether or not an HBA is in play, and if so which one, etc.

 

[Wade Blackwell]

Of course, I assumed it was in the debug file;

FreeBSD 9.3-RELEASE-p25 #0
CPU: Intel(R) Xeon(R) CPU 5130 @ 2.00GHz (1995.04-MHz K8-class CPU)
real memory = 4831838208 (4608 MB)
avail memory = 4092051456 (3902 MB)
ada0 at ata2 bus 0 scbus2 target 0 lun 0
ada0: <ST4000VN000-1H4168 SC46> ATA-9 SATA 3.x device
ada0: Serial Number S301L3TB
ada0: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada0: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ata3 bus 0 scbus3 target 0 lun 0
ada1: <ST4000VN000-1H4168 SC46> ATA-9 SATA 3.x device
ada1: Serial Number Z3042LMJ
ada1: 150.000MB/s transfers (SATA, UDMA5, PIO 8192bytes)
ada1: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6

Base Board Information
Manufacturer: Dell Inc.
Product Name: 0TW856
Version: A01
Serial Number: ..CN137407A700XL.
Asset Tag: Not Specified

 

[Wade Blackwell]

There is a raid/storage controller,
I used the on-board controller as the raid card could not recognize the full drive size. Looking through the "lspci" below is all I see for storage controllers;

05:08.0 SCSI storage controller: LSI Logic / Symbios Logic SAS1068 PCI-X Fusion-MPT SAS (rev 01)
I'm not sure if that's what the drives are connected to.

 

[DrKK]

Wow. That's older equipment for sure; the CPU's that fit that equipment have been end-of-lifed for years. If I'm looking at the right materials, that server *CAPS OUT* at 8GB of ECC RAM.

Maybe someone with more familiarity than I have (which is zero) with server hardware from this generation can give you some insights. The 1068 chipset for the HBA is not exactly smooth sailing either. For example: https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/

My guess is that your equipment is a bit too old to be profitably put into service as a modern FreeNAS. Even as you solve problem X, problem Y will crop up due to age, or end-of-life. Maybe someone will have useful ideas, though.

 

[Wade Blackwell]

Yes agreed,
The back story is this was a temp location while I did some heavy maintenance on my primary storage device. I didn't get the data off quick enough. At this point I just want to see if I can recover the data and put this box back to bed. Thanks for the responses, yeah I think you're right, 8G is maxed out on mem ;-)

 

[DrKK]

I see where you've stated you didn't use that 1068 HBA, but still...

 

[DrKK]

Good luck. Maybe someone can give you some idea of the probability of a successful recovery, and give some ideas for how to proceed. @cyberjock?

 

[Wade Blackwell]

I also found the link below which isn't flattering but speaks to some of the hate I'm currently experiencing.

https://lists.freebsd.org/pipermail/freebsd-hackers/2012-September/040480.html

 

[Robert Trevellyan]

Upon more looking in the command history I found this (which I did not execute);

If you didn't, then who did? Potential unauthorized access to the system/malicious tampering? Rebooting is known to change device numbers, e.g. ada0 to ada1, but from adaX to adY is completely unexpected. You would have to connect a drive to a different controller, or flash a controller with different firmware.

 

[Wade Blackwell]

Well now that's the question. I perused "history" and saw it there. I was baffled to be honest. The network was fairly locked down so if there was an honest to goodness rooted box or the storage device itself was rooted I'd be surprised. So there's not a situation, ever, where. Freenas would execute that command without human intervention?

 

[Robert Trevellyan]

So there's not a situation, ever, where. Freenas would execute that command without human intervention?

I can only imagine FreeNAS doing anything like that if told to wipe a drive or mark it as new in the GUI. I don't know if either of those would show up as dd commands in a log. Regardless, it's not going to initiate this on its own.

 

[Wade Blackwell]

Thanks everyone for the responses,
So with the below information how can I tell if I'm going to be able to recover the volume;

[root@cre-freenas-01] ~# gmirror status
Name Status Components
mirror/Data4t COMPLETE ada0 (ACTIVE)
ada1 (ACTIVE)

My gmirror inexperience is showing here but what's this command looking for to attempt a rebuild?

[root@cre-freenas-01] ~# gmirror rebuild mirror/Data4t
gmirror: Too few arguments.
[root@cre-freenas-01] ~# gmirror rebuild
gmirror: Too few arguments.
[root@cre-freenas-01] ~# gmirror rebuild mirror/Data4t ?
gmirror: No match.
[root@cre-freenas-01] ~# gmirror rebuild mirror/Data4t ada1
gmirror: No such device: mirror/Data4t.
[root@cre-freenas-01] ~# gmirror rebuild mirror/Data4t
gmirror: Too few arguments.
[root@cre-freenas-01] ~# gmirror rebuild Data4t
gmirror: Too few arguments.
[root@cre-freenas-01] ~# gmirror rebuild mirror/Data4t
gmirror: Too few arguments.

How can I determine if this is a lost cause. I'd like to bring the volume up just long enough to get the data off of it.

 

[Wade Blackwell]

Well this looks promising;

[root@cre-freenas-01] ~# gmirror rebuild Data4t ada1
[root@cre-freenas-01] ~# gmirror rebuild Data4t ada0
gmirror: Provider ada0 is the last active provider in Data4t.
[root@cre-freenas-01] ~# gmirror status
Name Status Components
mirror/Data4t DEGRADED ada0 (ACTIVE)
ada1 (SYNCHRONIZING, 0%)

 

[Robert Trevellyan]

Why are you trying to use gmirror to recover this?

 

[Wade Blackwell]

Is there another way? That's the feedback I'm looking for, how do I recover this?

 

[danb35]

I perused "history" and saw it there.

You mean the output of the "history" command? That shows commands that were typed at the shell prompt. Someone, while logged into your machine as root, typed and ran those commands at some time. They weren't part of a script; they were actually entered at the shell prompt. The question is when--"history" shows the time, but I don't know if (or, more likely, how) it can be made to give the date.

So there's not a situation, ever, where. Freenas would execute that command without human intervention?

I can't address that one way or the other (though I strongly doubt it), but that's not what happened with you.

I'm also pretty sure that the "ada1: Previously was known as ad6" is a red herring. I see that routinely on boot of my machine, and it doesn't seem to be a problem.

 

[Wade Blackwell]

Well that's good news,
What I'm left with is the following and I'm not certain why the mountpoint is not working;

[root@cre-freenas-01] ~# mount /dev/mirror/Data4t /mnt/temp
mount: /dev/mirror/Data4t: Invalid argument

 

[DrKK]

There is a point where a post jumps the shark.

 

[Wade Blackwell]

Sorry that comment is lost on me. So back to the original question of can this volume/mirror be recovered and/or mounted? How can I validate it one way or the other? I have tried the following;

I have a device for the mirror
[root@cre-freenas-01] ~# ll /dev/mirror/Data4t
crw-r----- 1 root operator 0x62 Jan 19 19:24 /dev/mirror/Data4t

I don't seem to be able to mount said device
[root@cre-freenas-01] ~# mount /dev/mirror/Data4t /mnt/temp
mount: /dev/mirror/Data4t: Invalid argument

Some not super helpful messages in /var/log/debug.log about the failing zvol
Jan 22 10:38:53 cre-freenas-01 alert.py: [middleware.notifier:212] Popen()ing: zpool list -H -o health data0
Jan 22 10:38:53 cre-freenas-01 alert.py: [middleware.notifier:212] Popen()ing: /sbin/zpool status -x data0

Dmesg from last boot shows;
GEOM_MIRROR: Cancelling unmapped because of ada0.
GEOM_MIRROR: Cancelling unmapped because of ada1.
GEOM_MIRROR: Device mirror/Data4t launched (2/2).
GEOM: mirror/Data4t: corrupt or invalid GPT detected.
GEOM: mirror/Data4t: GPT rejected -- may not be recoverable.

Storage rookie comments welcome as long as they come with some helpful information on weather the data on this volume is recoverable. Thanks all.