ZFS Pool vanished after repacing failed USB stick

[Cyberhide]

Hello everyone,

I recently had a USB stick fail (running 9.1.1). It was corrupted beyond usability (couldnt mount it to save anything). I replaced the USB stick with a new one and loaded 9.3 on it figuring I would upgrade at the same time. After booting up, it was unable to find my ZFS pool "Volume1".

I can see the disks:

Code:

[root@freenas] /tmp# camcontrol devlist
<ST4000VN000-1H4168 SC46>          at scbus0 target 0 lun 0 (ada0,pass0)
<ST4000VN000-1H4168 SC46>          at scbus1 target 0 lun 0 (ada1,pass1)
<HUA723030ALA640 MKAOA870>         at scbus2 target 0 lun 0 (ada2,pass2)
<Hitachi HDS723030ALA640 MKAOA5C0>  at scbus3 target 0 lun 0 (ada3,pass3)

I can see the partitions:

Code:

[root@freenas] /tmp# gpart show -l
=>        34  7814037101  ada0  GPT  (3.7T)
          34          94        - free -  (47k)
         128     4194304     1  (null)  (2.0G)
     4194432  7809842696     2  (null)  (3.7T)
  7814037128           7        - free -  (3.5k)

=>        34  7814037101  ada1  GPT  (3.7T)
          34          94        - free -  (47k)
         128     4194304     1  (null)  (2.0G)
     4194432  7809842696     2  (null)  (3.7T)
  7814037128           7        - free -  (3.5k)

=>        34  5860533101  ada2  GPT  (2.7T)
          34          94        - free -  (47k)
         128     4194304     1  (null)  (2.0G)
     4194432  5856338703     2  (null)  (2.7T)

=>        34  5860533101  ada3  GPT  (2.7T)
          34          94        - free -  (47k)
         128     4194304     1  (null)  (2.0G)
     4194432  5856338703     2  (null)  (2.7T)

But when I check zdb, I cant find any labels on any of the drives:

Code:

[root@freenas] /tmp# zdb -l /dev/ada0p2
--------------------------------------------
LABEL 0
--------------------------------------------
failed to unpack label 0
--------------------------------------------
LABEL 1
--------------------------------------------
failed to unpack label 1
--------------------------------------------
LABEL 2
--------------------------------------------
failed to unpack label 2
--------------------------------------------
LABEL 3
--------------------------------------------
failed to unpack label 3

No amount of zpool import combinations work:

Code:

[root@freenas] /tmp# zpool import -a
[root@freenas] /tmp# zpool import Volume1
cannot import 'Volume1': no such pool available
[root@freenas] /tmp# zpool status -v
no pools available

I also do not see any warnings/errors about zfs, geom, or the drives themselves:

Code:

[root@freenas] /tmp# dmesg | grep -i zfs
ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
            to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)

[root@freenas] /tmp# dmesg | grep -i geom
GEOM_RAID5: Module loaded, version 1.1.20110927.40 (rev 00ce00e5abb4)
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ada2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software

[root@freenas] /tmp# dmesg | grep -i ada
ada0 at ata2 bus 0 scbus0 target 0 lun 0
ada0: <ST4000VN000-1H4168 SC46> ATA-9 SATA 3.x device
ada0: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada0: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada0: Previously was known as ad4
ada1 at ata3 bus 0 scbus1 target 0 lun 0
ada1: <ST4000VN000-1H4168 SC46> ATA-9 SATA 3.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: 3815447MB (7814037168 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad6
ada2 at ata4 bus 0 scbus2 target 0 lun 0
ada2: <HUA723030ALA640 MKAOA870> ATA-8 SATA 2.x device
ada2: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada2: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada2: Previously was known as ad8
ada3 at ata5 bus 0 scbus3 target 0 lun 0
ada3: <Hitachi HDS723030ALA640 MKAOA5C0> ATA-8 SATA 3.x device
ada3: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada3: 2861588MB (5860533168 512 byte sectors: 16H 63S/T 16383C)
ada3: Previously was known as ad10

Infact, the swap partitions on the drives seem to be functioning fine:

Code:

[root@freenas] /tmp# swapctl -l
Device:       1024-blocks     Used:
/dev/ada0p1.eli   2097152         0
/dev/ada1p1.eli   2097152         0
/dev/ada2p1.eli   2097152         0
/dev/ada3p1.eli   2097152         0

I'm stumped on this, and I rather perplexed why the labels would just vanish. I know I lost the zdb cache from the USB stick, but loosing my zpool because I replaced a USB stick is beyond me.

As for the "restore from backup"... This is a home NAS and I have the critical data from it backed up. But due to its size (4 x 3tb), I dont have it all backed up. I would really like to recover some of the data, if possible.

Any help the community can give me on this would be greatly appreciated.

Thanks!

-Chris

 

[cyberjock]

I don't know what the heck happened, but can you post your hardware and the debug file from your server?

 

[Cyberhide]

Hello cyberjock,

I've attached the 300kb debug file from the freenas-debug command "options -acghlTnsytz"

The server is a white box pc with ECS H61H2-I3 Motherboard, Intel Celeron G1610, 4GB RAM (2x 2GB Sticks), 2x Seagate 4TB NAS HDD, and 2x Hitachi 3TB UltraStar drives. I was half way through a slow expansion of my drives. I was swapping out my aging 3tb Hitachis with 4tb Seagates one by one every month or two (as I could afford it). I was doing them cold with the server powered down. I had no issues doing the replacements, and it went smooth for the first 2.

 

[SweetAndLow]

Will since I'm the first one here I'll say it, you don't have hardware that meets the minimum spec. Specifically your memory, you should have 8GB minimum. What I'm not sure of is that the problems you are seeing can be proved to be related to your low memory. Strange things can happen when not using enough money.

 

[Cyberhide]

I read through the docs for 9.1.1 pretty well when I installed my current system, and it didn't saying anything scarey about using 4GB of RAM. Here is the part about RAM:

It is possible to use ZFS on systems with less than 8 GB of RAM. However, FreeNAS® as distributed is configured to be suitable for systems meeting the sizing recommendations above. If you wish to use ZFS on a smaller memory system, some tuning will be necessary, and performance will be (likely substantially) reduced. ZFS will automatically disable pre-fetching (caching) on systems where it is not able to use at least 4 GB of memory just for ZFS cache and data structures. This post describes many of the relevant tunables.

So I'm doubtful, and really hoping that was not the cause of my situation.

 

[cyberjock]

I asked for the debug because I was expecting you to say you didn't have the minimum RAM. You don't, so I think you've lost the battle. If you are willing to do paid support you can call iXsystems and get a quote for a 3 hour block of support. It won't be cheap, and I'd be willing to bet since you have a backup you won't want to spend the money.

9.1.1 didn't have the scary warning because it was still being debated whether to put it in the manual or not. We did, just after 9.2.0 came out IIRC.

If you plan to use ZFS deduplication, a general rule of thumb is 5 GB RAM per TB of storage to be deduplicated.

If you only plan to use UFS, you may be able to get by with as little as 2GB of RAM.

If you don't have at least 8GB of RAM with ZFS or 2GB of RAM with UFS, you should consider getting more powerful hardware before using FreeNAS® to store your data. Otherwise, data loss may result.

^^^ from the 9.2.1 manual I had handy.

 

[Robert Smith]

What is this?

Code:

ada0: Previously was known as ad4
ada1: Previously was known as ad6
ada2: Previously was known as ad8
ada3: Previously was known as ad10

Check this thread out: http://readlist.com/lists/freebsd.org/freebsd-current/17/85399.html

 

[Cyberhide]

I modified the recommended changes to the boot loader.conf file and rebooted. The "Previously was known as" warnings are now gone (along with the symlinks in /dev), but the zfs labels are still missing from the zfs partitions.

The sad thing is I had a new motherboard/cpu/memory sitting next to this system I was going to upgrade to, but the USB stick failed before I got it upgraded. Great!

Unless someone can offer any additional help, I am going to consider this a lost cause. I guess I can chalk this up to a lesson learned.

 

[Robert Smith]

Code:

zdb -l /dev/ada0p2

Are you sure you have the correct device here? Have you tired

zdb -l /dev/ada0
zdb -l /dev/ada0p1
zdb -l /dev/ada0p3
zdb -l /dev/ad4
zdb -l /dev/ad4p1
zdb -l /dev/ad4p2
zdb -l /dev/ad4p3
maybe others (and try other disks too)...

 

[titan_rw]

The fact that zdb doesn't see any labels is the most concerning. If you manage to figure out what happened, or a get any further, let us know.

 

[cyberjock]

ad are devices that haven't been assigned ada or da device names. You can't access ad devices as a regular course of business. :p

If zdb -l (anything) would have worked, there would have been an output of some kind when he did "zpool import". But he didn't, which is the tell-tale that "you dun f*cked somethin' up son!"

Oh, and for the record, I've always been told that ECS was such low quality that ECS was alleged to be an acronym for "Every Customer is a Sucker". No joke, they had quality control issues so bad that it was laughable. They'd have the CMOS reset jumper installed on motherboard when shipped out of their warehouses.

 

[fta]

I know this is old and you've probably already wiped the disks and started over, but your pool was fine. Low memory had nothing to do with this. I'll give the solution for future googlers:

Code:

GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software

Your disks are encrypted. Before you can list the labels or import the pool, you need to do a

Code:

geli attach [-dprv] [-j passfile] [-k keyfile] prov

with your passphrase and/or keyfile. See the geli man page for full details on the attach command.

 

[SweetAndLow]

I know this is old and you've probably already wiped the disks and started over, but your pool was fine. Low memory had nothing to do with this. I'll give the solution for future googlers:

Code:

GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:     Crypto: software

Your disks are encrypted. Before you can list the labels or import the pool, you need to do a

Code:

geli attach [-dprv] [-j passfile] [-k keyfile] prov

with your passphrase and/or keyfile. See the geli man page for full details on the attach command.

I'm pretty sure this isn't the case. That log message is referring to the swap partition that is encrypted. If the main pool was encrypted they would have gotten a message alerting to that.

 

[fta]

Doh! You are correct, sir. That's p1, not p2.