raidz2-0 lost single device, can't import

[mclare]

I tried to offline a drive in my raidz2, system hung, now I can't remount. I'll admit to not knowing what state I was in when the problem occurred.

Here's my status pre-hang:

Code:

NAME                                                 STATE     READ WRITE CKSUM
        tank                                                 DEGRADED     0     0     0
          raidz2-0                                           ONLINE       0     0     0
            usb-JMicron_Generic_DISK03_0123456789ABCDEF-0:3  ONLINE       0     0     0
            usb-JMicron_Generic_DISK04_0123456789ABCDEF-0:4  ONLINE       0     0     0
            usb-JMicron_Generic_DISK02_0123456789ABCDEF-0:2  ONLINE       0     0     0
          usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1    DEGRADED     0     0     0  external device fault
        cache
          usb-Generic_Power_Saving_USB_000000000260-0:0      ONLINE       0     0     0

I'll admit I was trying to offline usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1 because I didn't know what it was doing there and was trying to mirror with another listed drive in a new pool. Paying for my ignorance.

Now when I try to import I get

Code:

 pool: tank
     id: 9999656061205060660
  state: UNAVAIL
status: One or more devices are faulted.
action: The pool cannot be imported due to damaged devices or data.
config:

        tank                                                 UNAVAIL  insufficient replicas
          raidz2-0                                           ONLINE
            usb-JMicron_Generic_DISK03_0123456789ABCDEF-0:3  ONLINE
            usb-JMicron_Generic_DISK04_0123456789ABCDEF-0:4  ONLINE
            usb-JMicron_Generic_DISK02_0123456789ABCDEF-0:2  ONLINE
          usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1    FAULTED  corrupted data

What to do next?

My thanks in advance. This is my journey away from an ironically not linux friendly old drobo into a new external USB enclosure.

 

[myoung]

Your raidz2 vdev was striped with usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1. When usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1 faulted, your pool was lost. It is unrecoverable if you cant get usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1 back online.

Do you have backups?

 

[PhiloEpisteme]

If I am reading the tabs fro up above correctly it looks like what you have is a pool, tank made of two vdevs. One of them is raidz2 the other is a single disk. That single disk vdev failed and because a pool is lost if any single vdev is lost your pool may be toast if you can't recover that single disk. Someone should certainly correct me if I am wrong.

usb-JMicron_Generic_DISK01_0123456789ABCDEF-0:1

Out of curiosity, what is the model etc of this drive?

EDIT: @myoung beat me to the punch.

 

[PhiloEpisteme]

My thanks in advance. This is my journey away from an ironically not linux friendly old drobo into a new external USB enclosure.

Oh, I missed this bit here. USB enclosures are bad news bears. Put simply, they add additional hardware between your drive controller and your drive in the form of the USB interface hardware. If anything there goes wrong and corrupts your data or drops your drive you may be in big trouble.

 

[mclare]

Thanks for the quick responses.
I was worried that was the case - and I exacerbated the problem in my attempt to do something about it.

I only have partial backups, and maybe the drobo the data came from would work with only one drive: in summary "arg".

Any limp back to live advice? A readonly mount would help me squirrel some data away before I rebuilt.

As it was asked, it's "ORICO Aluminium 5 Bay 3.5 inch USB3.0 SATA" - bunch of 1TB and 2TB WD drives. Plan was raidz2 would de-risk the USB situation.

 

[PhiloEpisteme]

As it was asked, it's "ORICO Aluminium 5 Bay 3.5 inch USB3.0 SATA" - bunch of 1TB and 2TB WD drives. Plan was raidz2 would de-risk the USB situation.

Based on that answer it sounds like your ORICO bay is doing some kind of RAID as well. This is highly recommended against.

Any limp back to live advice? A readonly mount would help me squirrel some data away before I rebuilt.

If you cannot recover those USB drives you won't be able to access your pool anymore. Because FreeNAS distributes data across vdevs you're not going to be able to get anything usable off of the single, surviving vdev without the other.

As you work past this snafu keep in mind a couple of quick rules of thumb that can go a long way to data integrity.

1. If you lose a single vdev and cannot recover it, your pool is lost.
   1. Redundancy within vdevs is therefore very important if the data is something you don't want to lose.
   2. Reliability within vdevs is also very important, hence the recommendation against USB drives and especially drive bays that do any kind of RAID array vdevs on your behalf.
2. FreeNAS uses ZFS, a software RAID solution. It likes direct access to raw disks. You don't want any RAID controllers between your drives and your system. Some RAID cards can be flashed to IT mode and therefore function as an HBA, these are fine.
3. FreeNAS on its own is not a substitute for a backup. The best backups live offsite and are themselves redundant in some way. Worse but still better than nothing are on-site backups but they are vulnerable to catastrophes such as fires. Worse are backups that live in the same FreeNAS box. If something catastrophic happened to that chassis you'd lose it all.

My advice would be to rebuild you FreeNAS system using bare disks, no RAID cards, USB drives, or RAID enclosures. If you have more disks than can fit your board's SATA or SAS ports pick up an HBA such as this one. If your case won't fit all of your drives consider either buying a bigger chassis that will or pick up an HBA such as this one which can connect to another chassis or enclosure housing a SAS expander.

Before you purchase any new parts you should go through the hardware recommendation guide. Then, feel free to post a very detailed post to Will it FreeNAS? Folks will be happy to help you refine your build. It is a good idea to detail your budget, intended use case etc. Once you've done that you'll have a good chance that your system will be able to provide the data integrity, performance, and scalability that fits your budget and risk tolerance.

 

[mclare]

Thanks for your help - I only wish I'd ask "why's that drive in that odd position" sooner. I'll be a little more aggressive in my recovery, then possibly rebuild keeping in mind your advice.

 

[myoung]

Make sure that you create your pools through the GUI in the future. You will see many errors and have to do much more work before you can stripe a raidz vdev with a single disk.