Insufficient replicas for degraded raidz pool

[liulan]

Hi all,

I am new to this forum and need some help on my encrypted pool. I have an encrypted 4 disks raidz1 pool. All disks are WD 4TB RE 7200rpm

System specs are:
Build FreeNAS-9.10.2-U6 (561f0d7a1)
Platform Intel(R) Core(TM) i3-4160 CPU @ 3.60GHz
Memory 5985MB non-ECC
Motherboard Asus B85M-G

All drives are connected directly to motherboard.

A couple of weeks ago. One of the 4 disks bad sectors and dropping off from the pool. I brought 4 new WD 10TB Red and replaced the disks one by one. i.e.
1) shutdown server, unplugged one of the 4TB disks and replaced with a WD 10TB Red
2) when resilvering completed, shutdown, unplugged second 4TB disk and replaced with a WD 10TB Red
3) when resilvering in 2) completed, shutdown, unplugged third 4TB disk and replaced with a WD 10TB Red
4) when resilvering in 3) completed, shutdown, unplugged fourth 4TB disk and replaced with a WD 10TB Red. However, before resilvering of the fourth disk could be completed, the third disk dropped off. I reconnected the third disk and the pool had the two disks resilvering. I think the good data in third disk was over written during the resilvering. Now the pool is unstable and has 1.9M errors. It keeps resilvering in every restart of the server.

I lost about 30% of the data in the pool. In the meantime, the bad sector disk was sent back for a warranty exchange.

Now I have 3 out of 4 encrypted disks of the original raidz pool (zvol0) and was trying to re-import them as a new pool or the original pool zvol0.

I managed to decrypt the disks using geli attach -p -k /data/geli/zvol0.key /dev/disks, where the disks are the devices of the encrypted 4TB disks.

When I try
zpool import -f zvol0

I get

Code:

pool: zvol0
id: 9275593362984646439
state: UNAVAIL
status: One or more devices are missing from the system.
action: The pool cannot be imported. Attach the missing devices and try again.
see: http://illumos.org/msg/ZFS-8000-3C
config:

zvol0										 UNAVAIL	 insufficient replicas
 raidz1-0									UNAVAIL	 insufficient replicas
   17979444800093306151		  UNAVAIL	 cannot open
   ada1p2.eli							   ONLINE	 
   10454733551089412823		  UNAVAIL	 cannot open
   463514057373486810			  UNAVAIL	 cannot open

I suspect that when I cycled the disks out one by one, each disk was detached and hence all disks have different volume information.

How can I fix this and re-import the degraded pool so that I can resilver with a new 4TB disk?

Many thanks

 

[kdragon75]

I'm sorry to hear about the issues with the new disk. Did you perform any burn in testing?
As each disk was detached in a different state, I don't think there is a supported way of rebuilding the array wit the old disks as they would need to be resilvered to ensure consistency with the rest of the array (which is not there). Let see if anyone else can chime in. Otherwise, its back o the backups.

 

[liulan]

Thanks, Kdragon75
I just swapped the old disks with brand new ones. Should have tested the stability of the system first. Now I learn the lesson.

Since the data is still there. What will happen if I create a new pool with the old 3 disks plus a new 4tb blank disk?

 

[kdragon75]

Since the data is still there. What will happen if I create a new pool with the old 3 disks plus a new 4tb blank disk?

without some data recovery magic, you would just be making a new pool.