Failing pool with .system dataset in it - how to proceed?

[Hazar]

I moved my .system dataset (and swapfile) to an internal USB drive (named usbdrive) 7 months ago. Today, my scheduled scrub alerted me that I have a checksum error on that USB drive.

On my first zpool status -v usbdrive it only mentioned checksum errors and no data was corrupted. However, after I issued a zpool scrub usbdrive it says I have a corrupted swapfile. While I'm using 0 bytes of swap, I issued a swapoff (because if it ever swaps to it and tries to read back, probably the next thing I see is a kernel panic). Further iterations of scrub resulted in differing amounts of cksum errors; which clearly shows the USB drive is on its highway to hell. Now it shows that I have an unrecoverable error at usbdrive:<0x8>. I stopped at this point.

The system is still working but as you can see it is a ticking bomb. And regular stream of writing to .system dataset is not helping either.

How can I proceed to change this USB drive? Is there any instructions or guidance to replace drives that contains .system dataset?

I have a FreeNAS config backup but I think .system dataset contains much more of than. Do I need to try mounting my system dataset somewhere and back it up?

Code:

# zfs list -r usbdrive
NAME														USED  AVAIL  REFER  MOUNTPOINT
usbdrive													870M  2.74G  36.5M  /mnt/usbdrive
usbdrive/.system											421M  2.74G   402M  legacy
usbdrive/.system/configs-810048d7feed436fae88f4409435135f  15.1M  2.74G  15.1M  legacy
usbdrive/.system/cores									  722K  2.74G   722K  legacy
usbdrive/.system/rrd-810048d7feed436fae88f4409435135f		19K  2.74G	19K  legacy
usbdrive/.system/samba4									  91K  2.74G	91K  legacy
usbdrive/.system/syslog-810048d7feed436fae88f4409435135f   2.62M  2.74G  2.62M  legacy
usbdrive/jails											  411M  2.74G	21K  /mnt/usbdrive/jails
usbdrive/jails/.warden-template-pluginjail				  411M  2.74G   409M  /mnt/usbdrive/jails/.warden-template-pluginjail

Note: I don't store my jails in this drive.

System:
FreeNAS-9.10.2-U3 (e1497f269)
HP Microserver G8 with 12G RAM

Thanks in advance.

 

[m0nkey_]

Wow. You put the .system dataset on a USB drive?!

Move it immediately. Details here: http://doc.freenas.org/11/system.html#system-dataset

 

[DrKK]

Wow. You put the .system dataset on a USB drive?!

How about: "Wow. You put any data whatsoever on a USB drive in a ZFS system?" :)

I am not sure what his config is. Apparently, he has a boot drive, and then, it looks like from what he's said, some other "jail" drive, and then, he's got some kind of data drive? drives? On which he put his .system dataset? Quite a strange situation sir.

 

[DrKK]

How about: "Wow. You put any data whatsoever on a USB drive in a ZFS system?" :)

I am not sure what his config is. Apparently, he has a boot drive, and then, it looks like from what he's said, some other "jail" drive, and then, he's got some kind of data drive? drives? On which he put his .system dataset? Quite a strange situation sir.

But then again, he says "I don't store my jails on the usb drive", and there is a "usbdrive/jails" dataset. So I have no idea what's going on.

 

[Hazar]

Hi, thanks for the replies.

The idea behind "system dataset on usb drive" was redirecting the small writes elsewhere to enabling spindown to my data drives. Appearantly while my rationale was valid, it was a bad idea and threated my system stability just in 7 months. Lesson learned.

Replacing that USB drive with a small SSD was always in my list, but I never get to it. Now I'm ordering one.

I moved my system dataset to my main pool using GUI. Is it safe to remove my usbdrive (failed drive) now? I'll deattach the volume using GUI and physically remove the disk afterwards, is that correct?

ps: My storage config is pretty simple:

Code:

~# zpool list
NAME		   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
freenas-boot  7.38G  1.30G  6.07G		 -	  -	17%  1.00x  ONLINE  -
tank		  14.5T  7.07T  7.43T		 -	11%	48%  1.00x  ONLINE  /mnt
usbdrive	  3.72G   450M  3.28G		 -	30%	11%  1.00x  ONLINE  /mnt

Code:

~# camcontrol devlist
<ST4000DM000-1F2168 CC54>		  at scbus0 target 0 lun 0 (pass0,ada0)
<ST4000DM000-1F2168 CC54>		  at scbus1 target 0 lun 0 (pass1,ada1)
<ST4000DM000-1F2168 CC54>		  at scbus2 target 0 lun 0 (pass2,ada2)
<WDC WD40PURX-64GVNY0 80.00A80>	at scbus3 target 0 lun 0 (pass3,ada3)
<SanDisk Cruzer 8.02>			  at scbus7 target 0 lun 0 (pass4,da0)
<HP iLO Internal SD-CARD 2.10>	 at scbus8 target 0 lun 0 (pass5,da1)

Sandisk = usbdrive
Internal SD = freenas-boot

My jails dataset resides on tank zpool. The jail dataset on failed drive is unused. FreeNAS don't destroy old datasets while moving it, and I don't bother to remove it; that's the reason why it was there.

Thanks in advance.