What mayhem can a faulty boot drive cause?

[Krowvin]

I would like to know if anyone has experienced odd behavior with their boot drives (USB).

To preface, I used to use the Sandisk Cruiser Fit 16GB x2. For the last year or so I've been using two Patriot 16GB flash drives. I kept my drives in a mirrored configuration thinking it would be great to have if one flash drive fails. However, when I set this up I had not read the post in the forums about mirroring drives to be a bit of a gimmick. Sure you have that additional drive for failure, but the hardware will not know when the drive has failed and will continue to try and boot from it. After I read that I decided I would wait until another flash drive failure to switch back to a single USB boot device. It's worth mentioning that I run weekly configuration backups (Might be a part of the reason for these drives failing on me, frequent read/writes over the year)

I have my jail drives in a mirrored configuration using two 250GB Samsung 850 EVO. I'm not 100% that this is the best solution, but I have placed my Plex cache on them as well as my temporary download for Usenet/Sabnzbd. A few weeks ago one of these drives failed, I did the unthinkable and cleaned the dust out of my system. I imagine I did a poor job of it and released the magic smoke. When I plugged the system back in two of pools showed failed drives. The RAIDZ2 pool had one failed disk, and my Mirrored jail pool showed another. Today I replaced the jail SSD to restore the mirror and started the resilver process.

The resilver seemed slow and the percentage did not move when viewing the status.

Code:

 zpool status 

After an hour or so my jails became unresponsive. I could not restart them and the plugins would not respond either. I decided to reboot as my zpool status was showing a write speed of 100-200KB/s (Something like that). When I restarted I got the error "slow spa_sync" as it tried to recover my snapshots. This is where the juice of my post comes in, I would have thought the drives would be the problem. That perhaps my other mirrored drive was failing, maybe I had a failed HBA? Did my Supermicro motherboard finally decide to crawl into a grave?

In a leap of despair, I decided to restore from a configuration I had emailed myself. Only this time, I did not create a USB mirror like I had before. The system seems to be 100% okay now and my disk resilver finished within 5 minutes. (Side note: I did have to make a correction on my iohyve after doing an upgrade through USB, supposedly the iohyve directory has changed and you need to create a system link to fix this? There's a bug report for it here. #228 )

Am I a tremendous fool for thinking I'm safe, did I get lucky enough that someone wants to come rub my head? Is there something I should do in order to prevent this in the future, or would anyone like to offer me some criticism on what to do better with my setup when I do decide to upgrade/replace?

Cheers, and thanks loads for reading!

Here are my system specs:

Show : System Specs

Version: FreeNAS-9.11-STABLE - (Current)
CPU: Intel Xeon E3-1230V3 Haswell 3.3GHz | BX80646E31230V3
RAM: Cruicial 32GB ( 4 x 8GB) SDRAM DDR3 ECC
Motherboard: Supermicro MBD-X10SL7 Intel C222
Case: Supermicro CSE-822T-400LPB
Storage:
Western Digital Red NAS
WD20EFRX 2TB x 6
RAIDZ2
Western Digital Red NAS
WD40EFRX 4TB x 8
RAIDZ1
Samsung 850 EVO - Jails
MZ-75E250B/AM 250GB x 2
Mirrored

 

[Arwen]

My personal opinion is that many USB flash drives are of poor quality, and before ZFS was used on them, (starting with FreeNAS 9.3 if I remember correctly), we simply saw un-explainable failures. Now we start to get a clue of how bad they are. That said, ZFS does have more of an impact on them than the old UFS RAM disk used for FreeNAS 9.2 and before.

Of course you have to test each and every one of the USB flash drives you buy. There are so many fake ones out there, that lie about their size. It's fraud, but they may have your money long before you find that out. There are even USB flash drives claiming to be 1TB in size! But cost 1/2 as much as a good quality 64GB one.

So I was glad my FreeNAS Mini came with a SATA DOM. More expensive that a 2.5" SATA SSD, but fits use cases for smaller form factor.

 

[tvsjr]

If the system is trying to access the boot drive and is repeatedly getting exceedingly long delays/timeouts/etc., it's entirely possible that it could totally tank performance on the system as a whole.

That's why I recommend small SSDs, and can't understand people who won't go that way. You're spending, typically, at least a grand or two, sometimes more, to build this system that's supposed to be a pinnacle of data security. Then, you run the whole thing off a $10 USB stick. That rates pretty high on my WTF-O-Meter.