Mirrored boot and recurrent checksum errors

[Clevinger]

I am running the latest FreeNAS 9.10 with boot mirrored to three MicroSD cards attached via Kingston USB adapters. Over the past two months, I have had three checksum errors (each time only a single checksum error) occur, replacing after each occurrence.

Kingston 8 GB -> replaced with Kingston 8 GB
Kingston 8 GB -> replaced with Samsung 32 GB -> replaced with Samsung 32 GB
Sandisk 16 GB, no errors to date

Now I normally wouldn't question a drive failure, but the sheer number of failures over such a short time frame makes me wonder if something else is awry. Is it possible/probable the problem exists on the USB interface and not the MicroSD cards? I could slowly replace the drives with USB mounted SSDs, but if it's a USB issue I would not want to waste the money.

Additional info: I have no spare SATA ports on the motherboard, but could add a SATA controller if necessary. Unfortunately, this is a non-ECC setup. Prior to the mirrored setup I was running FreeNAS off a single USB mounted flash drive which failed after several years.

Any insight would be appreciated.

 

[Mirfster]

I am running the latest FreeNAS 9.10 with boot mirrored to three MicroSD cards attached via Kingston USB adapters.

No offense, but that seems like a pretty janky setup. Toss in the Non-ECC and it is even worse...

If you are serious about using FreeNAS and your data, I would recommend that you give consideration to establishing a concrete system/foundation. Otherwise, everything else riding on that is compromised.

P.S. You never mentioned your current System Configuration, so best we can do is speculate.

 

[DrKK]

So let me answer the OP's original question.

I believe what you are seeing is legitimate. A lot of consumer-grade flash products (especially, in our experience, Kingston stuff of the past several years), throws errors fairly rapidly. The poorest quality stuff is usually the SD cards, and especially, microSD cards. In general, we don't particularly recommend SD cards for this purpose. ZFS of course makes every little tiny error show up immediately, whereas the typical consumer application will just silently have the occasional error and no one is harmed.

It is very believable---and I know this to be true because I have seen it on my own systems---for Kingston SD cards or USB thumb drives to suffer failures on the order of "weeks" when put into service on FreeNAS. Accordingly, I stopped using them about 2 years ago. I now use SanDisk, which has never once thrown a single error at least for me. I think if you replace with a good quality low profile USB drive, you should be fine. As always, I recommend the Sandisk Cruzer (or Cruzer Ultra) Fit.

 

[DrKK]

Here is a link to DrKK's officially recommended thumb drive for booting FreeNAS:

https://www.amazon.com/dp/B005FYNSZA/?tag=ozlp-20

 

[Clevinger]

Thank you for the replies, I didn't realize MicroSD cards were such low quality, well I knew Kingston would be, but I figured Samsung would be most robust. Next failure I'll replace with a SanDisk thumbdrive.

Not sure it's relevant now but system config:

1. FreeNAS version: FreeNAS-9.10-STABLE-201606270534 (dd17351)
2. Hardware:
   1. Motherboard (Model): Gigabyte GA-970A-D3 AM3+
   2. CPU (Model): AMD Athlon II X2 270
   3. RAM Size (in GB and model): 2 x Corsair XMS3 8GB DDR3 1333 MHz (PC3 10666) Desktop Memory (CMX8GX3M1A1333C9)
3. Hard Drives
   1. Data: RAID-Z2
      1. ada0: HGST Deskstar NAS 3 TB
      2. ada1: Seagate Barracuda 7200 3 TB
      3. ada2: WD Black 3 TB
      4. ada3: WD Red 3 TB
      5. ada4: Seagate Barracuda 7200 3 TB
      6. ada5: Seagate Barracuda 7200 3 TB
   2. Boot: Mirrored
      1. da0: Samsung 32 GB MicroSD
      2. da1: Kingston 8 GB MicroSD
      3. da2: Sandisk 16 GB MicroSD

Scrub: every 10 days
SMART short: every 2 days
SMART long: every 15 days
UPS backup: shutdown after 5 minutes on UPS power

Aside from the non-ECC RAM, I don't really think my config is that janky, obviously a HD-based boot would be preferable but it's not possible without a SATA addon card and I figured the mirror setup would avert catastrophic failures which it has thus far. When I first built the system I didn't understand that ECC was necessary. I am planning to switch to it once I have the money for a new motherboard/CPU/RAM combo. In the mean time and ongoing, I backup important data to external hard drives, so I have about 5x external redundancy, 2x of which is off-site. All HDs were initially Seagate, but I have been replacing with different models as they die to hopefully avoid clustered failures. I do realize WD Black is not ideal for a NAS, but it's spilled milk now, I bought it initially for the 5 year warranty as three Seagates died prior to 5 years.

 

[Mirfster]

Aside from the non-ECC RAM, I don't really think my config is that janky,

Okay, I meant "janky" in the nicest, endearing term possible. :)

Good that you do have backups and I would assume you are keeping FreeNAS config backups too? Best advice I could offer for now is to go with the mirrored USB Sticks (as mentioned by the Doctor ;) )

 

[Clevinger]

Yeah, I backup config whenever config changes, wish there was a way to automate that.

 

[Mirfster]

Yeah, I backup config whenever config changes, wish there was a way to automate that.

Ask and ye shall receive... courtesy of @Bidule0hm "Scripts to report SMART, ZPool and UPS status, HDD/CPU T°, HDD identification and backup the config"

P.S., it is also listed under "Recommended Reading" in my sig...

 

[Robert Smith]

Yeah, I backup config whenever config changes, wish there was a way to automate that.

Here is another one, I have been using for two years:
https://forums.freenas.org/index.php?posts/134874/

 

[Clevinger]

Thanks!