Corrupt file on brand new install?

[jimwise00]

Hi!

Just performed a brand new install of FreeNAS-11.2-U5, and am seeing a pool error in the install pool from the first boot.

Error is this:

Code:

[jwise@erebor /etc/rc.d]$ sudo zpool status -v freenas-boot
  pool: freenas-boot
 state: ONLINE
status: One or more devices has experienced an error resulting in data
    corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
    entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: none requested
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     3
      mirror-0  ONLINE       0     0     6
        ada0p2  ONLINE       0     0     6
        ada1p2  ONLINE       0     0     6

errors: Permanent errors have been detected in the following files:

        freenas-boot/ROOT/default:<0x2601a>
        //conf/base/etc/rc.d/ldconfig

Assuming I should reinstall and see if this repeats, but any guidance before I do?

 

[jimwise00]

So definitely something wrong with my system -- a shell at the end of installation shows a clean pool status, but the above is back (with different files) after first boot of the system.

 

[jimwise00]

For context, ada0 and ada1 are WD Green m.2 SATA devices, connected to an ASMedia ASM106x m.2 SATA controller.

Given that a single bad m.2 should be recoverable from the mirror, should I assume a SATA controller problem? Or something else?

No errors in dmesg.

 

[melloa]

I'd post your complete server specs, so the folks here could provide a better answer/solution.

 

[Johnnie Black]

Certain SSDs get corrupt when used as boot, it's a known issue:
https://redmine.ixsystems.com/issues/35065

 

[jimwise00]

Thank you, @Johnnie Black . I'll try the tunable there, and report ID information to the issue thread if that works.

@melloa -- the server is:

Hp DL380Gen8, 96 GB ECC RAM, 2 x 8 core processor.

The storage in question are two WD Green 120GB, attached to one of these:

https://www.amazon.com/gp/product/B017IM54GM/ref=ppx_yo_dt_b_search_asin_title?ie=UTF8&psc=1

 

[melloa]

Hp DL380Gen8, 96 GB ECC RAM, 2 x 8 core processor.

Interesting... I'm testing a DL380e Gen 8 to use as a LAB.

I got two SSDs connected to its SATA ports, modified the power circuit for the additional video card to power them, and added a LSI HBA to control the drives. It is running ESXi with FreeNAS as a VM (LSI in passthrough), but for the initial tests FreeNAS was booting from the SSDs without any issues.

After I cut the PCI power to build my SATA one, I've found this, but I'm not sure it would work, as I've not tested. If anyone decide to cut the plug to modify to a SATA power, please use a voltmeter as the HP doesn't see to care too much about wire voltage color code. Also note there are two grounds that should not be connected together - use only one of them.

The only thing I don't like about those blades is that they are 2.5" drives and I didn't find any driver with 4TiB for it for a good price. The ones I saw were desktop grade.

@jimwise00, is your chassis 2.5" or 3.5" drives? I've seen blades like mine for about $100 at a local recycle place, but they are all 2.5".

 

[melloa]

Certain SSDs get corrupt when used as boot, it's a known issue:

That was identified over an year ago and closed by Chris, but I haven't seen any discussion after the post Eric Loewenthal mentioned his WD drivers were acting up as well.

It is a possibility, but I'd test one of those SSDs connected to the motherboard SATA port to make sure it is not the SATA controller.

One thing I've noticed with those HP servers is that they are very sensitive to mods, no HP hardware, etc.

Mine was reporting overheat conditions during tests from a non-installed sensor and reseting three/four times/day, until I realized the ground wires I was using to power my boot SSDs were on two different circuits, causing an unbalanced scenario, reseting the server with false reading.

The configuration I've mentioned above is soaking for a week at the lab, by the way, so not a long time run yet.

 

[jimwise00]

I've got the 3.5" model. It's easy enough to use a blank to put 2.5" SSDs in two of the bays, but that would leave only 6 bays for storage.

 

[sretalla]

I had issues with those WD green 3D nand disks as boot, swapped them for kingston SSDs and had no problem since.

If you run a scrub on your boot volume, you will see some files are not healthy eventually.

There's a few threads and a number of bugs around on this topic (here's just one example https://www.ixsystems.com/community/threads/corrupted-ssd-mirrored-boot-drive.70688/#post-488620 ).

 

[melloa]

I've got the 3.5" model. It's easy enough to use a blank to put 2.5" SSDs in two of the bays, but that would leave only 6 bays for storage.

Can you try connecting them to the MB SATA ports, if you have how to power them?

Or try another SSD as suggested by @sretalla ?

 

[jimwise00]

Yeah -- at $32 for two, I'm going to try the Kingston SSD which people seem to have better results with, per the thread @sretalla linked.

Will report back.

 

[Constantin]

I don't see a motherboard listed, but SuperMicro boards also offer a somewhat pricey option with SATADOMs as boot drives. I use two in a mirror and scrub them regularly.

While Supermicro boards make the use of SATADOMs easy (look for orange/yellow SATA plugs), any SATA connector will do, as long as you can provide external power to the SATADOM. For example, my iXsystems Mini XL came with single 16GB SATADOM and got it's power from an IDE power plug, as opposed to directly from the motherboard.

Whatever you do, I suggest buying a SSD or SATADOM with at least 32GB of memory to allow for lots of system updates before the thing is 80% full and you get an email from the system announcing its unhappiness re: exceeding this quota. I went for 64GB since that'll likely last me through FreeNAS 12.xxx :)

 

[melloa]

I don't see a motherboard listed

It is a HP DL380 Gen 8.

 

[jimwise00]

Quick update. Replaced the two WD Green M.2 devices with two of these:

https://www.amazon.com/gp/product/B07P22T3VD

and have not seen a single error since! Good outcome.

 

[anmnz]

Certain SSDs get corrupt when used as boot, it's a known issue:
https://redmine.ixsystems.com/issues/35065

That was identified over an year ago and closed by Chris, but I haven't seen any discussion after the post Eric Loewenthal mentioned his WD drivers were acting up as well.

The issue was closed on Redmine because issues were migrated to Jira and it now lives, still open, here:

https://jira.ixsystems.com/plugins/servlet/mobile#issue/NAS-100276

(Someone can come up with a more canonical URL -- that's the one it gave me on mobile.)