Pool Corruption on Power Loss

[ninjai]

I have a fairy simple setup for my home NAS - 1 FreeNAS server running all the recommended hardware, and a USB external hard disk running ZFS for local replication for backups.

I don't recall having a problem with this in the past, but lately it seems any time my NAS shuts down unexpectedly, the USB disk pool becomes corrupted and I have to manually online the pool. The "corruption" is always similar, which involves metadata and the mountpoint. IE:

Code:

errors: Permanent errors have been detected in the following files:

        <metadata>:<0x0>
        <metadata>:<0x1b>
        /mnt/FreeNAS-Backup/

/mnt/FreeNAS-Backup is my pool's mountpoint. I don't really know what to make of this. Oracle's doc on the <metadata> things confuses and makes no sense to me:

f an object in the metaobject set (MOS) is corrupted, then a special tag of <metadata>, followed by the object number, is displayed.

If the corruption is within a directory or a file's metadata, the only choice is to move the file elsewhere. You can safely move any file or directory to a less convenient location, allowing the original object to be restored in its place.

I guess the two things I'm looking to find an answer for on this thread are:

1. If my mountpoint is corrupt, what does that even mean? I can access all my data still.
2. Why does this always happen to my USB drive and never my SATA pool? I might also note I don't think I experienced this once on FreeNAS 11.2, just on 11.3.

 

[Heracles]

1 FreeNAS server running all the recommended hardware, and a USB external hard disk

A little problem here... External USB are a No-Go for FreeNAS. If you have one, then you are not using the "recommended hardware" at all...

So what you should do is to migrate these data out of that USB drive. You can use another server, a cloud service or more, but an external USB is not recommended at all for FreeNAS storage.

 

[ninjai]

A little problem here... External USB are a No-Go for FreeNAS.

See that's the fun part. I knew this, but recently I've started dealing with IX Systems directly for selling IX NAS appliances to clients. I specifically mentioned this question on my call with them and their engineer said "I've never heard of that, it's perfectly fine to use a USB disk for these purposes"

-_-

 

[HoneyBadger]

See that's the fun part. I knew this, but recently I've started dealing with IX Systems directly for selling IX NAS appliances to clients. I specifically mentioned this question on my call with them and their engineer said "I've never heard of that, it's perfectly fine to use a USB disk for these purposes"

-_-

Did you clarify with the iX engineer that this was a single-disk USB pool? If your enclosure or drive doesn't properly respect cache flushes, then it could result in a situation where metadata is being reported as written to ZFS when it's actually floating in the spinning disk's volatile RAM cache.

What's the make/model of your USB disk? (Hopefully the answer doesn't indicate an SMR-based device, that's just icing on the cake.)

Regarding the 11.2/11.3 change, I've noticed some different behavior with USB in 11.3 (such as USB NICs - I use them for additional or OOB management, not data transfer, so don't @ me) which could potentially be contributing to this as well.

 

[morganL]

Most USB drives do not have "power Loss protection".... if the power is lost during a sequence of writes, the media can be corrupted.

Most, but not all, SATA SSDs do have "power loss protection" which typically requires extra capacitors to sustain power while completing a write. If a write is acknowledged, it will be properly committed to flash. On power loss, there should be no corruption of data that has been acknowledged.