ZFS Checksum Errors on New Drive During Resilvering

[SeaWolfX]

I recently replaced a disk in my ZFS mirrored vdev as one of the drives was showing signs of being unhealthy (my first time attempting a resilvering).

The four drives in my ZFS vdev are connected via a Chieftec hotswap backplane to a X11SCH-F motherboard.

I removed the bad drive, replaced it with a new identical one (Seagate Exos 78E Enterprise drive), run S.M.A.R.T short-, conveyance-, and long tests on it, updated the alias to point to the new drive and started the resilvering using zpool replace.

During the resilvering some checksum errors showed up in zpool status and they were also present after the resilvering was done.

I checked the SMART metrics which showed no concerning values. I then decided to do a badblocks test (I know, I should have done that before resilvering), but it finished without any errors.

Now I am not sure how to proceed. Could the checksum error I saw just be false errors due to the resilvering? Is this something I need to be concerned about and investigate further (if so what should I look for) or should I just resilver the drive again and ignore any checksum errors?

(Edit: Please let me know if this is not the correct sub for this issue and I'll move it)

Before Resilvering

During Resilvering

After Resilvering

Comparing SMART Metrics of Both Drives in Mirror

Badblock Test

 

[Arwen]

The "Load_Cycle_Count" for both old and new drives seems high. It's possible that the drives have some of the power saving options enabled that NAS specific drives don't have. Perhaps the re-silver paused at times, which allowed the drive to park it's heads. Then when ZFS resumed, it had a time out failure because of the time it took to re-load the heads on to the platters.

For hard disk drives that don't use head parking, the "Power_Cycle_Count" would be roughly the same as the "Load_Cycle_Count".

Other than that, I don't have a useful suggestion.

 

[SeaWolfX]

The "Load_Cycle_Count" for both old and new drives seems high. It's possible that the drives have some of the power saving options enabled that NAS specific drives don't have.

I am not sure to be honest. Would that be enabled by default and how can I check this?

 

[elvisimprsntr]

I noticed your mobo has a RAID controller.

Re: https://www.supermicro.com/en/products/motherboard/X11SCH-F

Intel® C246 controller for 8 SATA3 (6 Gbps) ports; RAID 0,1,5,10

My understanding is RAID controllers are to be avoided. Read the following links.

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

ZFS - Wikipedia

en.wikipedia.org

Some forum members have been successful in getting Supermicro to send them firmware that will re-flash the on-board controller to IT mode.
Your other option might be to install a separate non-RAID HBA.

 

[SeaWolfX]

I noticed your mobo has a RAID controller.

Re: https://www.supermicro.com/en/products/motherboard/X11SCH-F



My understanding is RAID controllers are to be avoided. Read the following links.

What's all the noise about HBA's, and why can't I use a RAID controller?

This is relevant to FreeNAS and TrueNAS CORE. Some parts of it might also be relevant to Scale, but I don't really know how reliable the Linux drivers are. 1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with...

www.truenas.com

ZFS - Wikipedia

en.wikipedia.org

Some forum members have been successful in getting Supermicro to send them firmware that will re-flash the on-board controller to IT mode.
Your other option might be to install a separate non-RAID HBA.

Maybe I have misunderstood this, but I am not actually using any HW RAID feature on the card. I did a lot of research on this forum before selecting a motherboard and this one was very often recommended. Why would that be if there might be an issue with onboard RAID controller on this MB when using ZFS.

 

[Arwen]

I am not sure to be honest. Would that be enabled by default and how can I check this?

You don't specify which TrueNAS release, Core, (based on FreeBSD) or SCALE, (based on Linux). But I am guessing SCALE based on your disk naming convention.

Linux has a tool called hdparm which can show the current settings;

Code:

# hdparm -B -M /dev/sda

/dev/sda:
 APM_level      = 254
 acoustic      = 254 (128=quiet ... 254=fast)

This shows my 2.5" laptop drive's settings, which used to have quite increasing "Load_Cycle_Count". Now I have disabled both "features" and no more increasing "Load_Cycle_Count".

These can be changed with;

Code:

# hdparm -B 254 -M 254 /dev/sda

I don't know the TrueNAS Core / FreeBSD tool name or how to use it.

 

[SeaWolfX]

You don't specify which TrueNAS release, Core, (based on FreeBSD) or SCALE, (based on Linux). But I am guessing SCALE based on your disk naming convention.

Linux has a tool called hdparm which can show the current settings;

Code:

# hdparm -B -M /dev/sda

/dev/sda:
 APM_level      = 254
 acoustic      = 254 (128=quiet ... 254=fast)

This shows my 2.5" laptop drive's settings, which used to have quite increasing "Load_Cycle_Count". Now I have disabled both "features" and no more increasing "Load_Cycle_Count".

These can be changed with;

Code:

# hdparm -B 254 -M 254 /dev/sda

I don't know the TrueNAS Core / FreeBSD tool name or how to use it.

You are correct, I'm on Linux.

Seems like APM is not supported on my drives though :/

Doing a bit of searching it seems I might be able to do something similar using the SeaChest tool. Not sure how to install it on Linux or what parameters to change though.

In any case; is 45K LLC such a high number? The drives have a rating of 600K and they have been up and running for about 1.5 years. At this rate they should be good at least for 20 years before they reach their limit.

 

[Arwen]

The point is that you have ZFS checksum errors. It's a guess on my part that these so called features are interfering with ZFS' reads, causing the checksum errors. So I suggested eliminating something and seeing if that improved the situation.

In my case, I was getting 1,000s of LCC, in far excess of the expected life of the disk. It was parking, but ZFS would wake it up. However, for me, before the change, it did not create any ZFS errors, checksum or otherwise. Thus, as I said, just a guess.