SOLVED single "Read Errors" on different drives

[Arwen]

What are the disk manufacturer & models on this second pool?

 

[Lesani]

I only really have one pool at this time.

I had HGST drives

=== START OF INFORMATION SECTION ===
Vendor: HGST
Product: HUS724040ALS640
Revision: A3A0
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches

I had the feeling the DELL (Seagate) drives gave less errors, so I an now slowly replacing the ever unhealthy, degrading and faulting hgsts with

=== START OF INFORMATION SECTION ===
Vendor: SEAGATE
Product: ST4000NM0023
Revision: GS0F
Compliance: SPC-4
User Capacity: 4,000,787,030,016 bytes [4.00 TB]
Logical block size: 512 bytes
LU is fully provisioned
Rotation Rate: 7200 rpm
Form Factor: 3.5 inches

I have replaced the SAS cables with brand new ones, and routed them as far away from the power delivery as possible in this server chassis. I am still running into the same issue with read errors and drives faulting sooner or later. The HGST seem to be the worst offenders, with the last HGST drive (SDD) showing 62 read errors and pending a replacement, there is currently a resilver going on after I replaced an HGST with a Dell/Seagate drive (SDI) which is already showing checksum errors as well...

 

[NugentS]

Well if it isn't the disks, must be the controller. Maybe???

 

[Lesani]

Whats a good hba to get that doesn‘t burst the bank? I have 2 mini-sas connectors going to the backplane.

SDD(hgst) is now faulted overnight with too many errors… SDI(Dell/Seagate) is accumulating more abd more checksums

 

[NugentS]

The answer to that question is always LSI - it then depends on what flavour you want.
Caveats:
1. Not MegaRAID
2. It needs to flashed to IT Mode

Top Picks for FreeNAS HBAs (Host Bus Adapters)

We have our top picks for getting fast and reliable FreeNAS HBAs (host bus adapters) for SAS and SATA, using proven options for FreeNAS and ZFS

www.servethehome.com

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop...

www.truenas.com

FreeNAS® Quick Hardware Guide

This is the FreeNAS Quick Hardware Guide. It is not meant as a replacement for the Hardware Recommendations Guide - it is just a sort of quick reference for popular parts. To download this document use the orange button. If you know of...

www.truenas.com

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

Hardware Buying Guide - some tips for not getting ripped off

I've been giving this information out piecemeal for years, and should maybe have it summarized in one place. There's been a lot of people who are wary of secondary market sources for hardware, such as eBay, due to bad experiences with other...

www.truenas.com

 

[Lesani]

quick update: Brand new cables, different drives (the dell drives described above), same situation. Drives get faulted because of read errors.

 

[Samuel Tai]

Unfortunately, it appears your HBA is going bad.

 

[sretalla]

I note that SCALE at the version you report seems to run ZFS 2.1.2, which may be impacted by this:

Is this bug relevant on TrueNAS - ZFS corruption related to snapshots post-2.0.x upgrade

On Linux there is a nasty bug: when sending native encrypted snapshots to another pool sometimes the source pool is marked as degraded with data errors. It's reported here: ZFS corruption related to snapshots post-2.0.x upgrade. This is a sample zpool output: zpool status -v pool: rpool...

www.truenas.com

ZFS corruption related to snapshots post-2.0.x upgrade · Issue #12014 · openzfs/zfs

System information Type Version/Name Distribution Name Debian Distribution Version Buster Linux Kernel 5.10.0-0.bpo.5-amd64 Architecture amd64 ZFS Version 2.0.3-1~bpo10+1 SPL Version 2.0.3-1~bpo10+...

github.com

 

[Lesani]

Quick update, new HBA arrived, pool is now resilvering (80%) after replacing a drive showing as degraded, apart from a few crc errors when scrubbing before the replacing/resilvering (that were probably real, because of the ordeal before) no additional errors appeared yet. With the other HBA a replace/resilver already lead to lots of "read" errors on all drives, so I assume the problem is fixed and was indeed the HBA. Will update a last time in a few days when I have transferred some data again.

 

[revengineer]

As mentioned above, it could still be cooling. I have an LSI HBA and it worked fine for over a year. Then errors started to occur randomly although they increased when the HBA was taxed during a scrub twice a month. I ended up attaching a 40 mm noctua fan onto the heat sink and the errors went away. I have a Supermicro server case, I modified it and took out the forced air jet-engine fans. While I have seven fans inside the box, I concluded that the air flow is just not good enough for the HBA.

 

[Lesani]

Okay, final update: It was the HBA. I got the same exact HBA, only the board looks a bit newer than the one my server was shipped with, and this one has not produced any errors since its in the system (apart from and after the initial resilver/scrub directly after replacing).

This issue is solved, I do not think overheat can be the issue since the HBA is identical, except maybe the thermal compount on the "older" one may be completely dried out. Thanks for all the help!

 

[revengineer]

I am glad it's working for you again. These HBAs should last forever. If you find that this new one suffers an untimely death you can revisit the idea of a thermal issue. A 40 mm fan is only ~$10, just saying...

 

[Lesani]

I am glad it's working for you again. These HBAs should last forever. If you find that this new one suffers an untimely death you can revisit the idea of a thermal issue. A 40 mm fan is only ~$10, just saying...

I actually will even replace the thermal compound on the "old" one and see if it won't run again without errors as well ;) no worries your idea/suggestion is duely noted! thanks!