all disks failing in pool at the same time

[digity]

Initially 4 drives reported degraded or faulted simultaneously in my 4 vdevs of 4 x SAS HDD RAIDZ1 pool (picture attached). I figured I had the worst luck. Now almost all of them are degraded or faulted, so now I suspect something else is afoot as 12 drives simply don't fail within a day or two of each other. I am getting read errors while trying to back up data in the pool's datasets and replicate to other pools, so there is a real problem somewhere.

The only changes that has occurred on the pool or server before these issues were an update to TrueNAS 12.0-U2.1 and a weird issue where I renamed a iSCSI connected zvol on this pool and had to modify the VMFS volume to get ESXi to re-mount it again.

Again, I imagine 12 of 16 drives failing is very strange. Any ideas what is really going on and how to resolve it?

P.S. - The drives are 900 GB SAS 10k RPM HDDs in a Dell PowerVault MD1220 disk shelf connected to LSI SAS-9207-8e HBA PCI-e card.

 

[JohnDigital]

Id initially suspect cabling issues of some sort, either data or power, the shelf, possibly the HBA itself. Id reseat all the connections and such. Im not sure how well youll come out from this. Something major it appears is wrong.

The only changes that has occurred on the pool or server before these issues were an update to TrueNAS 12.0-U2.1 and a weird issue where I renamed a iSCSI connected zvol on this pool and had to modify the VMFS volume to get ESXi to re-mount it again.

This is curious, not sure of the implication.

 

[Heracles]

I agree that cabling and the controler are the main suspects here. But that also demonstrate you that no server, TrueNAS or other, can be more than a single point of failure. This is why backups are mandatory and is what will protect you against such a situation. Raid-Z1 is not up to the task anymore with drives as big as the ones we have today.

If you do not already have a backup, do one ASAP. Once done, I would re-design the pool to get rid of Raid-Z1. 2x 6 drives Raid-Z2 would give you the same space but each vdev would keep a minimum of redundancy if a single drive fails in it. Still, even that would not protect you against loosing it all as it seems to happen here. Backups are not optional.

 

[HoneyBadger]

I second the suspicion on a common piece of cabling or hardware. Is your MD1220 in Unified mode, and what's the cable path from your 9207-8e to the JBOD look like?

and a weird issue where I renamed a iSCSI connected zvol on this pool and had to modify the VMFS volume to get ESXi to re-mount it again.

You probably caused a new NAA ID or serial number to get generated for the CAM target layer (iSCSI target) which made VMware see it as a clone/snapshot volume. (Aside: VMFS on RAIDZ generally isn't good for performance or space efficiency vs. mirrors, and the usual caveats about SLOG/sync write safety also apply here.)

 

[digity]

I second the suspicion on a common piece of cabling or hardware. Is your MD1220 in Unified mode, and what's the cable path from your 9207-8e to the JBOD look like?

You probably caused a new NAA ID or serial number to get generated for the CAM target layer (iSCSI target) which made VMware see it as a clone/snapshot volume. (Aside: VMFS on RAIDZ generally isn't good for performance or space efficiency vs. mirrors, and the usual caveats about SLOG/sync write safety also apply here.)

Yes, it's in unified mode. I believe both ports of the HBA are connected to the disk shelf. I'll double check the seating of the cables.

As for the RAID config, I thought having multiple vdevs was good for performance? Do I not get the speeds of 12 drives if there's 3 non parity drives each in 4 vdevs? If not, how do I get the most performance and space from 16 drives (redundancy is lowest priority since the pool is backed up)?

 

[Heracles]

Do I not get the speeds of 12 drives if there's 3 non parity drives each in 4 vdevs?

No, you don't.

With 4 vdevs, consider you have the performance of 4 drives. For that, with mirrors, you would have 6 vdev, so the performance of 6 drives.

The 2x 6 drives Raid-Z2 I proposed would drop that performance to what you could achieve with 2 drives. The difference would be that each vdev would be way more robust than your actual Raid-Z1. With mirrors, the risk is lowered compared to Raid-Z1 in a different way. The proper rebuild of a mirror after a failure requires the proper functioning of a single drive while with your 4 drives Raid-Z1, you need 3 drives to be perfect and error free for the rebuild to succeed. That is, the risk of mirror would still be 3 times lower than your actual level. With Raid-Z2, the vdev will be rebuilt and fixed even if one of the remaining drive contains an error.

 

[digity]

So I powered down everything, verified the SFF-8088 cables were indeed properly seated in all ports, removed the second SFF-8088 cable from the disk shelf and HBA adapter, powered everything up and TrueNAS then said the pool was degraded instead of unavailable (which it was before powering down) and the resilvering process had started. Once the resilvering completed, I cleared the pool's status which changed the pool's status to online, kicked off a scrub and this time it said only two drives were degraded.

In an attempt to make sure these two drives are indeed my culprit I cleared the pool's status again, scrubbed again and this time no disk were degraded and the pool's status was online. I kicked off 4 to 5 more scrubs and still there were no more degraded drives and the pool's status remained online. I'm not sure what's going on here. Furthermore, all of the backup jobs and TrueNAS replication tasks that were failing while the pool was degraded and unavailable are now completing successfully. I imagine if the drives were indeed bad, the jobs/tasks should still fail whether TrueNAS identifies the bad sectors or not.

What's going on here? I can't imagine removing that one SFF-8088 cable was the culprit as TrueNAS was still reporting bad drives after the fact. Any ideas?