Suddenly lost 3 pools

[namnnumbr]

Last night, all 3 pools that I had running were no longer detected. I decided to export the pools and attempted to reimport, only to receive an I/O error.

When I attempt to view any attached drives `ls /dev/disk/by-id`, I only see my boot disk. This happens to be the only drive directly attached to the motherboard.

Each pool has its own backplane, so since all 3 pools are down, it's not a backplane issue (unless they all failed simultaneously). Additionally, the drives are a mix of ssd and hdd, and the hdd are from different lots, so it's not a simultaneous lot failure.

Since very non-boot drive is routed through an HBA and sata expansion card, it makes me wonder if my hba suddenly died somehow. If I run `lspci -nn | grep -i hba` nothing seems to be detected.
I've tried "unplugging and replugging" -- what else should I do to debug?

 

[Samuel Tai]

Watch the console messages during boot. If the HBA doesn't emit any console messages, it's a good chance it's dead. If there's another PCI-E port you could move it to, that would rule out a failed motherboard slot.

 

[namnnumbr]

Thanks for the suggestion. I was not seeing any console messages initially. I tried removing the sas expander from the mix for one of the breakout cables and the HBA came back (and drives were recognized). So I'm guessing the issue is with the expansion card OR the cables to it. I'm going to get some replacement cables to try as a next step.