FreeNAS 11.2 - Some drives UNAVAIL after reboot

[8181988]

Hi,

I’m pretty new to FreeNAS and ZFS and I’m a little out of my depth troubleshooting an issue I’m having. My hardware is listed below.

• Motherboard - ASUS Prime x370-Pro
• CPU - AMD Ryzen 5 1600
• RAM - 32GB
• Drives -
  • 15x10TB shucked WD EasyStore drives in RaidZ3(8 connected directly to the motherboard and 7 connected via 2 Rivo PCIe SATA adapters)
  • an NVME boot drive (not sure what brand)
• OS - FreeNAS-11.2-U6

Some background on the issue: tldr I nearly ran out of space on this NAS (I know that I should expand if I start getting to high usage; I was basically freeing up other drives by stashing some data on this NAS to add the drives to the pool afterwards). I started clearing things from this NAS to free up space once I saw I was running very low and while I was doing that the FreeNas web GUI stopped responding (could have been an unrelated network issue; that has happened relatively frequently), I rebooted and the pool was showing LOCKED and under Disks in Use on the Dashboard page of the GUI, it is only showing 9 of the drives.

Looking under Storage > Disks, it lists all the drives but ada10 through ada13 are showing “Unused” in the Pool column. The other drives are showing they are in the “bigdata” pool like normal. And looking at smartctl they are all passing so I dont think the drives themselves are failing.

Running zpool status only shows the boot drive.

Code:

root@bigdata[~]# zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:01 with 0 errors on Wed Jun 23 03:45:01 2021
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          nvd0p2    ONLINE       0     0     0

errors: No known data errors

The “bigdata” pool isn’t showing up. Running status on that pool name shows:

Code:

root@bigdata[~]# zpool status -x bigdata
cannot open 'bigdata': no such pool

Then running zpool import shows the pool but list some of the drives as UNAVAIL.

Code:

root@bigdata[~]# zpool import
   pool: bigdata
     id: 7767287315413224548
  state: UNAVAIL
 status: One or more devices are missing from the system.
 action: The pool cannot be imported. Attach the missing
        devices and try again.
   see: http://illumos.org/msg/ZFS-8000-3C
 config:

        bigdata                                             UNAVAIL  insufficient replicas
          raidz3-0                                          UNAVAIL  insufficient replicas
            15226000968890956198                            OFFLINE
            gptid/2140a413-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/226a4606-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/237215b3-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/248b4d89-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/25b2ca3d-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/26cfa543-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/27e71dfe-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/28f8c628-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/2a1d7c9a-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            gptid/2b34aa7c-e7f0-11e9-8f78-4cedfb7500ce.eli  ONLINE
            6712324017752305159                             UNAVAIL  cannot open
            1901687950625073763                             UNAVAIL  cannot open
            11734792618802854983                            UNAVAIL  cannot open
            1167907853653725744                             UNAVAIL  cannot open

Checking in the /var/log/messages, I am seeing some of the following:

Code:

Jun 29 00:24:16 bigdata (ada0:ahcich0:0:0:0): READ_FPDMA_QUEUED. ACB: 60 00 80 0
1 40 40 00 00 00 01 00 00
Jun 29 00:24:16 bigdata (ada0:ahcich0:0:0:0): CAM status: ATA Status Error
Jun 29 00:24:16 bigdata (ada0:ahcich0:0:0:0): ATA status: 41 (DRDY ERR), error:
84 (ICRC ABRT )
Jun 29 00:24:16 bigdata (ada0:ahcich0:0:0:0): RES: 41 84 6f 02 40 40 00 00 00 10
 00
Jun 29 00:24:16 bigdata (ada0:ahcich0:0:0:0): Retrying command

I only see this error for ada0 and that drive was previously having issues (although it passed smartctl as well) so this may also be unrelated.

As I mentioned in the hardware section, 7 of the drives are using PCIe to SATA adapters. One of those adapters has four drives attached to it so I thought the issue may be with one of the adapters. I tried replacing the adapter and the drives are still showing UNAVAIL in zpool import. So it looks like that isn’t the issue.

I’m not sure where to go from here. Is reconnecting these drives possible? Is this recoverable?