Disk online, but unavailable after reboot, pool has no errors nor degraded

[corywinter]

Recently upgraded from Core to Scale. I restarted my system this morning. During the reboot, the system accidentally rebooted back into Core due to that being the default in the boot loader. Prior to reboot my 4 disk raidz2 appeared fine in the UI. However, during the next reboot from Core to Scale I received an email:

ZFS has detected that a device was removed.

impact: Fault tolerance of the pool may be compromised.
eid: 2
class: statechange
state: UNAVAIL
host: nas
time: 2023-02-22 13:15:21-0330
vpath: /dev/sde2
vguid: 0x29CB958513F785F4
pool: 0x352B24CE78C77E67

When I view the pool in the UI I see:

But if I check the status of the pool from the command line I see:

root@nas[~]# zpool status shelcordata
pool: shelcordata
state: ONLINE
scan: scrub repaired 0B in 09:41:19 with 0 errors on Mon Feb 13 12:41:19 2023
config:

NAME STATE READ WRITE CKSUM
shelcordata ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
f44af8ac-8fe8-11ec-8bde-002590d5247d ONLINE 0 0 0
e653c0cc-d5b7-11e9-b7b3-002590d5247d ONLINE 0 0 0
7b48878c-8f4a-11ec-8bde-002590d5247d ONLINE 0 0 0
7e7febc0-d4e6-11e9-b7b3-002590d5247d ONLINE 0 0 0

errors: No known data errors


I'm not sure what this means. I've opened the case and replaced the SATA cables and reseated everything. Any idea of the best way to proceed. Could this have been caused by the accidental boot into Core?

Thanks for any help,
Cory

 

[corywinter]

To further the weirdness of my setup, my system started as FreeNAS, years ago, and has followed a normal upgrade path over that time, leading to TrueNAS Core. My system was setup on with the boot pool on a USB drive which was later mirrored to another USB drive. So, I had two USB drives mirrored for my boot pool, 4 drives in raidz2 for my system dataset set and storage. I added two more drives (mirrored) earlier this week. However, here is what my system looks like today:

It looks like one of my 4TB drives from my raidz2 has been taken for my "freenas-boot" pool.

 

[Patrick M. Hausen]

How are your drives connected?

 

[corywinter]

The USB boot devices are connected one to the motherboard and the other on the back panel. The motherboard is a supermicro X9SCM-F. The hard drives are in a rosewill RSV-L4412 case which are installed in hot swap bays that are then connected directly to the motherboard.

 

[corywinter]

I decided to backup my configuration and reinstall Scale on my two USB drives. After that I reimported my config and it appears that everything is back to how it should be!

Easy peasy!