SOLVED Yup. Another No pool found

[magickarle]

Strange thing.
After a reboot, I had no more pool.
Can't zpool import:
sudo zpool import
no pools available to import

I did my best to gather all info needed by doing searchers and posting here what's needed.
Sorry if I missed something, I'm not a Guru. Hope to contribute soon tho!

The only thing i remember doing was changing a UID and GID of a recently create user and group (since the GUI doesn't allow me).
The change didn't reflect in GUI so I rebooted.

sudo zpool status -v MainPool
cannot open 'MainPool': no such pool

sudo zpool import -f -F -R /mnt MainPool
cannot import 'MainPool': no such pool available

sudo cat /var/log/middlewared.log | grep MainPool
[2023/05/08 19:19:45] (ERROR) PoolService.import_on_boot():1790 - Failed to import MainPool
libzfs.ZFSException: cannot import 'MainPool' as 'MainPool': no such pool or dataset
middlewared.service_exception.CallError: [EZFS_NOENT] Failed to import 'MainPool' pool: cannot import 'MainPool' as 'MainPool': no such pool or dataset
middlewared.service_exception.CallError: [EZFS_NOENT] Failed to import 'MainPool' pool: cannot import 'MainPool' as 'MainPool': no such pool or dataset
[2023/05/08 19:19:45] (ERROR) PoolService.import_on_boot():1800 - Failed to import MainPool
middlewared.service_exception.CallError: [ENOENT] Pool MainPool not found.
middlewared.service_exception.CallError: [ENOENT] Pool MainPool not found.
[2023/05/08 19:19:45] (DEBUG) SystemDatasetService.setup():356 - Root dataset for pool 'MainPool' is not available, and dataset 'MainPool/.system' does not exist, temporarily setting up system dataset on boot pool
[2023/05/08 19:19:48] (DEBUG) middlewared.render_body():209 - /mnt/MainPool/somesharename: path does not exist, omitting from NFS exports
[2023/05/08 19:19:48] (DEBUG) middlewared.render_body():209 - /mnt/MainPool/somesharename1: path does not exist, omitting from NFS exports
[2023/05/08 19:19:48] (DEBUG) middlewared.render_body():209 - /mnt/MainPool/somesharename2: path does not exist, omitting from NFS exports
[2023/05/08 19:19:48] (DEBUG) middlewared.render_body():209 - /mnt/MainPool/pictures/somesharename3: path does not exist, omitting from NFS exports
... "Bunch of other smb anf nft driver not mapped"

The HBA is still passthrough:
sudo lspci
00:00.0 Host bridge: Intel Corporation 440FX - 82441FX PMC [Natoma] (rev 02)
00:01.0 ISA bridge: Intel Corporation 82371SB PIIX3 ISA [Natoma/Triton II]
00:01.1 IDE interface: Intel Corporation 82371SB PIIX3 IDE [Natoma/Triton II]
00:01.2 USB controller: Intel Corporation 82371SB PIIX3 USB [Natoma/Triton II] (rev 01)
00:01.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 03)
00:02.0 VGA compatible controller: Device 1234:1111 (rev 02)
00:03.0 Unclassified device [00ff]: Red Hat, Inc. Virtio memory balloon
00:05.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:08.0 Communication controller: Red Hat, Inc. Virtio console
00:10.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
00:11.0 Ethernet controller: Intel Corporation 82599 Ethernet Controller Virtual Function (rev 01)
00:12.0 Ethernet controller: Red Hat, Inc. Virtio network device
00:1b.0 Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)
00:1e.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
00:1f.0 PCI bridge: Red Hat, Inc. QEMU PCI-PCI bridge
01:01.0 SCSI storage controller: Red Hat, Inc. Virtio SCSI

but no disk found:
lsblk
NAME MAJ:MIN RM SIZE RO TYPE MOUNTPOINT
sda 8:0 0 32G 0 disk
├─sda1 8:1 0 1M 0 part
├─sda2 8:2 0 512M 0 part
└─sda3 8:3 0 31.5G 0 part
sr0 11:0 1 1.6G 0 rom

 

[NickF]

If lsblk is not showing you your disks, then that’s your first clue. You said “HBA is still passthru” can you explain? How is the system configured

 

[magickarle]

If lsblk is not showing you your disks, then that’s your first clue. You said “HBA is still passthru” can you explain? How is the system configured

Truenas is in VM under host proxmox.

The HBA PCI is being "passthrough" to the VM (so HBA is reserved and connected directly to the VM truenas).
All 16 drives are connected to the HBA so they are supposed to be detected (as previously when the MainPool was working)

 

[magickarle]

Seem my signature is not showing:
Asrock romed8 - AMD EPYC 7282 Single socket
128GB ECC
HBA LSI Broadcom SAS3008 Fusion MPT -->16x 14TB SAS3 (8 wide 2Xmirror)
Onboard: 2x 2TB nvme Raid1
Proxmox -->TrueNas Scale
1x 32Gb boot
OS Version:TrueNAS-SCALE-22.12.2
Product:Standard PC (i440FX + PIIX, 1996)
Model:Common KVM processor
Memory:96 GiB

 

[NickF]

I would suggest booting into TrueNAS on bare metal off of a thumb drive or something and see if the pool will import. Sounds like a problem with your hypervisor but in order to rule out a hardware problem that’s where I would start

 

[magickarle]

Yeah... Though it wouldn't be a quick fix agrr

I'll remove the passthrough and try on the proxmox see if all drives are seen.

 

[NickF]

I mean if you wanted quick and easy you shouldn’t have run inside of a hypervisor :P

SCALE uses KVM also ya know :)

 

[magickarle]

I mean if you wanted quick and easy you shouldn’t have run inside of a hypervisor :P

SCALE uses KVM also ya know :)

Yes. But I though it would be more secure and stable to have it let alone without anything running beside storage stuffs

 

[magickarle]

So final verdict: backplane dead.
I've bypass one drive and connected it directly to HBA and pouf detected.

If cheap enough, I'll have a spare one on the side juste in case... Still under warranty at least.

 

[jgreco]

So final verdict: backplane dead.

I'm very sorry to hear that. It's a royal PITA. If it's a recent vintage Supermicro chassis, the vaguely good news is that it is no longer like trying to disassemble an internal combustion engine to get at the backplane. The bad news is replacements are pricey.

Please do come back after your scrapes, cuts, and maybe bruises and let us know how it went. Someone will buy you a virtual beer, I'm sure, just to hear the story.

 

[magickarle]

Nah it's a Chinese branded backplane.
They are very reputable BUT far away so it's going to take a while to get it.
So I've bought one extra as "redundancy" and have send the defect one.
They'll give away 2x2.5 hot bay as customer service (better than nothing. I got spaces for them)

 

[NickF]

Nah it's a Chinese branded backplane.
They are very reputable BUT far away so it's going to take a while to get it.
So I've bought one extra as "redundancy" and have send the defect one.
They'll give away 2x2.5 hot bay as customer service (better than nothing. I got spaces for them)

Is it though? Yours did just die...

What brand is it?

 

[jgreco]

What brand is it?

"Shenzhen Back Alley Back Plane Half Bent Sheet Metal Electronics Co."

 

[magickarle]

Is it though? Yours did just die...

What brand is it?

Innovision tech

 

[magickarle]

"Shenzhen Back Alley Back Plane Half Bent Sheet Metal Electronics Co."

No. The PCB look to be high quality.
I was getting good speed on it (8 wide 2x mirror) read: 12.81GB/s (IOPS 165GB) and write 8.3GB/s (IOPS 302 GB).
I've attached some closeup pics.

 

[NickF]

Ah I found the "Shenzhen Back Alley Back Plane Half Bent Sheet Metal Electronics Co." official store

https://www.aliexpress.com/store/1100472294?spm=a2g0o.detail.100005.1.281658af0hEiio

Anyway it looks like this chip cooked itself

Looks to be a clone or knockoff of this maybe?

MAX® II EPM240 CPLD - Product Specifications | Intel

MAX® II EPM240 CPLD quick reference with specifications, features, and technologies.

www.intel.com

 

[jgreco]

Oh, Innovision, the fake Supermicro folks.

 

[NickF]

Oh, Innovision, the fake Supermicro folks.

Those drive caddies were made by Innovision first!

 

[magickarle]

Oh, Innovision, the fake Supermicro folks.

Didn't know

 

[jgreco]

Those drive caddies were made by Innovision first!

I suppose that could be, Cite? I'm too lazy to go pull some Gen 1 Supermicro trays tonite.