SOLVED TrueNAS Scale not recognizing disks after migration to Proxmox

[UltraHKR]

Hi, everybody I hit a very worrisome glitch in TN Scale v22.12.0

Yesterday I migrated from VMware ESX 6.7 to Proxmox 7.3-4

My previous setup was:
ESXi 6.7.0
Supermicro X8DTH
192GB RAM
2x Intel Xeon X5650
LSI 9211-8i IT + SAS Expander (connected to Supermicro SC836-TQ backplane) passthrough to WinVM
LSI 9211-4i4e IT + IBM V3700 Storwize Expansion Chassis (24x 1.2TB SAS 10k RPM) passthrough to TN-S
- This is setup as 2x VDEV RAID-Z2 (12x 1.2TB HDDs)

Procedure:

• Export VM's to temporary storage
• Shutdown ESXi
• Physically disconnect JBOD chassis
• Install & Upgrade Proxmox
• Import VM's to temporary storage
• Physically connect JBOD chassis (something happened maybe?)
• Setup PCIe Passthrough (Setup with ROMbar=1)
• Notice a bunch of error in TNS boot output
• Halt VM
• Disable ROMbar
• Boot TN Scale
• Notice 5!! disks not showing up properly but ZFS array was up
• Notice in TNS WebUI / Storage "Mixed Capacity VDEVs"
• Notice in lsblk that some have different UUID's format.
• Tried export/disconnect but it failed
• Proper shutdown

Today I booted TN Scale to gather info & post this, 3 disks show as "Unassigned", which are part of the same 5 that had different ID in lsblk.

Can somebody help me, I'm pulling my hair out!!!

dmesg output:

TrueNAS Scale dmesg - Pastebin.com

Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time.

pastebin.com

lsblk output:

Code:

root@truenas[~]# blkid
/dev/sda1: PARTUUID="68384e79-e240-4505-a1a1-ddd92664180d"
/dev/sda2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="4955871521863246775" TYPE="zfs_member" PARTUUID="dc411eed-3595-4867-9094-79fb9175455d"
/dev/sdc2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="3399248928835922137" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="738842b9-55ba-4946-8686-90eed73e31f4"
/dev/sde2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="7269767215980076810" TYPE="zfs_member" PARTUUID="08315d22-108c-47ac-91bd-2cc2da9e48e3"
/dev/sdb2: LABEL_FATBOOT="EFI" LABEL="EFI" UUID="4AB7-8AC9" BLOCK_SIZE="512" TYPE="vfat" PARTUUID="6a90c1a7-e551-4ff7-88e0-a684a7e76da1"
/dev/sdb3: LABEL="boot-pool" UUID="1236061093502694888" UUID_SUB="7532598079954394882" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="59ce9fad-67cd-4850-817a-18c40d9e2139"
/dev/sdf2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="2987918591311655878" TYPE="zfs_member" PARTUUID="bc8b3dad-8d5b-4138-a259-5e7a6e9281da"
/dev/sdd1: PARTUUID="ab8759d3-f08a-40eb-9042-4f6831cb6f09"
/dev/sdd2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="3221088706848709231" TYPE="zfs_member" PARTUUID="b761fade-8ddc-46ff-96a1-32c873c2b836"
/dev/sdh2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="6616522105816249851" TYPE="zfs_member" PARTUUID="227d2f5b-a380-43dc-8fd6-1fe8294d628c"
/dev/sdg2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="12511531234521748538" TYPE="zfs_member" PARTUUID="2abe9dfd-e8eb-4f4c-9943-aa19b5be759a"
/dev/sdq1: PARTUUID="78e93054-f1b8-4768-8101-ff6b7712c52d"
/dev/sdq2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="17939442023012658277" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="d3fd8ff8-a83b-43cd-ac61-287ef0655284"
/dev/sdi2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="8350735502036130272" TYPE="zfs_member" PARTUUID="9ec04b9d-137e-4b65-824e-f0fd4718d487"
/dev/sdr1: PARTUUID="c55f2071-7101-49a1-824b-c6490fa91340"
/dev/sdr2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="13130528867977332524" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="9c203372-8564-482e-9eb7-ba1c75776855"
/dev/sdm1: PARTUUID="aa623047-17b0-4608-b2f8-c11c7ab30a5d"
/dev/sdm2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="568340652667323563" TYPE="zfs_member" PARTUUID="1bb07cb7-777c-4564-a688-f523b3150fad"
/dev/sdn1: PARTUUID="c1b9babc-3b77-4ecb-b1aa-fc033bd0bca5"
/dev/sdn2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="14720096478000702204" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="3ba11014-166f-4ed1-a350-3415dd42d8d5"
/dev/sdo1: PARTUUID="d09f62e0-be67-4b8d-a784-26797079831b"
/dev/sdo2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="14184427881434225388" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="070fb579-6145-4fc5-b68a-70364fc6754f"
/dev/sdv1: PARTUUID="0cca545c-2ed4-416a-a523-f7495af0e119"
/dev/sdv2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="2946630033926563072" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="c8e29a1a-23d1-45c2-979c-163cdecc2920"
/dev/sdp1: PARTUUID="e6d52ef1-187a-4bcd-afda-c9b055ffae81"
/dev/sdp2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="1168633136597703895" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="6a59c659-8eb2-4edd-8ee7-d0b16d627896"
/dev/sds1: PARTUUID="e61167c0-1c8a-4ecd-b2d2-735e6e97af27"
/dev/sds2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="4002431468198896063" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="9e55bc79-61bf-484d-b834-0c82e750aa03"
/dev/sdl1: PARTUUID="5be0bda2-d061-4878-9429-bae0c61da6b7"
/dev/sdl2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="13716147173639506093" TYPE="zfs_member" PARTUUID="616693d5-0e88-4c5b-9a4e-191c766b95c7"
/dev/sdk2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="3822929049176563958" TYPE="zfs_member" PARTUUID="52cebb77-18a0-42dc-a84b-eb021ed6f78a"
/dev/sdj2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="4917454713229674698" TYPE="zfs_member" PARTUUID="c22f0886-a4e6-4dc1-90ab-fb734ad883df"
/dev/sdt1: PARTUUID="5bb7b301-b6f7-496b-980e-06b8eab4a1ef"
/dev/sdt2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="5072772609505700777" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="b3cf51b4-a9b9-4534-a68e-ba91fd046f1e"
/dev/sdu1: PARTUUID="d8617411-3cc7-4098-ada2-c3831c35c113"
/dev/sdu2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="1160298841561373888" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="7760f847-8618-497e-b71b-c380261cb38d"
/dev/sdy2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="16094987467223522341" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="d0043e32-bd33-4977-a7b7-84e91a5af2f6"
/dev/sdw1: PARTUUID="33b2d14d-52a0-4d96-9325-f05b3e267414"
/dev/sdw2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="6507156504864696298" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="7e393f26-b02a-4200-818e-68b1bc9aef9d"
/dev/sdx1: UUID="4a6ca3b8-9768-f652-7b09-f2667931aaba" UUID_SUB="4f46ef2d-214a-241a-4ba6-022218dc845e" LABEL="truenas:swap0" TYPE="linux_raid_member" PARTUUID="2dae8428-3f12-4f3a-894b-5672e3de5eae"
/dev/sdx2: LABEL="V3700-JBOD" UUID="9805043433048200498" UUID_SUB="10344357589208717768" BLOCK_SIZE="4096" TYPE="zfs_member" PARTUUID="be7e22ae-a56b-45f5-b904-070da97b98d0"
/dev/sdc1: PARTUUID="9273cecc-79fe-42a9-b613-ee1c548e0730"
/dev/sde1: UUID="4a6ca3b8-9768-f652-7b09-f2667931aaba" UUID_SUB="a18c0836-8bad-6de2-20bd-aa11ae46b854" LABEL="truenas:swap0" TYPE="linux_raid_member" PARTUUID="65eda642-eeaf-44cc-8bec-5f3e9b2eabe5"
/dev/sdb1: PARTUUID="780b4767-bb40-4d67-b013-37a73c2e0a4d"
/dev/sdf1: UUID="fbb7b793-7fba-fecd-2ab9-d8b3cbca371d" UUID_SUB="b75b3875-b9aa-2d7d-af7f-575df251d4e3" LABEL="truenas:swap1" TYPE="linux_raid_member" PARTUUID="fb0e0ea6-d51d-4698-a5c6-4d2311bc6eea"
/dev/sdh1: UUID="fbb7b793-7fba-fecd-2ab9-d8b3cbca371d" UUID_SUB="9c90f238-1b33-ce7c-8cbe-86c590fb1bc6" LABEL="truenas:swap1" TYPE="linux_raid_member" PARTUUID="465a7540-3dfe-41a7-904f-3932484992be"
/dev/sdg1: UUID="fbb7b793-7fba-fecd-2ab9-d8b3cbca371d" UUID_SUB="8a3dc387-9d0e-63e8-6991-251ab5234bca" LABEL="truenas:swap1" TYPE="linux_raid_member" PARTUUID="139769b4-3c86-4395-a8c5-5754a93e8b0f"
/dev/sdi1: PARTUUID="0c4488e2-756c-4aeb-94f9-575e51e6fba2"
/dev/sdk1: PARTUUID="c825ef61-0018-47ad-aa45-7e7543cb0828"
/dev/sdj1: PARTUUID="9e769708-5e6d-473f-91a6-2589f5bf11c9"
/dev/sdy1: UUID="4a6ca3b8-9768-f652-7b09-f2667931aaba" UUID_SUB="153ff119-8c0e-8c1c-af2d-7581d24789d1" LABEL="truenas:swap0" TYPE="linux_raid_member" PARTUUID="3efe22ab-861a-4f27-94fd-d18e4fdc17d9"
/dev/mapper/md127: UUID="a699f358-dfd7-4671-80b3-d7a602945c8e" TYPE="swap"
/dev/mapper/md126: UUID="511e28c1-0641-4856-a5b0-420aade4345d" TYPE="swap"

TrueNAS Scale Images

 

[morganL]

Its complex...you've changed hypervisors and potentially recabled. Is there any evidence that the issue is caused by SCALE 22.12?

 

[UltraHKR]

Its complex...you've changed hypervisors and potentially recabled. Is there any evidence that the issue is caused by SCALE 22.12?

In fact it was fairly simple.

No recabling done, since I replaced ESXi with Proxmox, HW wise everything stayed were it was.

Hypervisors shouldn't mess with data, but Proxmox or more specifically MDADM does, and that messes with TN-Scale UUID, disk labels or something because exporting the ZFS array and running:
mdadm --zero-superblock /dev/sd[a-z][1-2]

Rebooting and then importing the pool fixes it...

I found it because running zpool list, showed something about disk labels are different, which is an already known bug.

MD RAID headers can cause Linux to steal disks from ZFS · Issue #634 · openzfs/zfs

I received a report from a veteran Gentoo user that MD RAID had stolen two disks from a raidz2 vdev. It seems that ZFS leaves a hole for a bootloader that an MD RAID header can occupy, causing this...

github.com

 

[UltraHKR]

Also im a little worried about that error log at almost the very top of the dmesg, something crashed but I don't what it is to track it down.

 

[UltraHKR]

Also im a little worried about that error log at almost the very top of the dmesg, something crashed but I don't what it is to track it down.

Sorry calltrace @ 0.640856 in dmesg time

 

[morganL]

Sorry calltrace @ 0.640856 in dmesg time

Don't know, but if system is now operating fine, I'd wait for more evidence of a real issue.

Congrats on working out the MDADM issue...beyond my skill set.

 

[UltraHKR]

Don't know, but if system is now operating fine, I'd wait for more evidence of a real issue.

Congrats on working out the MDADM issue...beyond my skill set.

Got it...!!! (maybe)

https://cve.report/CVE-2022-26878/1a48d18b

Search fo commit: "bcf64fb327c8ded638ebd72cb44fd80e3dd28d03"

How can I report this?

 

[morganL]

For this type of CVE, we probably wait for debian to fix and test.
Do you know if Debian has incorprorated the fixes yet?
If so which versions?

 

[UltraHKR]

CVE-2022-26878

https://nvd.nist.gov/vuln/detail/CVE-2022-26878#VulnChangeHistorySection

Link from Debian Security Tracker
EDIT: added description
2 EDIT: Added NIST Change History

 

[morganL]

It will get fixed with later OS updates, but the description doesn't match the problem you reported, so I'd be reluctant to prioritize without evidence that it fixes something significant.

Description

drivers/bluetooth/virtio_bt.c in the Linux kernel before 5.16.3 has a memory leak (socket buffers have memory allocated but not freed).