selveskii
Cadet
- Joined
- Apr 1, 2021
- Messages
- 1
I am virtualizing TrueNAS Core using Proxmox and tries to build a pool from an array of NVMe drives. I have seven drives in total, from three different brands. Two of them are WD Blue SN550. When building a new pool using the web interface, I found neither WD SN550 showed up as storage devices:
root@truenas[~]# geom disk list
Geom name: vtbd0
Providers:
1. Name: vtbd0
Mediasize: 34359738368 (32G)
Sectorsize: 512
Mode: r1w1e2
descr: (null)
ident: (null)
rotationrate: unknown
fwsectors: 63
fwheads: 16
Geom name: nvd0
Providers:
1. Name: nvd0
Mediasize: 1024209543168 (954G)
Sectorsize: 512
Mode: r0w0e0
descr: ZHITAI PC005 Active 1TB
lunid: a428b70016ba0042
ident: ZTA11T0JA2050000EE
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd1
Providers:
1. Name: nvd1
Mediasize: 1024209543168 (954G)
Sectorsize: 512
Mode: r0w0e0
descr: ZHITAI PC005 Active 1TB
lunid: a428b70015550042
ident: ZTA11T0JA205000123
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd2
Providers:
1. Name: nvd2
Mediasize: 1000204886016 (932G)
Sectorsize: 512
Mode: r0w0e0
descr: KINGSTON SA2000M81000G
lunid: 0026b76843b0e425
ident: 50026B76843B0E42
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd3
Providers:
1. Name: nvd3
Mediasize: 1000204886016 (932G)
Sectorsize: 512
Mode: r0w0e0
descr: KINGSTON SA2000M81000G
lunid: 0026b76843b07be5
ident: 50026B76843B07BE
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd4
Providers:
1. Name: nvd4
Mediasize: 1024209543168 (954G)
Sectorsize: 512
Mode: r0w0e0
descr: ZHITAI PC005 Active 1TB
lunid: a428b70016b90042
ident: ZTA11T0JA2050000ED
rotationrate: 0
fwsectors: 0
fwheads: 0
All seven devices are successfully passed inside the guest. Outputs from dmesg indicates that all of them are recognized as NVMe devices:
root@truenas[~]# dmesg | grep nvme
nvme0: <Generic NVMe Device> mem 0xfe800000-0xfe803fff,0xfe804000-0xfe8040ff irq 20 at device 0.0 on pci1
nvme1: <Generic NVMe Device> mem 0xfe600000-0xfe603fff irq 20 at device 0.0 on pci2
nvme2: <Generic NVMe Device> mem 0xfe400000-0xfe403fff irq 20 at device 0.0 on pci3
nvme3: <Generic NVMe Device> mem 0xfe200000-0xfe203fff irq 16 at device 0.0 on pci4
nvme4: <Generic NVMe Device> mem 0xfe000000-0xfe003fff irq 16 at device 0.0 on pci5
nvme4: <Generic NVMe Device> mem 0xfe000000-0xfe003fff irq 16 at device 0.0 on pci5
nvme5: <Generic NVMe Device> mem 0xfde00000-0xfde03fff,0xfde04000-0xfde040ff irq 16 at device 0.0 on pci6
nvme6: <Generic NVMe Device> mem 0xfdc00000-0xfdc03fff irq 16 at device 0.0 on pci7
When I tried to identify them, I got some errors:
root@truenas[~]# nvmecontrol identify nvme0
nvmecontrol: identify request returned error
root@truenas[~]# nvmecontrol identify nvme5
nvmecontrol: identify request returned error
I noticed that both and only the two faulty drives are having two memory address blocks. No idea why.
lspci from FreeBSD:
01:00.0 Class 0108: Device 15b7:5009 (rev 01)
02:00.0 Class 0108: Device 1e49:0001 (rev 03)
03:00.0 Class 0108: Device 1e49:0001 (rev 03)
04:00.0 Class 0108: Device 2646:2263 (rev 03)
05:00.0 Class 0108: Device 2646:2263 (rev 03)
06:00.0 Class 0108: Device 15b7:5009 (rev 01)
07:00.0 Class 0108: Device 1e49:0001 (rev 03)
15b7:5009, according to Google, is WD Blue SN550. They correspond to nvme0 and nvme5.
I am using Proxmox 6.3, which is their latest release, along with TrueNAS-12.0-U2.1. Both are installed freshly.
Hardware:
- CPU: AMD Threadripper 3960X
- Mobo: Gigabyte Aorus TRX40 Xtreme
- RAM: 128GB DDR4, out of which 32GB are dedicated to TrueNAS.
Notice that there is some timeout bug in Proxmox when dedicating too much RAM to a VM. I googled it, turned off memory balloon, and used huge page of size 2M.
On the host. I have 4 M.2 interfaces from the mobo, and additional 4 from a PCI-E x16 adapter card. Both SN550 are inside the adapter card. On the host Linux, I have the following lspci output:
01:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)
02:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)
21:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
22:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
23:00.0 Non-Volatile memory controller: Device 1e49:0001 (rev 03)
24:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
43:00.0 Non-Volatile memory controller: Device 1e49:0001 (rev 03)
44:00.0 Non-Volatile memory controller: Device 1e49:0001 (rev 03)
The middle four are from the PCI-E adapter card. 22 is for the host. 23, 43, 44 are from Zhitai and correctly identified. 1 and 2 are from Kingston and correctly identified. Since 23 is identified, I doubt the issue lies in the adapter card. Plus everything is virtualized anyway.
All the seven drives use the vfio-pci driver in Linux.
I have googled around and found no useful info on SN550's compatibility with FreeBSD. In fact, some people in this community mentioned this particular drive in their posts. Still, I wonder if there is something wrong with WD NVMe drives. Does anybody have any clue about that?
Many thanks!
root@truenas[~]# geom disk list
Geom name: vtbd0
Providers:
1. Name: vtbd0
Mediasize: 34359738368 (32G)
Sectorsize: 512
Mode: r1w1e2
descr: (null)
ident: (null)
rotationrate: unknown
fwsectors: 63
fwheads: 16
Geom name: nvd0
Providers:
1. Name: nvd0
Mediasize: 1024209543168 (954G)
Sectorsize: 512
Mode: r0w0e0
descr: ZHITAI PC005 Active 1TB
lunid: a428b70016ba0042
ident: ZTA11T0JA2050000EE
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd1
Providers:
1. Name: nvd1
Mediasize: 1024209543168 (954G)
Sectorsize: 512
Mode: r0w0e0
descr: ZHITAI PC005 Active 1TB
lunid: a428b70015550042
ident: ZTA11T0JA205000123
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd2
Providers:
1. Name: nvd2
Mediasize: 1000204886016 (932G)
Sectorsize: 512
Mode: r0w0e0
descr: KINGSTON SA2000M81000G
lunid: 0026b76843b0e425
ident: 50026B76843B0E42
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd3
Providers:
1. Name: nvd3
Mediasize: 1000204886016 (932G)
Sectorsize: 512
Mode: r0w0e0
descr: KINGSTON SA2000M81000G
lunid: 0026b76843b07be5
ident: 50026B76843B07BE
rotationrate: 0
fwsectors: 0
fwheads: 0
Geom name: nvd4
Providers:
1. Name: nvd4
Mediasize: 1024209543168 (954G)
Sectorsize: 512
Mode: r0w0e0
descr: ZHITAI PC005 Active 1TB
lunid: a428b70016b90042
ident: ZTA11T0JA2050000ED
rotationrate: 0
fwsectors: 0
fwheads: 0
All seven devices are successfully passed inside the guest. Outputs from dmesg indicates that all of them are recognized as NVMe devices:
root@truenas[~]# dmesg | grep nvme
nvme0: <Generic NVMe Device> mem 0xfe800000-0xfe803fff,0xfe804000-0xfe8040ff irq 20 at device 0.0 on pci1
nvme1: <Generic NVMe Device> mem 0xfe600000-0xfe603fff irq 20 at device 0.0 on pci2
nvme2: <Generic NVMe Device> mem 0xfe400000-0xfe403fff irq 20 at device 0.0 on pci3
nvme3: <Generic NVMe Device> mem 0xfe200000-0xfe203fff irq 16 at device 0.0 on pci4
nvme4: <Generic NVMe Device> mem 0xfe000000-0xfe003fff irq 16 at device 0.0 on pci5
nvme4: <Generic NVMe Device> mem 0xfe000000-0xfe003fff irq 16 at device 0.0 on pci5
nvme5: <Generic NVMe Device> mem 0xfde00000-0xfde03fff,0xfde04000-0xfde040ff irq 16 at device 0.0 on pci6
nvme6: <Generic NVMe Device> mem 0xfdc00000-0xfdc03fff irq 16 at device 0.0 on pci7
When I tried to identify them, I got some errors:
root@truenas[~]# nvmecontrol identify nvme0
nvmecontrol: identify request returned error
root@truenas[~]# nvmecontrol identify nvme5
nvmecontrol: identify request returned error
I noticed that both and only the two faulty drives are having two memory address blocks. No idea why.
lspci from FreeBSD:
01:00.0 Class 0108: Device 15b7:5009 (rev 01)
02:00.0 Class 0108: Device 1e49:0001 (rev 03)
03:00.0 Class 0108: Device 1e49:0001 (rev 03)
04:00.0 Class 0108: Device 2646:2263 (rev 03)
05:00.0 Class 0108: Device 2646:2263 (rev 03)
06:00.0 Class 0108: Device 15b7:5009 (rev 01)
07:00.0 Class 0108: Device 1e49:0001 (rev 03)
15b7:5009, according to Google, is WD Blue SN550. They correspond to nvme0 and nvme5.
I am using Proxmox 6.3, which is their latest release, along with TrueNAS-12.0-U2.1. Both are installed freshly.
Hardware:
- CPU: AMD Threadripper 3960X
- Mobo: Gigabyte Aorus TRX40 Xtreme
- RAM: 128GB DDR4, out of which 32GB are dedicated to TrueNAS.
Notice that there is some timeout bug in Proxmox when dedicating too much RAM to a VM. I googled it, turned off memory balloon, and used huge page of size 2M.
On the host. I have 4 M.2 interfaces from the mobo, and additional 4 from a PCI-E x16 adapter card. Both SN550 are inside the adapter card. On the host Linux, I have the following lspci output:
01:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)
02:00.0 Non-Volatile memory controller: Kingston Technologies Device 2263 (rev 03)
21:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
22:00.0 Non-Volatile memory controller: Samsung Electronics Co Ltd NVMe SSD Controller SM981/PM981
23:00.0 Non-Volatile memory controller: Device 1e49:0001 (rev 03)
24:00.0 Non-Volatile memory controller: Sandisk Corp Device 5009 (rev 01)
43:00.0 Non-Volatile memory controller: Device 1e49:0001 (rev 03)
44:00.0 Non-Volatile memory controller: Device 1e49:0001 (rev 03)
The middle four are from the PCI-E adapter card. 22 is for the host. 23, 43, 44 are from Zhitai and correctly identified. 1 and 2 are from Kingston and correctly identified. Since 23 is identified, I doubt the issue lies in the adapter card. Plus everything is virtualized anyway.
All the seven drives use the vfio-pci driver in Linux.
I have googled around and found no useful info on SN550's compatibility with FreeBSD. In fact, some people in this community mentioned this particular drive in their posts. Still, I wonder if there is something wrong with WD NVMe drives. Does anybody have any clue about that?
Many thanks!