I have TrueNAS Core 13.0-U5.1 running on a Dell R730xd (2 x Xeon(R) CPU E5-2623 v3, 128GB of DDR4 RAM).
Using a PCIe to NVMe expansion card, I configured 4 x 1TB NVMe drives in a striped mirror (RAID10 equiv).
I have TrueNAS setup for management on a 10Gb connection, iSCSI on its own 40Gb connection, and SAMBA/NFS on its own 40Gb connection.
I have an XCP-NG hypervisor host setup on another R730. I have a Mellanox ConnectX-3 card flashed for 40GbE directly connected to TrueNAS via a Mellanox QSFP+ DAC.
I have an XCP-NG hypervisor (8.2 or whatever is latest, I can't remember). It is setup with management on a 10Gb NIC, and two 10Gb NICs setup for multipathing for iSCSI.
I have a separate spin pool. From my desktop with 10Gb NICs, I can nearly max out my connection when pulling or pushing a large ISO file.
The troubleshooting:
Is there something I can do to increase iSCSI performance in TrueNAS for hypervisor platforms (specifically XCP-NG)?
UPDATE: See post #5 below. I've installed a 40Gb card to my XCP-NG host and directly connected it to the 40Gb card on my TrueNAS server and somehow get worse performance?
Using a PCIe to NVMe expansion card, I configured 4 x 1TB NVMe drives in a striped mirror (RAID10 equiv).
I have TrueNAS setup for management on a 10Gb connection, iSCSI on its own 40Gb connection, and SAMBA/NFS on its own 40Gb connection.
I have an XCP-NG hypervisor host setup on another R730. I have a Mellanox ConnectX-3 card flashed for 40GbE directly connected to TrueNAS via a Mellanox QSFP+ DAC.
I have a separate spin pool. From my desktop with 10Gb NICs, I can nearly max out my connection when pulling or pushing a large ISO file.
The troubleshooting:
I have a Windows Server 2022 VM with its storage running on the flash-based iSCSI previously mentioned. If I try and pull/push a large ISO file to the same TrueNAS share, it maxes out at ~120MB/s.I've run this same test with the VM's storage on an NFS share on the same flash, iSCSI on the spin pool, and NFS on the spin pool. EXACT same performance.When migrating the storage BACK to the flash-based iSCSI, I saw the host hit ~325MB/s.I created a SAMBA share off the flash storage and was able to hit 600MB/s pushing and 1GB/s pulling.
UPDATE: See post #5 below. I've installed a 40Gb card to my XCP-NG host and directly connected it to the 40Gb card on my TrueNAS server and somehow get worse performance?
Last edited: