Hi all! Long time lurker, first time poster :)
I was hoping to sanity check my setup, because perplexingly, I'm getting better performance with a virtualized installation than on bare metal (!?)
I'm using six Seagate EXOS 2x14 14TB drives (the ones that have dual actuators - sadly, seems no good way to take advantage of this with the sata variants of this drive, as the NCQ is shared on the sata port and saturates.. but unit testing using lvm & md stripes was entertaining).
The test hardware is a Ryzen 5600X on an AsRock B550M Steel Legend with onboard 500-series 6-port SATA controller. 32GB kingston ECC ram (24GB allocated to ARC).
System image is TrueNAS Scale 23.10.0.1.
For the virtualized test setup, I'm using Proxmox 8.1.3 and presenting 2 cores (kvm64, i440fx, SeaBIOS, VirtIO SCSI), 24GB (18GB ARC). I tested 2 cases - one with sata disks passed thru as block devices, and one with the entire SATA controller passed thru (OS running from NVME, so not a problem).
Here is a snippet of performance testing with single disk, stripe, raidz2 on bare metal and virtualized (and nvme for control). All tests were done using 60GB test files via dd with a 1MB block size. ZFS filesystem with compression confirmed turned OFF. I did some /dev/urandom tests as well as /dev/zero - numbers below are with /dev/zero since urandom was limited by cpu (confirmed with NVME testing).
I ran dozens of more tests, but these are the relevant ones for comparison. You can see single disk performance is comparable both on bare metal and virtualized, 6x stripe testing is 6x faster than single-disk, and raidz2 performance is comparable between bare metal and virtualized w/ block-device passthru, but raidz2 performance is about 75% faster when passing thru the onboard PCIe sata controller directly. I'm perplexed by this, and curious if anyone has any theories, or has seen something similar? I was expecting to see the opposite, naturally.
At first, I thought maybe caching on the Proxmox host, but this should only show up with block device passthru, not with the PCIe sata controller passed through directly to the TrueNAS Scale guest, and I didn't notice the huge difference with block device passthru anyways.
I double checked compression is disabled (identical pool and dataset config), and have confirmed throughput is still very high with random data.
My best guess is perhaps a different code path thru the virtualized bios implementation that alleviates some contention or serialization or something of that nature. Or perhaps the bare metal implementation is underperforming in 6x raidz2 configuration when contrasted with single disk writes (<2x single disk perf bare metal vs 3x single disk perf virtualized).
I was leery of this virtualized configuration, since I'm going to be exporting zVols to sibling VMs, but I've spent days running various tests, including moving random data back and forth from NVME to the spinning disk array, data integrity tests over drive pulls and resilvers, etc. No issues have cropped up.
Head-scratcher...
I was hoping to sanity check my setup, because perplexingly, I'm getting better performance with a virtualized installation than on bare metal (!?)
I'm using six Seagate EXOS 2x14 14TB drives (the ones that have dual actuators - sadly, seems no good way to take advantage of this with the sata variants of this drive, as the NCQ is shared on the sata port and saturates.. but unit testing using lvm & md stripes was entertaining).
The test hardware is a Ryzen 5600X on an AsRock B550M Steel Legend with onboard 500-series 6-port SATA controller. 32GB kingston ECC ram (24GB allocated to ARC).
System image is TrueNAS Scale 23.10.0.1.
For the virtualized test setup, I'm using Proxmox 8.1.3 and presenting 2 cores (kvm64, i440fx, SeaBIOS, VirtIO SCSI), 24GB (18GB ARC). I tested 2 cases - one with sata disks passed thru as block devices, and one with the entire SATA controller passed thru (OS running from NVME, so not a problem).
Here is a snippet of performance testing with single disk, stripe, raidz2 on bare metal and virtualized (and nvme for control). All tests were done using 60GB test files via dd with a 1MB block size. ZFS filesystem with compression confirmed turned OFF. I did some /dev/urandom tests as well as /dev/zero - numbers below are with /dev/zero since urandom was limited by cpu (confirmed with NVME testing).
| Configuration | Write Speed | Read Speed |
|---|---|---|
| Bare metal single disk | 265 MB/s | 264 MB/s |
| Bare metal 6x14tb stripe | 1.5 GB/s | 1.5 GB/s |
| Bare metal 6-disk raidz2 | 444 MB/s | 1.0 GB/s |
| Bare metal EVO 980 Pro single 1TB NVME | 2.6 GB/s | 3.9 GB/s |
| Proxmox 6-disk raidz2 (block device passthru) | 466 MB/s | 1.0 GB/s |
| Proxmox 6-disk raidz2 (controller passthru) | 809 MB/s (not always this high depending on space usage, but generally 720-800) | 1.0 GB/s |
| Proxmox single disk (controller passthru) | 260 MB/s | 262 MB/s |
| Proxmox EVO 980 Pro single 1TB NVME (block device passthru) | 2.8 GB/s | 3.9 GB/s |
I ran dozens of more tests, but these are the relevant ones for comparison. You can see single disk performance is comparable both on bare metal and virtualized, 6x stripe testing is 6x faster than single-disk, and raidz2 performance is comparable between bare metal and virtualized w/ block-device passthru, but raidz2 performance is about 75% faster when passing thru the onboard PCIe sata controller directly. I'm perplexed by this, and curious if anyone has any theories, or has seen something similar? I was expecting to see the opposite, naturally.
At first, I thought maybe caching on the Proxmox host, but this should only show up with block device passthru, not with the PCIe sata controller passed through directly to the TrueNAS Scale guest, and I didn't notice the huge difference with block device passthru anyways.
I double checked compression is disabled (identical pool and dataset config), and have confirmed throughput is still very high with random data.
My best guess is perhaps a different code path thru the virtualized bios implementation that alleviates some contention or serialization or something of that nature. Or perhaps the bare metal implementation is underperforming in 6x raidz2 configuration when contrasted with single disk writes (<2x single disk perf bare metal vs 3x single disk perf virtualized).
I was leery of this virtualized configuration, since I'm going to be exporting zVols to sibling VMs, but I've spent days running various tests, including moving random data back and forth from NVME to the spinning disk array, data integrity tests over drive pulls and resilvers, etc. No issues have cropped up.
Head-scratcher...
Ryzen 5600X / AsRock B550M Steel Legend / 2x32GB ECC KSM32ED8/32HC.
6x Seagate EXOS 2x14TB raidz2
2x Samsung EVO 980 Pro 1TB NVME mirror
TrueNAS Scale-23.10.0.1 / Proxmox 8.1.3.
6x Seagate EXOS 2x14TB raidz2
2x Samsung EVO 980 Pro 1TB NVME mirror
TrueNAS Scale-23.10.0.1 / Proxmox 8.1.3.
