We are in the process of rolling out a new storage server and have started to do some baseline performance testing and are seeing surprisingly poor performance. In our test we have provisioned a VM within Proxmox that is using a Zvol attached to the cluster via iSCSI. The TrueNAS server is connected to a 25GbE switch with 2 links that are aggregated and the Proxmox host is connected to the same switch but with a single 25GbE interface. The TrueNAS server pool that is backing the zvol has 10 - 7.68TB NVMe SSD's and when we run fio on the storage server directly we are seeing the following:
When we run the same command within the VM that is using the iSCSI disk we get the following results:
We have updated the MTU within the network stack to 9000 but to be honest I am not sure that is in affect and will be looking into it further.
Any advice on where to go next to try to figure why it is performing so poorly would be much appreciated.
Storage server specs:
TrueNAS Core 13.0-U5.3
AMD EPYC 9124
RAM 393204MB ECC
Micron 7450 x10 NVMe SSD
MT28800 NIC
Compute server specs:
Proxmox 8.0.4
AMD EPYC 7543P
RAM 500GB DDR4-4300 ECC
MT28800 NIC
Switch:
QSW-M5216-1T
fio --bs=128k --direct=1 --directory=/mnt/n-store/proxmox/nfs-share-01 --gtod_reduce=1 --ioengine=posixaio --iodepth=32 --group_reporting --name=randrw --numjobs=2 --ramp_time=10 --runtime=60 --rw=randrw --size=256M --time_based
READ: bw=8348MiB/s (8754MB/s), 8348MiB/s-8348MiB/s (8754MB/s-8754MB/s), io=489GiB (525GB), run=60001-60001msec
WRITE: bw=8344MiB/s (8749MB/s), 8344MiB/s-8344MiB/s (8749MB/s-8749MB/s), io=489GiB (525GB), run=60001-60001msec
When we run the same command within the VM that is using the iSCSI disk we get the following results:
READ: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s), io=12.9GiB (13.9GB), run=60036-60036msec
WRITE: bw=220MiB/s (231MB/s), 220MiB/s-220MiB/s (231MB/s-231MB/s), io=12.9GiB (13.9GB), run=60036-60036msec
We have updated the MTU within the network stack to 9000 but to be honest I am not sure that is in affect and will be looking into it further.
Any advice on where to go next to try to figure why it is performing so poorly would be much appreciated.
Storage server specs:
TrueNAS Core 13.0-U5.3
AMD EPYC 9124
RAM 393204MB ECC
Micron 7450 x10 NVMe SSD
MT28800 NIC
Compute server specs:
Proxmox 8.0.4
AMD EPYC 7543P
RAM 500GB DDR4-4300 ECC
MT28800 NIC
Switch:
QSW-M5216-1T