eclipse5302
Dabbler
- Joined
- Nov 14, 2022
- Messages
- 11
Hello all,
I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon 2690s, and I believe a Dell H310 flashed to IT mode. Not 100% on that last part, but it definitely has a regular HBA and not a RAID card pretending to be a HBA.
I'm upgrading this to a Dell R730xd with 384GB of memory, 2 Xeon 2667s, a Dell HBA330, and (24) 400GB SAS flash drives. Running the same fio command, I can't get this system to exceed 150k IOPS no matter what. I started by creating a 12-vdev flash pool, and when that performance was poor, I re-created the same 4-vdev flash pool as the old server. Even a single vdev flash pool can only hit 150k IOPS. All BIOS/FW is up to date, and I tried different versions of TrueNAS without success. I tried other drives, and they too are limited by this 150k IOPS limit. I even took 2 of these drives out and put them in the R720, created a single mirror pool, and that could easily hit 220k IOPS.
Any idea what is going on here?
I've looked around and tried everything I can think of, but I can't figure this out. I have an R720 with a 4-vdev mirror flash pool (8 SATA drives 400GB), and running fio on the hoist itself shows expected results of 500k IOPS and 1300MB/s. This host only has 64GB of memory, 2 Xeon 2690s, and I believe a Dell H310 flashed to IT mode. Not 100% on that last part, but it definitely has a regular HBA and not a RAID card pretending to be a HBA.
fio --filename=test --direct=1 --rw=randrw --randrepeat=0 --rwmixread=100 --iodepth=128 --numjobs=8 --runtime=60 --group_reporting --name=4ktest --ioengine=psync --size=4G --bs=4k
4ktest: (g=0): rw=rw, bs=(R) 4096B-4096B, (W) 4096B-4096B, (T) 4096B-4096B, ioengine=psync, iodepth=128
I'm upgrading this to a Dell R730xd with 384GB of memory, 2 Xeon 2667s, a Dell HBA330, and (24) 400GB SAS flash drives. Running the same fio command, I can't get this system to exceed 150k IOPS no matter what. I started by creating a 12-vdev flash pool, and when that performance was poor, I re-created the same 4-vdev flash pool as the old server. Even a single vdev flash pool can only hit 150k IOPS. All BIOS/FW is up to date, and I tried different versions of TrueNAS without success. I tried other drives, and they too are limited by this 150k IOPS limit. I even took 2 of these drives out and put them in the R720, created a single mirror pool, and that could easily hit 220k IOPS.
Any idea what is going on here?