I just pinned the TrueNAS VM CPUs to the socket that serves /dev/nvd1;
Code:
truenas# diskinfo -wS /dev/nvd1
/dev/nvd1
512 # sectorsize
29260513280 # mediasize in bytes (27G)
57149440 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL MEMPEK1J032GA # Disk descr.
PHBT809300ZU032P # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Synchronous random writes:
0.5 kbytes: 28.9 usec/IO = 16.9 Mbytes/s
1 kbytes: 25.9 usec/IO = 37.7 Mbytes/s
2 kbytes: 27.7 usec/IO = 70.5 Mbytes/s
4 kbytes: 38.6 usec/IO = 101.3 Mbytes/s
8 kbytes: 57.7 usec/IO = 135.3 Mbytes/s
16 kbytes: 126.0 usec/IO = 124.0 Mbytes/s
32 kbytes: 164.6 usec/IO = 189.9 Mbytes/s
64 kbytes: 254.7 usec/IO = 245.4 Mbytes/s
128 kbytes: 443.8 usec/IO = 281.7 Mbytes/s
256 kbytes: 809.4 usec/IO = 308.9 Mbytes/s
512 kbytes: 1538.2 usec/IO = 325.0 Mbytes/s
1024 kbytes: 3026.1 usec/IO = 330.5 Mbytes/s
2048 kbytes: 5986.5 usec/IO = 334.1 Mbytes/s
4096 kbytes: 11881.6 usec/IO = 336.7 Mbytes/s
8192 kbytes: 23689.6 usec/IO = 337.7 Mbytes/s
So bandwidth went up considerably for 0.5KiB < XFer < 16KiB. No luck yet on the latency, but will reboot with different PCIe BIOS setting and report back.
+++++ EDIT +++++
BIOS settings made no discernable difference
Making the *correct* BIOS change did make a difference (thanks
@HoneyBadger);
Code:
truenas# diskinfo -wS /dev/nvd1
/dev/nvd1
512 # sectorsize
29260513280 # mediasize in bytes (27G)
57149440 # mediasize in sectors
0 # stripesize
0 # stripeoffset
INTEL MEMPEK1J032GA # Disk descr.
PHBT809300ZU032P # Disk ident.
Yes # TRIM/UNMAP support
0 # Rotation rate in RPM
Synchronous random writes:
0.5 kbytes: 24.4 usec/IO = 20.0 Mbytes/s
1 kbytes: 27.2 usec/IO = 36.0 Mbytes/s
2 kbytes: 27.7 usec/IO = 70.5 Mbytes/s
4 kbytes: 35.0 usec/IO = 111.5 Mbytes/s
8 kbytes: 45.9 usec/IO = 170.4 Mbytes/s
16 kbytes: 75.6 usec/IO = 206.8 Mbytes/s
32 kbytes: 128.6 usec/IO = 242.9 Mbytes/s
64 kbytes: 219.7 usec/IO = 284.4 Mbytes/s
128 kbytes: 406.1 usec/IO = 307.8 Mbytes/s
256 kbytes: 772.0 usec/IO = 323.8 Mbytes/s
512 kbytes: 1494.6 usec/IO = 334.5 Mbytes/s
1024 kbytes: 2953.0 usec/IO = 338.6 Mbytes/s
2048 kbytes: 5860.1 usec/IO = 341.3 Mbytes/s
4096 kbytes: 11738.1 usec/IO = 340.8 Mbytes/s
8192 kbytes: 23331.2 usec/IO = 342.9 Mbytes/s
Boy did that help! Will now need to sift through ESXi settings to see which might help further.
+++++ /EDIT +++++
+++++ EDIT 2 +++++
Setting BIOS to "OS Control Mode" power regulator allows for these things to be adjusted on the fly in ESXi. The following settings in ESXi 7.0U2, within a custom power mgmt profile, come closest to setting the all-out performance profile in BIOS;
Code:
Power.PerfBiasEnable=0
Power.TimerHz=1000
Power.UseCStates=0
Power.UsePStates=0
With this it's then easy to switch between a power-saving profile and the custom profile on-the-fly (even while the VMs are powered on).
+++++ /EDIT 2 +++++