Counterintuitive Heads-Up on Asus Hyper M.2 Tuning

ISJ

Dabbler
Joined
Aug 4, 2019
Messages
45
I just wanted to give everyone a heads up, since dropping ~$4500 on this setup and getting poor results sucks.

TL;DR: While ASUS Hyper M.2 Cards state they are PCIE 4.0 compliant, in my setup dropping them to PCIE 3.0 removed hardware errors present in other users and also improved performance 2x strangely.

Test Results
Situation
Read (MB/s)
Read IOPS
Write (MB/s)
Write IOPS
Eight drives, no lvm, Encryption, RW, 64K chunk, PCIE 4​
4923​
75181​
4921​
75153​
Eight drives, no lvm, Encryption, RW, 64K chunk, PCIE 3​
7859​
120101​
7856​
120044​

The only thing that changed was BIOS setting the two slots to PCIE 3.0, the cooling was massive for this test by use of two 60W 120MM server fans to eliminate any throttling, so it's not cooling that's causing this. Both tests were just after bootup.

My Setup
8x Samsung 980 Pros (2TB)
2x ASUS Hyper M.2 Cards
Asus WRX80E-SAGE SE Wifi
OS: Debian 11.7 (hardware testing to follow with TrueNAS)
CPU: AMD Ryzen Threadripper PRO 5955WX
RAM: 128GB DDR4 3200MHZ

All drive placed in a MDADM array with the following command:
sudo mdadm --create /dev/md0 --level=0 --raid-devices=8 --chunk=64K /dev/nvme0n1 /dev/nvme1n1 /dev/nvme2n1 /dev/nvme3n1 /dev/nvme5n1 /dev/nvme6n1 /dev/nvme7n1 /dev/nvme8n1

Tested with:
sudo fio --ioengine=io_uring --direct=1 --name=fiotest --filename=./testfio --bs=64k --iodepth=256 --size=128GB --rw=randrw --runtime=60 --time_based --group_reporting --eta-newline=1​
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Moderator note: These are the TrueNAS Community Forums, a support forum for users of the TrueNAS storage appliance. You appear to be using random generic Linux (mdadm, wifi, Debian, nonrecommended hardware, etc) and as such, this does not belong in the TrueNAS sections of these forums. You are welcome to discuss this kind of thing in the Off-Topic section of the forums, or on any of the dozens of other forum sites dedicated to Linux. Your post has been moved to Off-Topic and a reminder to participants that the above is not recommended or supported; you should not be using mdadm for building metadevices or filesystems in TrueNAS.
 

NickF

Guru
Joined
Jun 12, 2014
Messages
760
FWIW if anyone finds this later.

This is why PCI-E Re-timers exist. The card in question for OP will probably work at PCI-E 4 speeds if it's in the slot closest to the CPU where you would typically put a GPU. The length of the traces on the motherboard plus the length of the traces this card added for him put it out of spec. A re-timer is kinda like a WiFi repeater but less crummy and makes this problem less of an issue.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,176
Sidenote: There are four tiers of NVMe adapters:
  1. Passive
  2. Redriver - basically redrivers just amplify the signal, extending it a bit
  3. Retimers - these reconstruct the clocks and retransmit the PCIe signals "restarting" the length limit.
  4. PCIe switches - besides retiming the signal, they can switch PCIe traffic, bypassing the need for bifurcation and/or allowing for additional devices, in a similar vein to a SAS expander.
 

ISJ

Dabbler
Joined
Aug 4, 2019
Messages
45
Moderator note: You appear to be using random generic Linux (mdadm, wifi, Debian, nonrecommended hardware, etc) and as such, this does not belong in the TrueNAS sections of these forums.

Sorry, thanks for moving it sir.
 

ISJ

Dabbler
Joined
Aug 4, 2019
Messages
45
FWIW if anyone finds this later.

This is why PCI-E Re-timers exist. The card in question for OP will probably work at PCI-E 4 speeds if it's in the slot closest to the CPU where you would typically put a GPU. The length of the traces on the motherboard plus the length of the traces this card added for him put it out of spec. A re-timer is kinda like a WiFi repeater but less crummy and makes this problem less of an issue.
This is an excellent observation that's likely true. I had to use slots 6 and 7 at the bottom of the board just due to the fact that it wont physically fit anywhere else.
 
Top