Slow Raid-Z2 speeds over 10g

Frikkie

Dabbler
Joined
Mar 10, 2019
Messages
41
Hello there!

At the moment I am witnessing around half the expected read speeds over SMB (170-230MB/s) when taking my setup into account. I cannot for the life of me figure out where exactly my problem lies and need some help working this out. I did not post this thread in the "Networking" section of the Forums due to the results of my testing so far and additional suspicions I have. Bear in mind that this is ONLY about the slow read speeds from my FreeNAS box. Write speeds over SMB are basically maxing out the wd red raidz2 pool performance capability at 430-450MB/s. I know not to expect 1.1GB/s unless both ends have insanely fast NVMe drives.
Alright, here goes...let me try to give you as much background as possible. :)


FreeNAS PC specs:
Mobo: EVGA X58 FTW3
CPU: E5645 (6 cores, 12 threads)
Memory: 24GB DDR3 ECC 1600Mhz (6x4GB)
HBA: LSI SAS 9207-8i IT Mode LSI00301
HBA Cables: 2x Mini SAS Cable SFF 8087 to 4 SATA 7pin
10G NIC: Mellanox ConnectX-3 Single Port
10G Cable: 10Gtek 10Gb/s SFP+ Cable 1m Direct Attach Twinax Passive DAC
Boot Drive: Intel 40GB Sata 2
Data Drives: 6x3TB WD Red in Raid-Z2

Main PC specs:
Mobo: MSI B350M Mortar
CPU: Ryzen 5 1600x
Memory: 16GB DDR4 3200Mhz (2x8GB)
10G NIC: Mellanox ConnectX-3 Single Port
10G Cable: 10Gtek 10Gb/s SFP+ Cable 5m Direct Attach Twinax Passive DAC
Boot & Data Drive: Samsung 850 pro 256GB

10G Network Switch:
MikroTik CRS305-1G-4S+IN

Detailed Phyiscal Disk Info:
6x3TB WD Red Drives:
Example drive:
1570964330673.png

Note that all the drives have the exact specs as in the screenshot above yet the device model numbers and firmware revisions do differ slightly.
2x Model WDC WD30EFRX-68AX9N0, Firmware 80.00A80
2x Model WDC WD30EFRX-68N32N0, Firmware 82.00A82
2x Model WDC WD30EFRX-68EUZN0, Firmware 82.00A82


Motherboard:
PCI-E Slot 1: HBA
PCI-E Slot 2: 10G NIC
PCI-E Slot 3: GPU

According to this section from the motherboard manual I should theoretically not run into any PCI-E bandwidth limitations:
1570964738955.png


FreeNAS SMB:
1570964853751.png


The Auxiliary Parameters I have set helped fix a weird bug where opening a windows network share, that has a boat load of files/folders and/or big files (such as my movies folder), on a windows client in windows explorer would result in the folder taking ages to "load".
The SMB service is solely bound to the IP address of the 10G NIC. The second FreeNAS NIC is used for a couple of jails and web gui access.

FreeNAS 10G NIC:
Options: mtu 9000

Main PC 10G NIC:
Jumbo Packet Value: 9014

Absolutely no idea what these different modes are for which can be found in the Mellanox ConnectX-3 Etherenet Adapter Properties. The descriptions for each mode use the words they are supposed to be describing (derp).
1570965534163.png


Mikrotik Switch:
Not 100% sure about the difference between "Actual MTU" and "L2 MTU" and if my settings are correct but the displayed relative RX and TX speeds during an iperf test lead me to believe this is not where my issue lies.
1570965271239.png

The only other settings I have changed for both sfp+ ports (one connected to main PC, one to FreeNAS PC) within the Mikrotik web gui is disabling auto negotiation, forcing 10G speed and enabling "full duplex".


Testing (all disk speed tests were run on a dataset with compression turned OFF and where possible the file size was set to larger than 24GB due to RAM caching!):
iperf:
Sender (server) = FreeNAS
1570965608187.png


Receiver (client) = FreeNAS
1570965634398.png


Results seem solid enough to me.

Example for CPU usage during a CrystalDiskMark test:
1570965979934.png


No CPU bottlenecking especially because drive encryption is not enabled.

DD tests:
I used block size = 4096k and count = 10000. This should result in a 40GB file being used?
1570968398419.png


using "zdb -U /data/zfs/zpool.cache MyPoolHere | grep ashift" gave me an ashift value of "12" which should correspond with the 4k physical sector size of my drives?

Questions:
1.) Is a single drive starting to fail/slow down, which in turn causes the entire pool to slow down? All S.M.A.R.T. and scrub tests, which I have setup to run periodically always return without any errors. I do not know of a way to test single drive performance if it is in a pool and you do not want to destroy said pool.

2.) Do the different drive firmware revisions have any impact on performance?

3.) Is SMB tuning needed either on the FreeNAS or main PC side?

4.) Could removing the GPU completely from the system and having the HBA and 10G cards in slots 1 and 3 (→PCI-E 2.0 x16, x16) be my first troubleshooting step?

5.) Is having more than 80% (82%) used pool storage of any consequence? I thought that system warning above 80% was for older versions of the FreeNAS OS.

Any help would be greatly appreciated. Thanks in advance!
 
Last edited:

Frikkie

Dabbler
Joined
Mar 10, 2019
Messages
41
Anyone with any ideas?
 

Kuro Houou

Contributor
Joined
Jun 17, 2014
Messages
193
Did you ever solve this? Running into similar issues
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
1.) Is a single drive starting to fail/slow down, which in turn causes the entire pool to slow down? All S.M.A.R.T. and scrub tests, which I have setup to run periodically always return without any errors. I do not know of a way to test single drive performance if it is in a pool and you do not want to destroy said pool.

2.) Do the different drive firmware revisions have any impact on performance?

3.) Is SMB tuning needed either on the FreeNAS or main PC side?

4.) Could removing the GPU completely from the system and having the HBA and 10G cards in slots 1 and 3 (→PCI-E 2.0 x16, x16) be my first troubleshooting step?

5.) Is having more than 80% (82%) used pool storage of any consequence? I thought that system warning above 80% was for older versions of the FreeNAS OS.

1) A single slow drive in a RAIDZ2 will definitely cause the whole pool to drag. You can try using solnet-array-test from here:


As the resources says, this is expected to be safe to run on a live pool - but I would test only one disk at a time out of an abundance of caution. (You have a Z2 - if you somehow roast a disk, you can survive that.)

2) Unless there's a known bug in one of those firmwares (eg: the Ironwolf 10TB issues) then no.

3) Possibly. What's your client OS? I know Windows 10 has a TCP autotuning window that was reported to be troublesome in the past.

4) If your system will boot headless, removing the GPU could be helpful for your power consumption. Assuming you aren't oversubscribed on PCIe lanes though I wouldn't expect it to make much of a speed difference.

5) No, the warning still applies - and to be clear, the "used space" issue is just a window into the larger issue of fragmentation that occurs over time. It's not a magical threshold value, merely a rule of thumb for "when most people will start to notice it." If the files you're trying to read are highly fragmented, reads will be slower, especially on RAIDZ2 where all drives in a vdev need to "work as a group" so to speak.
 
Top