SOLVED Slow Windows R/W Performance over 10GbE with my TrueNAS box

[rathijit]

Greetings!

I am quite new to this forum, and new to TrueNAS (moved from unRAID to gain better read performance). I am trying to decide the best way to setup my ZFS pool. I want the reliability of RAIDZ2, and at minimum 300MB/s read performance from my pool. Currently I get about 480MB/s write speed and 180MB/s read speed out of my 10GbE NIC from another PC connected directly via a 10GbE direct attach copper cable (measured by transferring single large files). Is this typical for my hardware and configuration shown below? Can anyone suggest improvements to the setup get better read performance beyond the current 180MB/s? I don't mind destroying my pool and rebuilding it better, and I don't mind reconfiguring the way the drives are connected to the HBAs.

Here are my hardware specs:

• Motherboard: Supermicro X9SCL (six SATA II on-board ports, plus two x8 PCIe 3.0 and one x4 in x8 PCIe 2.0 expansion slots)
• Processor: Xeon E3-1270v2
• HBA1: LSI 9207 8i (in IT mode) on the first x8 PCIe 3.0 slot
• HBA2: LSI 9201 8i (in IT mode) on the second x8 PCIe 3.0 slot
• 10GbE NIC: Intel X520-DA2 on x4 PCIe 2.0 slot (limited bandwidth of x4 lane)
• RAM: 16GB UDIMM ECC
• Hard Drives:
  • 12x HGST 4TB 7200RPM 64MB Cache SATA 6Gb/s 3.5" Internal Hard Drive
  • 2x Samsung EVO 870 1TB drives for Cache
  • 1x SSD 120GB for the boot pool connected via USB 2.0 internal port on motherboard

Current Pool consists of 2 VDEVs and cache drives as shown below:

• RAIDZ2
  • HBA2 Port 1: 4TB
  • HBA2 Port 2: 4TB
  • HBA2 Port 3: 4TB
  • HBA2 Port 4: 4TB
  • HBA2 Port 5: 4TB
  • HBA2 Port 6: 4TB
• RAIDZ2
  • HBA2 Port 7: 4TB
  • HBA2 Port 8: 4TB
  • HBA1 Port 1: 4TB
  • HBA1 Port 2: 4TB
  • HBA1 Port 3: 4TB
  • HBA1 Port 4: 4TB
• Cache:
  • HBA1 Port 5: 1TB SSD
  • HBA1 Port 6: 1TB SSD

 

[NugentS]

What do you use the system for?
What sort of files
How many users etc

Your cache drives probably aren't doing much of any use and may in fact be slowing things down as they use quite a lot of your already low memory to actually do things.

The vdevs, are they different pools (probably not), mirrored or striped? This will make a lot of difference

The 10Gb card is likely to be bottlenecked if you are using both ports - a single port is fine.

 

[rathijit]

What do you use the system for?
What sort of files
How many users etc

Your cache drives probably aren't doing much of any use and may in fact be slowing things down as they use quite a lot of your already low memory to actually do things.

The vdevs, are they different pools (probably not), mirrored or striped? This will make a lot of difference

The 10Gb card is likely to be bottlenecked if you are using both ports - a single port is fine.

This is for home use with 1 to 2 users at most. Mostly large media files and some smaller jpg, mp3, etc. Mostly mixed media.
The VDEVs are striped for a total of 28TB in a single pool. I need at least 24+TB.

The 10GbE NIC only has 1 port connected, and since I get 480MB/s write speeds, I guess it is not a bottleneck for the 180MB/s reads.

 

[Spearfoot]

Agree that you probably aren't gaining anything from the cache disks.

Your 10G NIC is constrained by running in an x4 slot, but these tunables might still help:

 

[rathijit]

Agree that you probably aren't gaining anything from the cache disks.

Your 10G NIC is constrained by running in an x4 slot, but these tunables might still help:
View attachment 49489

I thought a single x4 channel will be sufficient as long as I am using only one of the ports. In fact, I am able to get 500MB/s read and write out of the NIC on my previous unRAID SSD cache drive, so I don't think it is the NIC. I believe it is constrained by the ZFS pool setup I currently have, or the way I have them connected to the 2 HBAs.

Have you seen better read performance from 2x striped RAIDZ2 VDEVs with 6 drives in each VDEV?

 

[NugentS]

A single Port on the NIC is fine according to my maths / Wikipedia
The cache won't be helping - but otherwise all seems fine
The read speed isn't ideal and other than the cache - which is wasted, and possibly hindering I don't see anything that you are doing wrong. I would have hoped for twice that as read speed (ish)

 

[Spearfoot]

A single Port on the NIC is fine according to my maths / Wikipedia
The cache won't be helping - but otherwise all seems fine
The read speed isn't ideal and other than the cache - which is wasted, and possibly hindering I don't see anything that you are doing wrong. I would have hoped for twice that as read speed (ish)

Agree that an x4 slot should work fine for a single connection. But I've found that the tunables above give me line-rate performance where the defaults do not. I'm thinking it would be worth the effort to ensure the NIC isn't the bottleneck.

@rathijit : what results do you get running iperf?

 

[rathijit]

Agree that an x4 slot should work fine for a single connection. But I've found that the tunables above give me line-rate performance where the defaults do not. I'm thinking it would be worth the effort to ensure the NIC isn't the bottleneck.

@rathijit : what results do you get running iperf?

I get about 3.75Gbits/s (lower than 10GbE speeds, but expected out of an x4 PCIe 2.0 slot), which confirms why I see about 480MBytes/s writes. However, I don't understand why the read performance is around 180MBytes/s.

root@Jupiter[~]# iperf -s
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 10.0.0.2 port 5001 connected with 10.0.0.75 port 52429
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 4.36 GBytes 3.74 Gbits/sec

 

[Spearfoot]

I get about 3.75Gbits/s (lower than 10GbE speeds, but expected out of an x4 PCIe 2.0 slot), which confirms why I see about 480MBytes/s writes. However, I don't understand why the read performance is around 180MBytes/s.

Usually disk I/O is the bottleneck after you've stepped up to 10G networking. And the mismatch you're seeing does seem odd.

Still, I think my tuning suggestions will improve your network throughput. As an experiment, last night on one of my systems I put the 10G dual-port NIC into a PCIe v2.0 x4 slot and tested with iperf: I got > 9.8Gbps transfer rates, using the tunables I showed above.

I'm also using jumbo frames, which may or may not be feasible for your situation.

 

[rathijit]

Usually disk I/O is the bottleneck after you've stepped up to 10G networking. And the mismatch you're seeing does seem odd.

Still, I think my tuning suggestions will improve your network throughput. As an experiment, last night on one of my systems I put the 10G dual-port NIC into a PCIe v2.0 x4 slot and tested with iperf: I got > 9.8Gbps transfer rates, using the tunables I showed above.

I'm also using jumbo frames, which may or may not be feasible for your situation.

Ok, so I finally got around to applying the tunables on my NAS and rebooted the system. Sadly it did not make a difference. In fact, write speeds dropped from 480 to 300MB/s.

Next, I moved my NIC to one of the x8 PCIe 3.0 slots. So now the NIC has its own x8 slot, and the LSI 9207 8i has its own slot, and the remaining HDDs are connected to the motherboard directly (no HBA2). With this config also, I did not see any improvement.

Another thing I tried was using dd to check the disk write speed, and I get pretty good speeds this way (don't know if this indicates anything):

Code:

dd if=/dev/zero of=./large bs=2048 count=1m
1048576+0 records in
1048576+0 records out
2147483648 bytes transferred in 4.785794 secs (448720496 bytes/sec)

Please help! I am at a loss as to what is causing the 180MB/s read performance. Writes are steady at around 300MB/s.

 

[rathijit]

I have more to add:

I ran iperf both ways (PC --> NAS and NAS --> PC) and found that in the PC (client) --> NAS (server) test, I get about 3GB/s link speed, but on the NAS (client) --> PC (server) test, I get about 1.4GB/s link speed. The 1.4GB/s confirms why I am seeing 180Mbytes/s read speeds! How do I make TrueNAS run the NIC at full speed? Why is my transfer speed so low?

Moreover, my Windows PC used for the test claims to be running the NIC at full capabilities, as you can see in the screenshot below (highlighted line):

 

[Elliot Dierksen]

How do I make TrueNAS run the NIC at full speed? Why is my transfer speed so low?

With some additional flags (which escape me at the moment), you can get iperf to tell you about things like MSS and (more importantly) the TCP window size. That is the amount of data that the host will leave unacknowledged. A small TCP window will cause a host to not use all the bandwidth of a link waiting for what has already been sent to be acknowledged. I suspect that is where the issue resides.

 

[Samuel Tai]

I had to read your HW set up quite carefully, and I think your read speeds are being eaten up by PCI contention. Your disks are spread asymmetrically between your 2 HBAs, which have different port counts. Ideally, you don't want to split disks between HBAs, but your 2nd Z2 vdev has disks on both HBAs, whereas your 1st Z2 vdev only has disks on one HBA.

You may be better off removing your cache drives, and then moving the disks in the 2nd vdev that are split on the other HBA onto the slots just vacated by the cache drives. If you still want to retain the cache drives, try plugging them in directly to the motherboard's SATA ports.

 

[rathijit]

With some additional flags (which escape me at the moment), you can get iperf to tell you about things like MSS and (more importantly) the TCP window size. That is the amount of data that the host will leave unacknowledged. A small TCP window will cause a host to not use all the bandwidth of a link waiting for what has already been sent to be acknowledged. I suspect that is where the issue resides.

I see! Will check on this.

I had to read your HW set up quite carefully, and I think your read speeds are being eaten up by PCI contention. Your disks are spread asymmetrically between your 2 HBAs, which have different port counts. Ideally, you don't want to split disks between HBAs, but your 2nd Z2 vdev has disks on both HBAs, whereas your 1st Z2 vdev only has disks on one HBA.

You may be better off removing your cache drives, and then moving the disks in the 2nd vdev that are split on the other HBA onto the slots just vacated by the cache drives. If you still want to retain the cache drives, try plugging them in directly to the motherboard's SATA ports.

I agree about the performance hit splitting the drives between HBAs will cause, so to remove that variable from the analysis, I removed the cache drives and put them in its own striped pool for max throughput. I confirmed disk r/w is great with the dd command, but throughput over the 10GbE card is terrible. I will certainly move my disk config around to optimize it, but I feel like the network may be the bottleneck. You will notice that 2 or 3 posts up, I (possibly) identified this bottleneck: since I have only 1.4GB/s transfer speed in one direction, which matches my observation of about 180Mbyte/s transfer speeds, even though it is a 10GbE NIC.

 

[NugentS]

Can you run some speed tests on the stiped pool of SSD's. If anything should run well its them.
If they run like dogpoo then try them on the other HBA (don't both with the HDD's) and see if that makes any difference.
I think you have a hardware problem and will need to track it down which why I am suggesting just the two SSD's on one HBA, then the other and then on SATA Ports to see if there is any difference.

 

[Spearfoot]

I have more to add:

I ran iperf both ways (PC --> NAS and NAS --> PC) and found that in the PC (client) --> NAS (server) test, I get about 3GB/s link speed, but on the NAS (client) --> PC (server) test, I get about 1.4GB/s link speed. The 1.4GB/s confirms why I am seeing 180Mbytes/s read speeds! How do I make TrueNAS run the NIC at full speed? Why is my transfer speed so low?

View attachment 49675

Moreover, my Windows PC used for the test claims to be running the NIC at full capabilities, as you can see in the screenshot below (highlighted line):

View attachment 49674

Did you include all of the tunables I described above? Especially the congestion protocol (CUBIC)? Your network seems to be your bottleneck at this point, so that's the first thing you need to get squared away.

Have you optimized the network settings on your Windows PC as well? This is very NIC-specific, but I always make sure to maximize buffer sizes, set up RSS, etc. I don't know what version of Windows you're using, but here is a script I use to set up Windows 7 w/ an Intel X520-1 NIC:

Code:

netsh int tcp set global autotuninglevel=normal
netsh int tcp set global congestionprovider=ctcp
netsh int tcp set global rss=enabled
netsh int tcp set global chimney=enabled
netsh int tcp set global dca=enabled

And I set these registry values too:

Code:

[HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\services\Tcpip\Parameters]
"GlobalMaxTcpWindowSize"=dword:00800000
"TcpWindowSize"=dword:00800000

This all gives me a 4M TCP window size. I also use jumbo frames; if you're using a switch, you'll have to configure it to support those -- assuming you want to use them.

With these settings I get 9.87Gbps from Windows->FreeNAS and 9.29Gbps from FreeNAS->Windows.

It's not supposed to matter how disks are connected... but @Samuel Tai may have a point about your HDD setup suffering from PCI contention. At the very least, I would remove the two cache SSDs until you get a basic configuration working.

 

[rathijit]

Ok, to rule out possible hardware issues, I did the following:

Used a completely different motherboard and architecture (AMD instead of Intel).

Disconnected everything from the motherboard except for the bare essentials: RAM, Processor, GPU, 10GbE NIC, 1x SSD.

I created a pool out of the single SSD I had connected to the motherboard directly.

I ran a speed test from my MacBook Pro via a 2.5GbE network adapter dongle (to at least see if I was getting the full 2.5GbE line rate). I also switched out the dongle for 2 other brands (I have purchased 3 dongles in total for this test since they are relatively inexpensive).

I swapped the NIC with 2 other NICs sourced from 2 different sellers (one was brand new from Amazon, and one was used from eBay). So, I have tried a total of three x520 Intel NICs.

The best iperf 2 speed test result I get is still asymmetric: 2.35GbE in one direction (expected from a 2.5GbE dongle) and 0.7GbE in the other direction (WHY?!).

I then used a disk speed test app (Black Magic) to confirm that I was getting 250MB/s write and 90MB/s read from that single SSD pool (which matches the iperf results I got above). I have confirmed the SSD has about 500MB/s r/w performance.

I am confused how every single NIC and hardware combination I tried has given me poor read performance on TrueNAS-12.0-U5.1.

 

[NugentS]

One thing you haven't tested / tried / looked at is the switch (I am assuming there is a switch)
Can you try a direct connection - obviously not 2.5-2.5 as that won't work but a 10Gb to 10Gb and see what happens

 

[rathijit]

One thing you haven't tested / tried / looked at is the switch (I am assuming there is a switch)
Can you try a direct connection - obviously not 2.5-2.5 as that won't work but a 10Gb to 10Gb and see what happens

Actually I tried with and without the switch as well. It didn't make a difference.
Without the switch, using the 2.5GbE dongle, I just forced a manual IP address at both ends under the same subnet (/24).

 

[NugentS]

Oh, I am surprised that the Intel went down to 2.5 - I had assumed that that just wouldn't work.
I have a 10Gb client (windows), with a 10Gb server (TrueNAS Core) and a pool of SSD's with some spare space I can run some tests on for you to see if I can replicate.
When I read a large file from a SMB share on the HDD Pool (pool isn't big enough to saturate a 10Gb) I get on first pass 400MB/s. Pass 2 1.1 GB/s as the file is coming from ARC The file is 25GB. I am hopping through 2 switches. I get the same read results having copied the file to the SMB Share on the SSD Pool