Poor performance on 10g network

[egghog]

Hey all! Fairly new to this stuff and I have a question about performance.
I recently built a Truenas core system out of my old editing machine. I have 8x8TB seagate drives (4 are new Ironwolf disks, 4 are an older model, ST8000VN0002 disks which I previously used). They're in a RaidZ2 configuration.
Specs:
CPU: i7-4930k,
RAM: 32GB of Ripjaw DDR3 RAM.
Mobo: Asrock x79 extreme6
NIC: 2x Intel x540 T1 NICs (on both the server and my desktop)
Switch: Switch is the Netgear XS708E
Cables: Cat6 cables connecting everything, one 6 foot from server to switch, one 25 foot from the desktop.

I'm getting about 180MB/s max write speed from my SMB shares to windows, which seems normal. However my read speeds don't seem to be going above 220MB/s. I feel that the read speed should be much higher.

I did run an iperf test and got better speeds of 1.5gbit up and 4.2gbit down. I've also done smart tests on the disks and there seem to be no problems there.

I was told to check the management interface on the switch for error rates. It says port 1 had 6 CRC error packets. Not sure if that's the same thing. I also can't find anything that would indicate the connection is being limited somehow. All ethernet ports are set to full 10g (except internet), cable tester results came back ok.

Can anyone help solve this mystery? Any advice, suggestions, or thoughts are greatly appreciated.

 

[Davvo]

You could probably improve reads by adding more cache, though a fio test would be nice to see.

 

[NugentS]

Those iperf results are atrocious for 10Gb. You should be getting nearly 10Gb/s across the switch

 

[jgreco]

I did run an iperf test and got better speeds of 1.5gbit up and 4.2gbit down.

You can stop looking at other things until you fix this. This is your big picture problem. Be sure to be using iperf3 and multiple streams. You need to be able to max this out. Maybe not all the way, but @NugentS is fundamentally correct in that you should be able to get close -- or at least much closer -- to 10Gbps.

NIC: 2x Intel x540 T1 NICs (on both the server and my desktop)
Switch: Switch is the Netgear XS708E

And my gut feeling is that you may have knockoff X540 cards, though the real X540's work for crap too. You might want to review the 10 Gig Networking Primer for how to build a proper network without the crappy copper cabling. I realize that many users find copper to be comfortably familiar, but it seems to coincide with networking issues quite often.

 

[egghog]

You can stop looking at other things until you fix this. This is your big picture problem. Be sure to be using iperf3 and multiple streams. You need to be able to max this out. Maybe not all the way, but @NugentS is fundamentally correct in that you should be able to get close -- or at least much closer -- to 10Gbps.



And my gut feeling is that you may have knockoff X540 cards, though the real X540's work for crap too. You might want to review the 10 Gig Networking Primer for how to build a proper network without the crappy copper cabling. I realize that many users find copper to be comfortably familiar, but it seems to coincide with networking issues quite often.

Thanks for the reply. The NICs and switch are things I inherited from my old office, and it all seemed to work there, although I didn't set up the networking there so I can't be sure.

When I try to run iperf3 -s, I get "unable to start listener for connections, address already in use".

Regular iperf with 5 streams actually does worse, at just over 1gbit up and 1.7gbit down.

 

[NugentS]

The switch is probably fine.
The cards might be fine - the problem is more likley to be cables. Buy some decent Cat 6/7 cables and test them. Also get a x-over Cat 6/7 cable and test that directly without the switch. Its easy if its the cables, if it isn't the cables then you are into hardware issues or config issues.

Can you get into the switch and check its stats, look for error stats on the various ports and make sure its all configured correctly

"When I try to run iperf3 -s, I get "unable to start listener for connections, address already in use"." - On what? iperf3 runs by default on 5201 - you should have nothing on 5201 on TrueNAS

As an example I get about 7.2 Gbps on a single stream and 9.3 Gbps with 25 streams (in both directions) - which isn't perfect but I am happy with it
"iperf3.exe -P 25 -c iperf3_server_address"

 

[egghog]

@NugentS Sorry, didn't mean to post that blank quote reply and I'm not sure how to edit posts on this forum.

On the server, running "iperf3 -s" in the shell results in the error I mentioned. Using the same command but with iperf instead seems to work normally.
On the desktop, I have been using Jperf as it has a nice GUI, but I'm not sure if this is specific to iperf or will be compatible with iperf3. This stuff is not my strong suit so apologies for my ignorance in this :)

Side note, I now have a Truenas alert saying:
The following system core files were found: iperf.core. Please create a ticket at https://jira.ixsystems.com/ and attach the relevant core files along with a system debug. Once the core files have been archived and attached to the ticket, they may be removed by running the following command in shell: 'rm /var/db/system/cores/*'.
Not sure if this is the result of running iperf or attempting to run iperf3. But I'm not sure how to do what it's telling me in terms of attaching the core files and debug.

In any case, I've tried connecting the desktop to the switch with a brand new cat6 (20ft startech cable), no significant change in transfer speed. My Server's connection to the switch is also a brand new startech cable.
I will look into x-over cables, thanks for the tip!

I can't find anything on the switch settings. All ports are showing full speed (10G full for both the NAS and desktop, 1000M full for the internet), and there doesn't seem to be any limits or caps. I see 6 CRC error packets on port 1 (desktop). Nothing other than that.

 

[Ericloewe]

@NugentS Sorry, didn't mean to post that blank quote reply and I'm not sure how to edit posts on this forum.

After a few posts, your account will be upgraded to allow you to edit your own posts. If, in the meantime, you need something edited, just use the report feature to notify the mod team and we'll help you out.

 

[souporman]

I don't think crossover cables are even necessary these days, are they? Didn't they become kinda obsolete when Gigabit was ratified? Auto MDI-X should know what to do. No need to make a special cable.

EDIT: In fact, I run a regular old Cat6 from port #2 on my desktop directly to a port on my TrueNAS server, bypassing the switch altogether and it works great.

 

[Ericloewe]

I don't think crossover cables are even necessary these days, are they?

Pretty much.

Didn't they become kinda obsolete when Gigabit was ratified?

They were totally obsolete by the time 1GBase-T was ubiquitous, since it was mandated. Only the absolute cheapest, low-end crap for embedded systems would not support it these days. Even in 100Base-T, most modern stuff couldn't care less about crossover cables.
I'm not sure anyone even makes crossover cables beyond Cat. 5, because why on Earth would you need such a thing?

 

[jgreco]

why on Earth would you need such a thing?

To test the junior networking guys who are too lazy to eyeball a connector for faults, obviously.

 

[egghog]

To test the junior networking guys who are too lazy to eyeball a connector for faults, obviously.

Lol. In my defense I've already tried 2 sets of cables so I don't believe that's the issue.

 

[jgreco]

Lol. In my defense I've already tried 2 sets of cables so I don't believe that's the issue.

I didn't mean you. Occasionally @Ericloewe asks questions that have an obvious "to me" answer and I enjoy answering them. The opposite happens sometimes as well. It's just a forums thing. We do it for fun.

As for you... While bad cables are a possibility, I'm much more suspicious of your cards. Pull them and verify the Yottamarks against the Intel validator. Despite your having inherited them from a theoretically "legitimate" source, stuff happens and they could be fake.

 

[egghog]

I didn't mean you. Occasionally @Ericloewe asks questions that have an obvious "to me" answer and I enjoy answering them. The opposite happens sometimes as well. It's just a forums thing. We do it for fun.

OH, haha, copy that. Had to defend what little honor I've got :P

Well I checked the Yottamarks and they're both valid, so it looks like the cards are legit.

While Jperf is running its test, under "[SUM]" it does say 9.48 Gbits. This is when using 25 parallel streams.

On the reverse test however, it seems to only say 5.21 Gbits. To be honest I'm not sure which is up and which is down. But either way it's higher than the actual copying speed in Windows.

 

[jgreco]

But either way it's higher than the actual copying speed in Windows.

Yeah, it's supposed to be. One of the things that happens as speed increases is that the host platforms can't always keep up in a lock-step (singlethreaded) model, so it becomes important to run parallel streams in order to be able to maximize your throughput. However, Windows Copy utilizes only a single stream to do its trick. Therefore Windows Copy is expected to be slower -- perhaps significantly slower -- than the test case where you are running multiple parallel streams. This is such a problem on high speed networks for protocols like SMB that extensions such as SMB multichannel have been introduced to "fix" it.

9.48 Gbits

5.21 Gbits.

My guess here is that one of the operating systems involved isn't well tuned for 10Gbps, and it is quite possibly Windows. It is also quite possible that the affected system is just running out of steam, which often presents as capped iperf speeds, but then when you actually try to do "useful" stuff across the connection, the competition between kernel and userspace for CPU cycles and interrupts may cause a catastrophic performance collapse. I'm not saying that's what's happening to you, but be aware that such things can and do happen, especially when results are asymmetric like this. Evaluating this on Windows is outside my bailiwick, sorry.

 

[Morris]

My guess here is that one of the operating systems involved isn't well tuned for 10Gbps, and it is quite possibly Windows. It is also quite possible that the affected system is just running out of steam, which often presents as capped iperf speeds, but then when you actually try to do "useful" stuff across the connection, the competition between kernel and userspace for CPU cycles and interrupts may cause a catastrophic performance collapse. I'm not saying that's what's happening to you, but be aware that such things can and do happen, especially when results are asymmetric like this. Evaluating this on Windows is outside my bailiwick, sorry.

I run 10-Gb on Windows 10 and don't have these issues. Even on a VM hosted by TrueNAS I can hit 7-Gb.

 

[Ericloewe]

I don’t think anybody’s claiming that Windows is a bottleneck per se, rather that the issue at hand could be on the Windows machine. There are too many variables at play to point out a smoking gun at the moment.

 

[jgreco]

I run 10-Gb on Windows 10 and don't have these issues. Even on a VM hosted by TrueNAS I can hit 7-Gb.

That's really not relevant. How many cores does the VM have? How many cores are handling the network stack? What's the speed of the cores? How much kernel memory is allocated for buffers? Has the system done any automatic tuning to optimize for LAN vs WAN connections? What's the design of the ethernet card? How does it handle queueing? Interrupts? Is the VM talking to the NAS hosting it, and traffic is just traversing the bridge? Using a paravirtualized device? There are so many components that can contribute to network performance that it just isn't particularly useful to say "works for me". It works for me too. But that's because I tend to use high end gear and quality cards, plus I generally don't obsess too much over the results, as long as they don't suck too bad.

 

[Morris]

I just reread your original post. Two things caught my attention:

Your SMB read and write don't tell me the direction.
What storage is on your windows box?

A sequential copy from fast storage on the Windows box to a RAIDZ2 of the type you describe is not going to fly because the CPU will be very busy calculating CRC. I've never made complete sense of why yet a RAIDZ1 will have less CPU and fewer spindles in a RAIDZ1 will also use less CPU.

I have a large pool that's used as a backup target. The best configuration I could find for throughput is a strip of two RAIDZ1. I'm using 6 drives so 2 Strips of 3 disks in RAIDZ1. Depending on your application, this may be an acceptable compromise between throughput and reliability.

If you can move the data on the pool try reconfiguring and run benchmarks to find the best configuration.

 

[egghog]

I just reread your original post. Two things caught my attention:

Your SMB read and write don't tell me the direction.
What storage is on your windows box?

A sequential copy from fast storage on the Windows box to a RAIDZ2 of the type you describe is not going to fly because the CPU will be very busy calculating CRC. I've never made complete sense of why yet a RAIDZ1 will have less CPU and fewer spindles in a RAIDZ1 will also use less CPU.

I have a large pool that's used as a backup target. The best configuration I could find for throughput is a strip of two RAIDZ1. I'm using 6 drives so 2 Strips of 3 disks in RAIDZ1. Depending on your application, this may be an acceptable compromise between throughput and reliability.

If you can move the data on the pool try reconfiguring and run benchmarks to find the best configuration.

The storage on my windows box is mainly NVME with a couple sata SSDs. The drive I've been using to test is a 1TB Samsung 980 Evo Pro, but speeds are the same across all drives.

I don't currently have anywhere to move the data unfortunately. Anyway since I work with heavy video files I need the extra storage efficiency, and I think a mirrored Z1 might sacrifice too much capacity.

If it really is just a side effect of the topology then I'm willing to live with it, but it seems (at least anecdotally) that there's some other problem