Intel x520 settings to optimize reading speed?

[hellamasta]

Hello there, I'm Alessio and I'm new to this forum and quite the same concering TrueNAS.
I know the 10Gbit NIC topic has been discussed quite a lot and I read through a lot of posts finding no solution for my situation.
First thing first: the mine is NAS built with spare parts coming from a few old PCs. Here's a list of the hardware:
- AMD Phenom II X6
- gigabyte ga880gm-ud2h
- 8GB RAM (will be expanded to 16, the maximum the MB can support)
- EDUP SATA III PCI-e card
- 4 Seagate IronWolf 2TB
- 1 SSD 120GB for OS
I'm using TrueNAS core. I need to get access to the NAS from 3 PCs at my home, for photo and video editing. As my PCs have 2.5Gbps ethernet port, I decided to make a first step in order to upgrade my setup. First I bought a chinese 2.5Gbps card, installed it, edited the tunables and everything worked. I got a reading speed of about 240MB/s and writing 260/MB/s. However, if the NAS server was not used for a few hours, the card slowed down to a transfer rate of about 19MB/s. I returned it and tried with a Trendnet one, based on the same Realtek chip. Same thing. I investigated and understood the those chips are not stable and reliable. So I returned the second one and bough on ebay a Sun Oracle 7051223 based on intel x520-DA2. It is connected to a brand new Zyxel xgs1250-12 bought a couple of weeks ago.
Now, everything worked well immediately, however the reading speed is lower than the one I had with the chinese 2.5Gbps card.
I have about 190MB/s when reading, while writing is pretty better: 280MB/s.
I did some test with large files and turned out that reading is quite constant (from 175 to 191) while writing is at 285MB/s for a few seconds and then drops down to 180MB/s. I have a few GB of memory which get full very quickly and I noticed that de writing speed slows down when the memory is full.

I can understand that and the system will be upgraded asap, however I would like to get better reading speed, if possible. I tried with Tunable suggestions found on the forum:

Code:

sysctl kern.ipc.somaxconn=2048
sysctl kern.ipc.maxsockbuf=16777216
sysctl net.inet.tcp.recvspace=4194304
sysctl net.inet.tcp.sendspace=2097152
sysctl net.inet.tcp.sendbuf_max=16777216
sysctl net.inet.tcp.recvbuf_max=16777216
sysctl net.inet.tcp.sendbuf_auto=1
sysctl net.inet.tcp.recvbuf_auto=1
sysctl net.inet.tcp.sendbuf_inc=16384
sysctl net.inet.tcp.recvbuf_inc=524288

However it looked like they had no effect. I took a look to each parameter and, correct me if I'm wrong, saw that they are useful when handling an high number of connections sendind a lot of data in and out, which is not my case considering that I usually need a decent speed only from one PC at a time.

Can anyone help me in understanding the useful parameters and, maybe, how can I improve a little my system?

Thanks in advance.

 

[Nick2253]

It's possible that you're not exactly comparing apples to apples with the previous card.

For example, if the files you were testing on were in the ARC, then they would come out of memory (very fast) versus off the disk (not as fast). Not saying that's definitely going on, but could be.

 

[hellamasta]

It's possible that you're not exactly comparing apples to apples with the previous card.

For example, if the files you were testing on were in the ARC, then they would come out of memory (very fast) versus off the disk (not as fast). Not saying that's definitely going on, but could be.

Yes, I understand, but I would expect similar if not better performance.
I've just run a iperf3 test and the result is kinda strange.
In one direction i get almost a full 2.5Gbps, in the other 1Gbit less.
Any suggestion?

Code:

TRUNAS (client) TO WINDOWS (SERVER)
Accepted connection from 192.168.1.xxx, port 51084
[  5] local 192.168.1.yyy port 5201 connected to 192.168.1.xxx port 51085
[ ID] Interval           Transfer     Bitrate
[  5]   0.00-1.00   sec   257 MBytes  2.15 Gbits/sec
[  5]   1.00-2.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   2.00-3.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   3.00-4.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   4.00-5.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   5.00-6.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   6.00-7.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   7.00-8.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   8.00-9.00   sec   283 MBytes  2.37 Gbits/sec
[  5]   9.00-10.00  sec   283 MBytes  2.37 Gbits/sec
[  5]  10.00-10.00  sec   590 KBytes  2.35 Gbits/sec

WINDOWS(client) TO TRUENAS (server)
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------
Accepted connection from 192.168.1.yyy, port 55086
[  5] local 192.168.1.xxx port 5201 connected to 192.168.1.yyy port 43751
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-1.00   sec   134 MBytes  1.12 Gbits/sec
[  5]   1.00-2.00   sec   157 MBytes  1.31 Gbits/sec
[  5]   2.00-3.00   sec   168 MBytes  1.41 Gbits/sec
[  5]   3.00-4.00   sec   174 MBytes  1.46 Gbits/sec
[  5]   4.00-5.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   5.00-6.00   sec   173 MBytes  1.45 Gbits/sec
[  5]   6.00-7.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   7.00-8.00   sec   172 MBytes  1.45 Gbits/sec
[  5]   8.00-9.00   sec   172 MBytes  1.44 Gbits/sec
[  5]   9.00-10.00  sec   171 MBytes  1.44 Gbits/sec
[  5]  10.00-10.12  sec  21.1 MBytes  1.45 Gbits/sec
- - - - - - - - - - - - - - - - - - - - - - - - -
[ ID] Interval           Transfer     Bandwidth
[  5]   0.00-10.12  sec  0.00 Bytes  0.00 bits/sec                  sender
[  5]   0.00-10.12  sec  1.65 GBytes  1.40 Gbits/sec                  receiver
-----------------------------------------------------------
Server listening on 5201
-----------------------------------------------------------

 

[Nick2253]

If you try it with a different client, do you get the same results? What about between those two clients?

Have you tried a different cable with the TrueNAS system? Do you by chance have two different network connections active on TrueNAS?

 

[hellamasta]

If you try it with a different client, do you get the same results? What about between those two clients?

Have you tried a different cable with the TrueNAS system? Do you by chance have two different network connections active on TrueNAS?

Unfortunately I've just one 2.5Gbps capable machine. I'm waiting for a Chelsio 10Gbps NIC but will take some time.
I just tried the DAC. I've bought a fiber cable but still need to try it.
I've got three interfaces (two SFP+ on my Intel X520 + the 1Gbps on the mainboard). Only one active.
Or by "connection" do you mean two active clients accessing the NAS?

 

[Nick2253]

I've got three interfaces (two SFP+ on my Intel X520 + the 1Gbps on the mainboard). Only one active.
Or by "connection" do you mean two active clients accessing the NAS?

You got it on the first time. The reason I asked is because I've seen situations where multiple physical connections were active, and for some reason traffic was being routed out through the slower interface. However, I figured that wasn't the case here, since you were seeing faster than 1Gbps speeds on the "slow" side.

I would definitely try via a different cable, as well as the other SFP+ port. Unfortunately, without other computers, it's difficult to rule out other problems.

 

[hellamasta]

You got it on the first time. The reason I asked is because I've seen situations where multiple physical connections were active, and for some reason traffic was being routed out through the slower interface. However, I figured that wasn't the case here, since you were seeing faster than 1Gbps speeds on the "slow" side.

I would definitely try via a different cable, as well as the other SFP+ port. Unfortunately, without other computers, it's difficult to rule out other problems.

I tested with a fiber SR cable. Very same results. I really think it has to do either with
- hardware bottleneck when reading
- tunables in transmission buffer to be set

 

[Nick2253]

I'm familiar enough with tunables that yours look OK, but I'm not enough of an expert to advise more than that.

One piece of advice is that you can adjust the tunables on the fly and see if that makes a difference using the sysctl command. For example:

# sysctl net.inet.tcp.sendbuf_inc=16384

 

[hellamasta]

I'm familiar enough with tunables that yours look OK, but I'm not enough of an expert to advise more than that.

One piece of advice is that you can adjust the tunables on the fly and see if that makes a difference using the sysctl command. For example:

# sysctl net.inet.tcp.sendbuf_inc=16384

sorry for being so late in replying. I did a try, nothing changed.