Slow network speed

[lurki12]

Hi
I have just built a freenas server, and i have a problem with poor network speed/transfer, when i copy files its only 4-5mb/s. What can be the problem?
Should it be better speeds than that?
My setup is:
16gb ram
Ryzen 5 2600
ASUS PRIME B450M-A/CSM
3x WD Red
1x Kinstron 240gb

 

[Samuel Tai]

What specific models of WD Reds? WD has been using shingled magnetic recording (SMR), which have atrocious IO performance. See this post on SMR. From my own experience, WD40EFAX drives use SMR, and I experienced write delays up to 10s (that's full seconds, not milliseconds). The earlier WD40EFRX drives use conventional perpendicular magnetic recording (CMR), and latencies there never peak over 6 ms.

 

[SweetAndLow]

What specific models of WD Reds? WD has been using shingled magnetic recording (SMR), which have atrocious IO performance. See this post on SMR. From my own experience, WD40EFAX drives use SMR, and I experienced write delays up to 10s (that's full seconds, not milliseconds). The earlier WD40EFRX drives use conventional perpendicular magnetic recording (CMR), and latencies there never peak over 6 ms.

Let's not bring up stuff that doesn't matter to confuse people.

Let's start with running a iperf test between the client and server.

 

[HoneyBadger]

Let's not bring up stuff that doesn't matter to confuse people.

Let's start with running a iperf test between the client and server.

"When you hear hoofbeats, think horses." WD Reds specifically have been getting a lot of (bad) press for sneaking this technology into their drives, and it's known to cause significantly adverse effects on performance and data integrity on ZFS systems. Taking a quick check of the model number makes good sense here, as does also running iperf/

Simpler questions though for you, @lurki12 - in addition to the question about "please post your hard drive model number"

Is this a new problem that started recently, or has it "always been an issue?"
Are you copying over a regular wired network, or another media (wireless, power-line networking?)
Which direction is the "file copy" - from client to FreeNAS, or FreeNAS to client?
What is your pool configuration (I'm assuming RAIDZ1) and how full is it?

 

[lurki12]

What specific models of WD Reds? WD has been using shingled magnetic recording (SMR), which have atrocious IO performance. See this post on SMR. From my own experience, WD40EFAX drives use SMR, and I experienced write delays up to 10s (that's full seconds, not milliseconds). The earlier WD40EFRX drives use conventional perpendicular magnetic recording (CMR), and latencies there never peak over 6 ms.

The hard drives I use are WD red WD10EFRX.

Let's not bring up stuff that doesn't matter to confuse people.

Let's start with running a iperf test between the client and server.

I've done the ipref test and this is the result.

"When you hear hoofbeats, think horses." WD Reds specifically have been getting a lot of (bad) press for sneaking this technology into their drives, and it's known to cause significantly adverse effects on performance and data integrity on ZFS systems. Taking a quick check of the model number makes good sense here, as does also running iperf/

Simpler questions though for you, @lurki12 - in addition to the question about "please post your hard drive model number"

Is this a new problem that started recently, or has it "always been an issue?"
Are you copying over a regular wired network, or another media (wireless, power-line networking?)
Which direction is the "file copy" - from client to FreeNAS, or FreeNAS to client?
What is your pool configuration (I'm assuming RAIDZ1) and how full is it?

Im currentIy run over wired network, wanted to eliminate error sources. I've always had this problem because I just built this system. I copy the files from my computer to the freenas.

This is my pool configuration

 

[SweetAndLow]

The hard drives I use are WD red WD10EFRX.


I've done the ipref test and this is the result. View attachment 38424

View attachment 38425


Im currentIy run over wired network, wanted to eliminate error sources. I've always had this problem because I just built this system. I copy the files from my computer to the freenas.

This is my pool configuration View attachment 38426

Remove the cache it's not helping you in any way.

Your networking is just really slow because you are using unsupported hardware. Your motherboard has a Realtek RTL8111H nic.

Get a Intel nic.

 

[lurki12]

Remove the cache it's not helping you in any way.

Your networking is just really slow because you are using unsupported hardware. Your motherboard has a Realtek RTL8111H nic.

Get a Intel nic.

if i get an intel nic network card, what can i expect for transfer speeds?
Tried running iperf between my mac and an intel nuc , this result is when both are connected with cable

And this is when one is connected with cable and one on wifi

Is this a normal result?

 

[Samuel Tai]

That's about expected for an Intel Gigabit Ethernet adapter. Wireless throughput is always pretty bad, but 227 Mbps is pretty close to the maximum for an 802.11n connection (300 Mbps).

 

[Samuel Tai]

I also wouldn't state categorically his NIC is unsupported. The FreeBSD Hardware Compatibility Guide does state the re driver can handle RealTek RTL8111 NICs.

There are a couple of loader tunables you could try to see if they improve performance:

hw.re.msi_disable=1
hw.re.msix_disable=1

After setting these tunables, you'll need to reboot to make them active.

 

[HoneyBadger]

Tried running iperf between my mac

Ah, now we're getting places. Are you copying to an SMB share on your FreeNAS machine? If so, you're likely hitting the "strict sync" problem.

In the SMB auxiliary settings (Services -> SMB -> Little Wrench Icon) set strict sync = no; you may need to restart the service to see the effects.

If you're using AFP though, we'll have to check other spots like the Realtek related tuning.

 

[wkn]

I am also using a mainboard with low end Celeron and onboard Realtek NIC (re0) and i am getting in peak 118 MB/s (which is near network maximum) over SMB from a pool with single drive in. No tuning parameters for the network, just default.
Always repeating the same old blames does not really make them more true!

 

[lurki12]

I also wouldn't state categorically his NIC is unsupported. The FreeBSD Hardware Compatibility Guide does state the re driver can handle RealTek RTL8111 NICs.

There are a couple of loader tunables you could try to see if they improve performance:

hw.re.msi_disable=1
hw.re.msix_disable=1

After setting these tunables, you'll need to reboot to make them active.

I'm not sure if I did right

Ah, now we're getting places. Are you copying to an SMB share on your FreeNAS machine? If so, you're likely hitting the "strict sync" problem.

In the SMB auxiliary settings (Services -> SMB -> Little Wrench Icon) set strict sync = no; you may need to restart the service to see the effects.

If you're using AFP though, we'll have to check other spots like the Realtek related tuning.

Yes, I copy from a windows machine to a SMB share on my freenas
I have changed the SMB auxiliary settings ,strict sync = no.

Now that I was running iperf I got good results every other time.

 

[Samuel Tai]

Yes, you set the tunables in the correct place. However, both values should just be 1, instead of what you have, and the variables should be hw.re.msi_disable and hw.re.msix_disable. They won't take effect until after a reboot. You'll have to experiment with each one, to see whether 0 or 1 gets you the best performance.

 

[Samuel Tai]

Also, after changing the SMB parameters, you'll need to restart the SMB service to make the new parameters take effect.

 

[SweetAndLow]

I also wouldn't state categorically his NIC is unsupported. The FreeBSD Hardware Compatibility Guide does state the re driver can handle RealTek RTL8111 NICs.

There are a couple of loader tunables you could try to see if they improve performance:

hw.re.msi_disable=1
hw.re.msix_disable=1

After setting these tunables, you'll need to reboot to make them active.

These are nice, any idea what the draw back is. Looked like they affect the interrupt method. Why wouldn't this just be the default value.

 

[lurki12]

I have tried changing hw.re.msi_disable = 1 and 0 , hw.re.msix_disable = 1 and 0, with no difference.
I've always restarted the freenas after every change

 

[lurki12]

I have borrowed this card from a friend Intel 82574L Gigabit Ethernet Controller which I understand is compatible with this list
FreeBSD Hardware Compatibility Guide. i still have poor speed between windows pc and freenas when i run the same test from my mac i get almost double speed, still not full speed.

 

[HoneyBadger]

The low iperf scores are pointing to network issues; there's no reason that a direct link should be that slow.

Start with Layer 1 - Replace your network cable if possible. Is the path to your FreeNAS machine going through a "home hub" or other ISP-provided "router" that might be hitting a limit on its ports?

Also, what is the network card in your Windows PC?

 

[Samuel Tai]

This is starting to smell like a TCP tuning issue, and not necessarily a problem with FreeNAS.
1. On the client side, you may need to adjust Window's TCP behavior.

• From an admin Command Prompt, run netsh
  
  interface tcp set global ecncapability = enabled (This enables RFC 3168 Explicit Congestion Notification)
  interface tcp set global timestamps = enabled (This enables RFC 7323 TCP timestamps, which are good to have with Gigabit or faster networks)
  
  Also, you'll need to make sure the Receive-Side Scaling setting matches the capabilities of your NIC driver. Most desktop NICs can't handle RSS, so this should be disabled.
  
  interface tcp set global rss = disabled
  interface tcp set global rsc = disabled

2. On FreeNAS, I set a couple of tunables to match Window's congestion control of TCP CUBIC

• Loader tunable: cc_cubic_load=YES
• Sysctl tunable: net.inet.tcp.cc.algorithm=cubic

 

[SweetAndLow]

I have borrowed this card from a friend Intel 82574L Gigabit Ethernet Controller which I understand is compatible with this list
FreeBSD Hardware Compatibility Guide. i still have poor speed between windows pc and freenas when i run the same test from my mac i get almost double speed, still not full speed.
View attachment 38459

Are you using WiFi with your windows PC? Wifi will always be slower.