New freenas build w/ low network transfer speeds

[zoober]

Hello,

I'm new to freenas, with this build being my first experience with the setup. While the setup process went pretty smooth I was disappointed by the read/write speeds of the share (both CIFS and FTP)

My Build:
asrock c2750d4i
32GB (4x 8GB) kingston memory
6x WD30EFRX
sandisk cruzer fit 16gb
Seasonic SSR-450RM
Node 304

I'm running all 6x disks in a RAID-Z2. All hard drives are connected to the intel SATA ports. All disks have passed the long smartctl tests. The system is connected to a RT-AC68U router running Asuswrt-Merlin and connected via cat6 cable. The router reports that the port is using full duplex. I've also tried swapping out the ethernet cable.

Code:

C:\Users\Chris\Desktop\jperf-2.0.2\bin>iperf.exe -c 10.0.1.250 -P 1 -i 1 -t 20
------------------------------------------------------------
Client connecting to 10.0.1.250, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[268] local 10.0.1.49 port 52525 connected with 10.0.1.250 port 5001
[ ID] Interval       Transfer     Bandwidth
[268]  0.0- 1.0 sec  50.0 MBytes   419 Mbits/sec
[268]  1.0- 2.0 sec  64.3 MBytes   539 Mbits/sec
[268]  2.0- 3.0 sec  64.0 MBytes   537 Mbits/sec
[268]  3.0- 4.0 sec  63.8 MBytes   535 Mbits/sec
[268]  4.0- 5.0 sec  64.3 MBytes   539 Mbits/sec
[268]  5.0- 6.0 sec  64.4 MBytes   540 Mbits/sec
[268]  6.0- 7.0 sec  64.4 MBytes   541 Mbits/sec
[268]  7.0- 8.0 sec  64.1 MBytes   537 Mbits/sec
[268]  8.0- 9.0 sec  64.6 MBytes   542 Mbits/sec
[268]  9.0-10.0 sec  64.4 MBytes   540 Mbits/sec
[268] 10.0-11.0 sec  64.3 MBytes   539 Mbits/sec
[268] 11.0-12.0 sec  64.2 MBytes   539 Mbits/sec
[268] 12.0-13.0 sec  64.0 MBytes   537 Mbits/sec
[268] 13.0-14.0 sec  63.9 MBytes   536 Mbits/sec
[268] 14.0-15.0 sec  64.2 MBytes   539 Mbits/sec
[268] 15.0-16.0 sec  64.2 MBytes   539 Mbits/sec
[268] 16.0-17.0 sec  64.0 MBytes   537 Mbits/sec
[268] 17.0-18.0 sec  64.6 MBytes   542 Mbits/sec
[268] 18.0-19.0 sec  64.5 MBytes   541 Mbits/sec
[268] 19.0-20.0 sec  64.1 MBytes   538 Mbits/sec
[ ID] Interval       Transfer     Bandwidth
[268]  0.0-20.0 sec  1.24 GBytes   533 Mbits/sec

Code:

[root@Permanence ~]# dd if=/dev/zero of=/mnt/permanence_pool/Media/ddfile bs=204
8k count=10000                                                              
10000+0 records in                                                          
10000+0 records out                                                         
20971520000 bytes transferred in 35.223389 secs (595386205 bytes/sec

[root@Permanence /mnt/permanence_pool/Media]# dd if=/dev/zero of=ddfile bs=1m co
unt=20000
20000+0 records in
20000+0 records out
20971520000 bytes transferred in 38.986238 secs (537921099 bytes/sec)

When I transfer a 2GB movie file from my gaming machine to the NAS, I get a transfer rate between 60-64MB/s. When transferring the same file from the NAS, I get a transfer rate of 75-80MB/s.

Is this transfer rate to be expected? In my research of this configuration I was expecting to get around 100MB/s.

 

[m0nkey_]

Are your drives plugged into the Intel or Marvell controller?

 

[zoober]

The intel controller.

 

[jgreco]

Your gaming machine. It has the Realtek ethernet, or perhaps the Marvell ethernet, yes? And the built-in switch in a consumer grade NAT device, that could be problematic.

At least one of those two things are likely to be killing your network speeds. You can connect the NAS and the gaming PC directly together to see if that helps; if so, go and buy yourself a real ethernet switch. For the gaming machine, they love to include the cheap parts so that they can tick off all their feature requirements without increasing cost. If that turns out to be the problem, go get yourself an Intel Pro/1000 PCIe desktop adapter (~$35) and you'll probably discover that your iperf speeds suddenly hit about 1Gbps. Usually you can only get peak speeds when you've done everything right. Networking is funny that way.

 

[Ericloewe]

And the built-in switch in a consumer grade NAT device, that could be problematic.

It's your average mildly-intelligent 5-port switch chipset. Shouldn't cause any trouble...

Your gaming machine. It has the Realtek ethernet, or perhaps the Marvell ethernet, yes?

That sounds a lot more likely. For their Ivy Bridge boards, Asus decided to only ship Intel GbE, to everyone's cheers. It didn't catch on, nobody else did it and Asus returned to RTL8111s on their cheap-ass stuff the year after that.

 

[jgreco]

It's your average mildly-intelligent 5-port switch chipset. Shouldn't cause any trouble....

Yet that doesn't prevent them from causing all sorts of weird performance issues. Since it's easily checked, the smart money is on not guessing and actually testing.

 

[zoober]

My gaming PC has a Killer k1000 1Gb ethernet (or big foot, or whatever they're calling themselves now a days). I admit it doesn't have the best driver support, but I can still transfer at ~100Mb/s when accessing the SMB on my old Synology NAS. I hooked the freenas directly to computer and it still maxes out at 64MB/s.

My old synology has 7200 RPM drives while the freenas has 5400 RPM drives. Is this expected perfromance for 5400 drives?

 

[zoober]

Adding the below auxiliary parameters boosted the transfer speed up to 70MB/s.

Code:

ea support = no
store dos attributes = no
map archive = no
map hidden = no
map readonly = no
map system = no
oplocks = yes
socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=131072 SO_SNDBUF=131072 IPTOS_THROUGHPUT
read raw = yes
write raw = yes
oplocks = yes
max xmit = 131072
write cache size = 131072

 

[Ericloewe]

My gaming PC has a Killer k1000 1Gb ethernet

Well, there's your problem. Absolute garbage

 

[jgreco]

I can still transfer at ~100Mb/s when accessing the SMB on my old Synology NAS. I hooked the freenas directly to computer and it still maxes out at 64MB/s.

So 100Mb/s = 12MB/s. Your FreeNAS is five times faster than the Synology, then?

https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Well, there's your problem. Absolute garbage

The problem is that a network is a complicated thing, so every weak part tends to contribute to overall suckage. Mildly increasing the buffer sizes can help, increasing them further might help a little more. However, given the original iperf numbers, that look kinda crummy, I'm not sure it'll get that far. Do the iperf results get better if you give iperf a better window size?

 

[zoober]

Well, there's your problem. Absolute garbage

But if that's the problem, why am I able to get a 100+MB/s transfer to my synology using the same computer? If the problem is the crappy NIC, wouldn't I have the same issues with the synology? With the exception of the 5400 RPM disks, every piece of gear in the new NAS is leaps and bounds above what's in the synology.

 

[zoober]

So 100Mb/s = 12MB/s. Your FreeNAS is five times faster than the Synology, then?

https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Sorry, that was a typo. I'm getting 100-110MB/s transfer rate to the synology SMB, compared to the 70MB/s I'm getting on the freenas.

The problem is that a network is a complicated thing, so every weak part tends to contribute to overall suckage. Mildly increasing the buffer sizes can help, increasing them further might help a little more. However, given the original iperf numbers, that look kinda crummy, I'm not sure it'll get that far. Do the iperf results get better if you give iperf a better window size?

I'll try when I get home today. I'll also try and hook my macbook directly to the freenas and see what the iperf results are. At the very least the latter will rule out a few things.

 

[jgreco]

But if that's the problem, why am I able to get a 100+MB/s transfer to my synology using the same computer? If the problem is the crappy NIC, wouldn't I have the same issues with the synology? With the exception of the 5400 RPM disks, every piece of gear in the new NAS is leaps and bounds above what's in the synology.

Because as I've said like several times now in this thread, networking is very complicated and often breaks at the smallest provocation. I think you very much need to stop trying doing NAS protocol transfers and figure out the underlying network issues here. If you're not getting 1Gbps out of your network, that's pretty much cowpoop and layering the NAS complexity on top of a network that's clearly teh suck ... is only going to suck more. Chain is as strong as weakest link, or any of those other crappy sayings. All very true for a NAS.

Since I've had this discussion hundreds of times on this forum, the next step is usually where there's more "but... Synology... faster...!" so I'll even give you a simple example of it.

So let's say your new FreeNAS has a very fast Intel ethernet. CIFS passes off a 1MB file to the driver. The driver queues up a bunch of packets and starts gunning them out as fast as the network can take them, back to back. Your crappy gaming PC, on the other hand, can maybe buffer 16K of packets before it has to get its interrupt serviced, and since the driver isn't really good, the packets start piling up until finally one packet is lost in the deluge coming in from the NAS. So everything stops for a moment, and then the NAS resends the packet. Lather, rinse, repeat.

Meanwhile, your old NAS uses an inexpensive Gianfar ethernet chipset. This thing can't actually pump packets out literally back to back. It can go *pretty* fast, though. So the PC has a little bit of an easier time keeping up, and doesn't drop any packets.

Suddenly it *looks* to you like the old NAS is doing much better than the FreeNAS. It's not. It's just that your PC is sucking bad.

That's just one hypothetical example. There are a number of them. We don't know which of them it might be. It could even be the NAS. Who knows. But even once the network seems to be working right, that doesn't actually mean it is. It just means we're closer.

So at this point, you need to understand that there are reasons that I want to identify your network issues prior to even beginning to contemplate transfer speeds. And you'll notice from my very first message what I jumped on? Because I've had this discussion hundreds of times, and at least seventy percent of those, answer is... "put a real Intel Ethernet in there," and that was the solution that ultimately fixed it. Bad cabling, bad switching, configuration issues, user omissions ("I'm on wifi!" is my fav) make up most of the rest.

 

[jgreco]

Sorry, that was a typo. I'm getting 100-110MB/s transfer rate to the synology SMB, compared to the 70MB/s I'm getting on the freenas.

That's why I posted a link to the terminology primer. Please do type things out. It makes a difference. Communication failures between forum participants are as unhelpful as communication failures between machines on your network.

 

[m0nkey_]

@jgreco is correct. Intel NICs end to end here and have no such issue (regularly tops 100MB/sec). However, another machine that has Realtek does often struggle to keep up and sees at most 60-70MB/sec.