Tune new FreeNAS build

[Tysken]

Hi,
I have played around with freenas for a wile now, and used it in a vm on my htpc for backups.
After i did some research i decided to get a dedicated freenas box with these specs:

freenas 8.3
asrock b75 pro3 (cheap and has 8 compatible sata ports)
intel i3-3220 (dualcore with HT@3.3GHz)
16Gb ram
6x WD 2TB EARX that i already had

I did set up the disks in a raidz2 and dd (with bs=2048k cont=50k) gives me writes at 284 MB/s and reads at 295MB/s

BUT
I cant get CIFS run as fast as i hoped. I tried all kinds of settings in the gui after reading on the forum and the top result is
65MB/s reads
95MB/s writes

What i have played around with are the settings in the cifs service settings. RW support AIO sizes and so on

The highest cpu load on freenas i have seen in the reporting tab is about 43% so i guess that shouldn't be the problem.
I tried turning of hyper threading with a little bit lower results.
I tried a intel desktop 1000 pci nic without any difference.
The desktop has a sdd raid0 and can read files at about 110-120MB/s from my AMD E35 htpc so the bottleneck shouldn't be there.
I put an older ssd in the freenas box and the speeds are the same to the ssd too.

Somehow it feels that the problem is network related, when i copy the same file multiple times (i guess freenas cashes the files in ram) the transfer speed is rock steady at 67% when checking the task manager on my desktop.

Does anyone have an idea what i can try to get my speed up. Ore maybe a link to a thread i can read to get some ideas?

Sorry for my bad english and thanks in advance // Erik

 

[Tysken]

I read some more here on the forum and did a iperf test and got this:


Freenas sending:

[root@freenas] ~# iperf -c 192.168.1.4 -i 1
------------------------------------------------------------
Client connecting to 192.168.1.4, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 3] local 192.168.1.2 port 35878 connected with 192.168.1.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 3] 0.0- 1.0 sec 73.0 MBytes 612 Mbits/sec
[ 3] 1.0- 2.0 sec 72.0 MBytes 604 Mbits/sec
[ 3] 2.0- 3.0 sec 73.5 MBytes 617 Mbits/sec
[ 3] 3.0- 4.0 sec 73.5 MBytes 617 Mbits/sec
[ 3] 4.0- 5.0 sec 73.2 MBytes 614 Mbits/sec
[ 3] 5.0- 6.0 sec 73.1 MBytes 613 Mbits/sec
[ 3] 6.0- 7.0 sec 73.5 MBytes 617 Mbits/sec
[ 3] 7.0- 8.0 sec 73.2 MBytes 614 Mbits/sec
[ 3] 8.0- 9.0 sec 73.6 MBytes 618 Mbits/sec
[ 3] 9.0-10.0 sec 73.0 MBytes 612 Mbits/sec
[ 3] 0.0-10.0 sec 732 MBytes 614 Mbits/sec


Freenas recieving:

bin/iperf.exe -c 192.168.1.2 -P 1 -i 1 -p 5001 -N -l 1.0M -f m -t 6000 -T 1
------------------------------------------------------------
Client connecting to 192.168.1.2, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[168] local 192.168.1.4 port 59399 connected with 192.168.1.2 port 5001
[ ID] Interval Transfer Bandwidth
[168] 0.0- 1.0 sec 1.00 MBytes 8.39 Mbits/sec
[168] 1.0- 2.0 sec 106 MBytes 889 Mbits/sec
[168] 2.0- 3.0 sec 110 MBytes 923 Mbits/sec
[168] 3.0- 4.0 sec 109 MBytes 914 Mbits/sec
[168] 4.0- 5.0 sec 110 MBytes 923 Mbits/sec
[168] 5.0- 6.0 sec 107 MBytes 898 Mbits/sec
[168] 6.0- 7.0 sec 110 MBytes 923 Mbits/sec
[168] 7.0- 8.0 sec 109 MBytes 914 Mbits/sec
[168] 8.0- 9.0 sec 109 MBytes 914 Mbits/sec
[168] 9.0-10.0 sec 107 MBytes 898 Mbits/sec

Does anybody have a idea?

 

[cyberjock]

Do you have a firewall on your Windows machine? My windows server has terrible network transfer speeds when using Symantec Endpoint protection. The system has good performance in all regards, except when transferring files. I get about 30MB/sec tops when the firewall is enabled. If I disable it the speeds immediately jump to 60-90MB/sec.

Other than that, it's tough to say without a 3rd machine to test each of those systems with. If a second machine tests slow speeds to your windows machine then your windows machine is likely the culprit.

I know I've had a hard time getting over about 700Mbit/sec with non-Intel NICs. Any reason you didn't try using a PCIe Intel NIC? PCI Gb NICs can bottleneck your network speeds.

 

[Tysken]

Thanks for your reply!
I have windows firewall on my desktop and i will try deactivating it when I get back from work later. My avira antivirus doesn't make any difference witch I already have tryed.

I am able to read files from my media pc running win 7 to the desktop at almost full gigabit speed (110-120MB/s) but I only get max 70MB/s from the Freenas to the same machine.

I don't have a intel pic-e nic laying around but if you think that my Realtek onboard pic-e nic might be the problem I buy a intel one and give it a try. Ore maybe I should try some live cd on the server and test the Ethernet speed that way.

Any more ideas?

 

[William Grzybowski]

Your realtek is probably the issue. It might work a little better with Windows or Linux, but the FreeBSD driver for this NIC is not so sophisticated to handle the bad hardware implementation.

 

[Tysken]

I just did some more tests with the intel pci nic and after pulling out my other pci nic and disabling everything i don´t really need in the bios i got almost the same results as with the realtek nic. Writes to freenas are 20-30MB/s faster than reads witch i think is strange because if the nic drivers ore the pci speed would be the bottleneck i think the speed should be the same for reads and writes?

Does anyone have more suggestions what i can try to find the bottleneck. Would be nice to get the speed up before i move all my stuff to the freenas server.

 

[cyberjock]

Did you try running the iperf test with the intel NIC? The iperf test really is a better indicator for your current issue.

As it is, since your iperf test showed 600Mbit/sec from server to workstation, there's no way you'll get more than about 90% of that. The fact that your iperf test isn't showing 900Mbit+ is, in my opinion, the leading indicator of something wrong with your network hardware. You should be getting 900Mbit+, regardless of your CIFS performance.

Since you are using an Intel PCI NIC, are you using any other PCI cards in that system? If you have another PCI device on the system it can bottleneck your NIC.

 

[Tysken]

I did a new install of freenas and i have everything unplugged including the drives.
The only card in the server is the intel nic and i disabled soundcard ,usb3 ,onboard nic and sata controlers in the bios.

i havent changed any settings ore activated any services in freenas and i am running on DHCP.
Now i am getting these iperf results on the intel nic:



Freenas Sending:

[root@freenas ~]# iperf -c 192.168.1.4 -i 1
------------------------------------------------------------
Client connecting to 192.168.1.4, TCP port 5001
TCP window size: 32.5 KByte (default)
------------------------------------------------------------
[ 6] local 192.168.1.20 port 48449 connected with 192.168.1.4 port 5001
[ ID] Interval Transfer Bandwidth
[ 6] 0.0- 1.0 sec 65.8 MBytes 552 Mbits/sec
[ 6] 1.0- 2.0 sec 65.5 MBytes 549 Mbits/sec
[ 6] 2.0- 3.0 sec 65.8 MBytes 552 Mbits/sec
[ 6] 3.0- 4.0 sec 65.4 MBytes 548 Mbits/sec
[ 6] 4.0- 5.0 sec 65.2 MBytes 547 Mbits/sec
[ 6] 5.0- 6.0 sec 65.4 MBytes 548 Mbits/sec
[ 6] 6.0- 7.0 sec 65.8 MBytes 552 Mbits/sec
[ 6] 7.0- 8.0 sec 65.2 MBytes 547 Mbits/sec
[ 6] 8.0- 9.0 sec 64.5 MBytes 541 Mbits/sec
[ 6] 9.0-10.0 sec 64.2 MBytes 539 Mbits/sec
[ 6] 0.0-10.0 sec 653 MBytes 548 Mbits/sec



Freenas Receiving:

bin/iperf.exe -c 192.168.1.20 -P 1 -i 1 -p 5001 -N -l 1.0M -f m -t 6000 -T 1
------------------------------------------------------------
Client connecting to 192.168.1.20, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[160] local 192.168.1.4 port 57931 connected with 192.168.1.20 port 5001
[ ID] Interval Transfer Bandwidth
[160] 0.0- 1.0 sec 82.0 MBytes 688 Mbits/sec
[160] 1.0- 2.0 sec 83.0 MBytes 696 Mbits/sec
[160] 2.0- 3.0 sec 82.0 MBytes 688 Mbits/sec
[160] 3.0- 4.0 sec 82.0 MBytes 688 Mbits/sec
[160] 4.0- 5.0 sec 82.0 MBytes 688 Mbits/sec
[160] 5.0- 6.0 sec 83.0 MBytes 696 Mbits/sec
[160] 6.0- 7.0 sec 82.0 MBytes 688 Mbits/sec
[160] 7.0- 8.0 sec 82.0 MBytes 688 Mbits/sec
[160] 8.0- 9.0 sec 82.0 MBytes 688 Mbits/sec
[160] 9.0-10.0 sec 85.0 MBytes 713 Mbits/sec
[160] 10.0-11.0 sec 81.0 MBytes 679 Mbits/sec
[160] 11.0-12.0 sec 82.0 MBytes 688 Mbits/sec


The performance is better with the realtek card but the difference in read and write is still there.
I have ordered a PCIe intel nic that i should get in a few days but i am not quite sure the nic is the problem. I do not understand why there is a difference in read versus write. Any more ideas?

 

[Tysken]

I did some more testing with iperf to both the desktop and my media pc and got similar results. I also played around with some iperf setting but they didn't do much difference.
I also tried with a crossed cable directly between the nas and desktop with the same results as i posted before on both the intel and realtek nic. So there shouldn't be a problem with cabling/switch.

Does anyone have a link to a thread with iperf settings and results of a well performing system? Whould be interesting to se if i use the correct settings to determine the max bandwith. I have searched the forum but i can only find posts of badly performing iperf logs.

 

[Stephens]

When you do iperf -s on the FreeNAS server, it gives you a TCP window size (probably 64k). Make sure your client side uses the same window size. You need some consistency in your window sizes (buffers) if you're going to do real tests.

Code:

Freenas Sending: TCP window size: 32.5 KByte (default) 
Freenas Receiving: TCP window size: 0.01 MByte (default)

Start with forcing 64k buffers.

I doubt the RealTek is the issue at this point as I get 900mbit/s+.

 

[Tysken]

Thanks for your reply!
I tryed using a 64k buffer and did also try some other sizes without getting better results.
Do you maybe have an exact iperf command that you use when you get 900+ mbit/s?

I just read the recomended hardware faq and my realtek onboard nic is listed, my guess is it wouldn´t if the max speed was 700mbit/s.

Tonight i think i try installing 8.2 instead of 8.3 and se if that makes anny difference. And tomorrow i get my 2 intel PCIe nic`s so i can do some testing with them.

 

[Stephens]

It also depends on the ability of your switch to pass the packets, and decent LAN cables.

From my FreeNAS server (RealTek 8111E), I just run "iperf -s" and it reports 64KByte TCP Windows size (ctrl-c to stop).

From my Windows box (Broadcom NetLink Gigabit), I run jperf in client mode to my FreeNAS IP, go down to Transport Layer Options, check TCP Window Size, then change it from 56 KBytes to 64KBytes, then "Run IPerf!".

Code:

[ ID] Interval       Transfer     Bandwidth
[156]  0.0- 1.0 sec  114464 KBytes  937689 Kbits/sec
[156]  1.0- 2.0 sec  112400 KBytes  920781 Kbits/sec
[156]  2.0- 3.0 sec  112064 KBytes  918028 Kbits/sec
[156]  3.0- 4.0 sec  107232 KBytes  878445 Kbits/sec
[156]  4.0- 5.0 sec  106632 KBytes  873529 Kbits/sec
[156]  5.0- 6.0 sec  110008 KBytes  901186 Kbits/sec
[156]  6.0- 7.0 sec  115000 KBytes  942080 Kbits/sec
[156]  7.0- 8.0 sec  109760 KBytes  899154 Kbits/sec
[156]  8.0- 9.0 sec  106656 KBytes  873726 Kbits/sec
[156]  9.0-10.0 sec  107160 KBytes  877855 Kbits/sec
[156]  0.0-10.0 sec  1101384 KBytes  901932 Kbits/sec
Done.

 

[Tysken]

Thanks!
I have tried a twisted cat 5e cable connected directly between the machines without switch and i get the exact same results as i get with my cat6 cables connected to the switch. I can also transfer files faster than this over the switch to another machine so i don´t think the problem is there.

My speeds are actually almost the same as yours when i run the server on freenas.
It is when i run the client on freenas the results are lower.

I also tried installing 8.2 instead without any luck.

 

[Stephens]

I couldn't make my Win7 PC act as server, likely due to firewalls, and I wasn't up to getting into changing that. Instead, I set up 1 FreeNAS server as iperf server and one as iperf client, and I get the same thing you do (about 786Mbits/sec in my case, so ~800Mbit vs ~900Mbits). At least you know your results are consistent with someone else using the RealTek 8111E chipset.

 

[Yell]

Not sure if it also helps for CIFS (because i use NFS and its listed unter NFS tuning):

Code:

# v15
vfs.zfs.zil_disable="1"

# v28
zfs set sync=disabled tank/dataset 

With disabled ZIL my write speed jumps from 59 MB/s to 107MB/s writing a 21 GB file (dd if=/dev/zero of=/mnt/NFS/somelargefile.bak count=2048k bs=10k)
That's nearly 100% increase (since i don't have a SSD to spare for ZIL)

Read the following wiki !
http://www.solarisinternals.com/wiki/index.php/ZFS_Evil_Tuning_Guide#Disabling_the_ZIL_.28Don.27t.29

(which doesn't bother me, as i only write and one access at the time)

 

[Tysken]

Thank you all for replying!
I got my PCIe nic today and everything works great now! :p
The iperf results went up to 900+ mbit/s and my cifs transfer speeds are about 110MB/s both ways.

The bottleneck was the realtek nic. Strange that it is on the recommended hardware list when it cant perform better than that.
Ore maybe it is just the implementation on my motherboard that doesn't work to well.

Anyway i am happy now and freenas is really a great product! Time to move some files

 

[cyberjock]

Thank you all for replying!
I got my PCIe nic today and everything works great now! :p
The iperf results went up to 900+ mbit/s and my cifs transfer speeds are about 110MB/s both ways.

The bottleneck was the realtek nic. Strange that it is on the recommended hardware list when it cant perform better than that.
Ore maybe it is just the implementation on my motherboard that doesn't work to well.

Anyway i am happy now and freenas is really a great product! Time to move some files

I was always convinced it was the controller. It may be on the recommended hardware list, but alot of other stuff is on that list that doesn't work quite 100% in FreeBSD. It really has alot more to do with the way the hardware is implemented and how much support FreeBSD has for that hardware than anything else. I've learned that even on my home network pre-FreeNAS that putting Intel PCIe NICs in all of the desktops and servers really does cause an instantaneous boost in network performance of 20-30%. I wouldn't have believed it if I hadn't seen it with my own eyes.