Do you really need to buy a hardware raid controllers when using zfs?

[Mike77]

It`s megabyte`s. But now I tried iperf between a desktop (i7, 16gb 1600 ram, evo and intel nic) and the freenas (i3, 8gb 1300 ram, two seagtes and the different nics), with a switch between the machines. The speed between the nics maxed-out at 57,7 MByte/s... tried the same between the realteks and they were abit faster, but not a lot. So i believe there most be something else...

 

[jgreco]

Connect the machines directly and try.

 

[Mike77]

Connect the machines directly and try.

Did it.
Sadly the results aren´t any better...
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 7] local 192.168.10.2 port 5001 connected with 192.168.10.1 port 49381
[ ID] Interval Transfer Bandwidth
[ 7] 0.0-10.0 sec 583 MBytes 489 Mbits/sec
[ 8] local 192.168.10.2 port 5001 connected with 192.168.10.1 port 49382
[ 8] 0.0-10.0 sec 577 MBytes 484 Mbits/

 

[jgreco]

Do you have a third machine you can introduce to run iperf on, preferably something with a known-good network card? Use that directly connected to each of the two existing machines and you may be able to isolate one of them as being the source of the slowness.

 

[Mike77]

Sadly not with an Intel NIC . I´ve tested it with a I7 laptop (Realtek NIC) but it wasn't any different.

But I just tried iperf with four connections which, if I read this correctly, gives me a sum of about 112 MByte/sec:
If this is right then I don't understand why copying files from the windows machine to the NAS share can't get to those speeds...
------------------------------------------------------------
Client connecting to 192.168.1.18, TCP port 5001
TCP window size: 0.01 MByte (default)
------------------------------------------------------------
[180] local 192.168.1.21 port 56132 connected with 192.168.1.18 port 5001
[172] local 192.168.1.21 port 56131 connected with 192.168.1.18 port 5001
[156] local 192.168.1.21 port 56129 connected with 192.168.1.18 port 5001
[164] local 192.168.1.21 port 56130 connected with 192.168.1.18 port 5001
[ ID] Interval Transfer Bandwidth
[180] 0.0- 5.0 sec 140 MBytes 28.1 MBytes/sec
[172] 0.0- 5.0 sec 141 MBytes 28.1 MBytes/sec
[156] 0.0- 5.0 sec 141 MBytes 28.2 MBytes/sec
[164] 0.0- 5.0 sec 141 MBytes 28.1 MBytes/sec
[SUM] 0.0- 5.0 sec 562 MBytes 112 MBytes/sec
[172] 5.0-10.0 sec 141 MBytes 28.3 MBytes/sec
[164] 5.0-10.0 sec 140 MBytes 28.1 MBytes/sec
[180] 5.0-10.0 sec 141 MBytes 28.2 MBytes/sec
[156] 5.0-10.0 sec 141 MBytes 28.1 MBytes/sec
[SUM] 5.0-10.0 sec 563 MBytes 113 MBytes/sec
[172] 10.0-15.0 sec 141 MBytes 28.1 MBytes/sec
[156] 10.0-15.0 sec 140 MBytes 28.1 MBytes/sec
[180] 10.0-15.0 sec 140 MBytes 28.0 MBytes/sec
[164] 10.0-15.0 sec 141 MBytes 28.2 MBytes/sec
[SUM] 10.0-15.0 sec 562 MBytes 112 MBytes/sec
[180] 15.0-20.0 sec 141 MBytes 28.3 MBytes/sec
[172] 15.0-20.0 sec 141 MBytes 28.2 MBytes/sec
[156] 15.0-20.0 sec 141 MBytes 28.1 MBytes/sec
[164] 15.0-20.0 sec 141 MBytes 28.2 MBytes/sec
[SUM] 15.0-20.0 sec 564 MBytes 113 MBytes/sec
[ ID] Interval Transfer Bandwidth
[172] 20.0-25.0 sec 140 MBytes 27.9 MBytes/sec
[156] 20.0-25.0 sec 140 MBytes 28.0 MBytes/sec
[180] 20.0-25.0 sec 142 MBytes 28.4 MBytes/sec
[164] 20.0-25.0 sec 140 MBytes 28.1 MBytes/sec
[SUM] 20.0-25.0 sec 562 MBytes 112 MBytes/sec
[172] 25.0-30.0 sec 140 MBytes 28.0 MBytes/sec
[156] 25.0-30.0 sec 142 MBytes 28.3 MBytes/sec
[180] 25.0-30.0 sec 142 MBytes 28.4 MBytes/sec
[164] 25.0-30.0 sec 140 MBytes 28.1 MBytes/sec
[SUM] 25.0-30.0 sec 564 MBytes 113 MBytes/sec
[180] 0.0-30.1 sec 847 MBytes 28.2 MBytes/sec
[156] 0.0-30.1 sec 844 MBytes 28.1 MBytes/sec
[164] 0.0-30.1 sec 844 MBytes 28.0 MBytes/sec
[172] 0.0-30.1 sec 843 MBytes 28.0 MBytes/sec
[SUM] 0.0-30.1 sec 3378 MBytes 112 MBytes/sec
Done.

 

[jgreco]

If this is right then I don't understand why copying files from the windows machine to the NAS share can't get to those speeds...

Because CIFS uses a single connection, not four connections.

 

[Mike77]

So should I understaand that the problem isn't with the NIC's, but with the fact that I´m copying from a windows machine to a cif.
Are cifs in some way constricted, or can I still do anything to speed it up?

 

[jgreco]

It's hard to know what the exact problem is, but NAS is a definite case of weakest-link-in-chain. Each individual subsystem must be top-notch in order to get best speeds.

1) If the network stack cannot communicate effectively, that limits you. Controllers such as the Realtek are worse because they consume lots of CPU in order to get high speeds ("Designed For Windows"), while the Intel cards tend to be designed for problem-free performance.

2) CIFS is single-threaded per client, and so a slightly slower CPU can sometimes provide dramatically slower service because latency matters.

3) ZFS and the ARC require both CPU and memory for good operation, because ZFS is using your CPU as your RAID controller and it loves memory for cache. Starve it of either and slowness results.

4) The disk subsystem will limit you if you pick a poor pool design. Mirrored is among the best options for performance.

And, worse, even if you're fairly careful of each of these subsystems, the end result is that these are layered on top of each other, so that any minor weakness in one layer tends to get magnified as poor performance.