Where does the performance gap (READ <-> WRITE) come from? Problems after MB change...

[Tobi]

Dear forum members,
because this is my one of my first posts, I want to say hello to you first:
My name is Tobi, I am 19 years old and started studying information sciences and media technologies in Stuttgart, Germany. I needed a big file server for media files (70% videos, 20% fotos, 10% other stuff).
I write this post because I have a problem with my (read) performance of this media server:
My first board (asus desktop mainboard, RaidZ1, non AES-NI cpu) had a transfer speed of ~110 MByte/s (in read and write 'mode') but no ecc. Now I upgraded to a SuperMicro board for ecc ram as suggested in the hardware recommendations (https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/).
I expected that this new setup would deliver the same ~ 110MByte/s but it doesn't it is much slower:

Hardware:

• CPU: Intel i3-4160T (3,1 GHz, dual)
• MB: Supermicro X10SL7-F (flashed with Firmware V16 IT Mode)
• RAM: 2x Kingston ValueRAM - DDR3 - 8 GB - DIMM 240-PIN - 1600 MHz / PC3-12800 - CL11
• HDD: 6x Seagate Barracuda ST3000DM001 (3TB each - 18 TB total)
• FreeNAS running directly from a NAND flash USB stick (NO VM)

The test machine (Windows 8.1) should not be the problem (AMD 8350, 16GB ram, Intel PRO1000 1GbE network card, Samsung SSD, ...).

Configuration:

• RaidZ2
• FreeNAS-9.3-STABLE-20150513
• compression: OFF
• dedup: OFF
• Share type: Windows / CIFS
• atime: ON
• all in all: no special configuration, just a minimal functional setup

With this setup I get the following measurements with Crystal Disk Mark. I know it is not the best tool but it underlines completly what I can see transfering video and photo files to the NAS. The speeds are exactly the same as shown in the file transfer window from windows:

Code:

-----------------------------------------------------------------------
CrystalDiskMark 3.0.3 x64 (C) 2007-2013 hiyohiyo
                           Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

           Sequential Read :    85.682 MB/s
          Sequential Write :    95.979 MB/s
         Random Read 512KB :    81.567 MB/s
        Random Write 512KB :    93.407 MB/s
    Random Read 4KB (QD=1) :    11.185 MB/s [  2730.7 IOPS]
   Random Write 4KB (QD=1) :     8.582 MB/s [  2095.1 IOPS]
   Random Read 4KB (QD=32) :    93.620 MB/s [ 22856.3 IOPS]
  Random Write 4KB (QD=32) :    72.302 MB/s [ 17651.8 IOPS]

  Test : 500 MB [N: 0.0% (0.0/10736.2 GB)] (x3)
  Date : 2015/06/24 20:27:17
    OS : Windows 8.1  [6.3 Build 9600] (x64)

First I tought the issue would be the local network, but it worked well with the previous setup in the same network at 110 Mbyte/s (read & write @ 110MBs). I can not explain the gap of 10 MByte/s between read and write it shows now. I thought - in case the speeds show different values - the read speed would be faster than write...

To be sure about the network I run iperf:

Code:

bin/iperf.exe -c 192.168.1.14 -P 1 -i 10 -p 5001 -f g -t 60
------------------------------------------------------------
Client connecting to 192.168.1.14, TCP port 5001
TCP window size: 0.00 GByte (default)
------------------------------------------------------------
[200] local 192.168.1.11 port 51724 connected with 192.168.1.14 port 5001
[ ID] Interval       Transfer     Bandwidth
[200]  0.0-10.0 sec  0.96 GBytes  0.83 Gbits/sec
[200] 10.0-20.0 sec  0.97 GBytes  0.83 Gbits/sec
[200] 20.0-30.0 sec  0.96 GBytes  0.83 Gbits/sec
[200] 30.0-40.0 sec  0.97 GBytes  0.83 Gbits/sec
[200] 40.0-50.0 sec  0.97 GBytes  0.83 Gbits/sec
[200] 50.0-60.0 sec  0.97 GBytes  0.83 Gbits/sec
[200]  0.0-180.0 sec  5.79 GBytes  0.28 Gbits/sec
read on server close failed: Connection timed out.
Done.

My first question: Is 0.83 Gbits/sec the practical maximum of Gigabit Ethernet? I thought with GbE you could get nearly 100% aka >0.9[x] Gbits/sec? Is that typical for the supermicro mainboard?

Still this doesn't explain the gap between READ 85.682 MB/s in comparison to WRITE 95.979 MB/s to me.

The next thing I tested was the HDD speed:

Code:

dd if=/dev/zero of=testfile bs=1024 count=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 0.248443 secs (206083581 bytes/sec)

AND

dd if=testfile of=/dev/zero bs=1024 cou
nt=50000
50000+0 records in
50000+0 records out
51200000 bytes transferred in 0.087034 secs (588275934 bytes/sec)

I repeated the test several times with the same results.

I also checked the switch (TP LINK TL-SG1008D), changed the cables (all CAT 6A) and checked the cpu and ram stats in the web console of freeNAS (cpu never over 50%, ram always more than 5GB free).

At the moment I am very frustrated because for my large file transfers I need every MByte/s.
I have absolutely no idea where my bottleneck is...

Do you have any ideas what might cause this problems and how to get 'full' speed (> 100MByte/s) out of my freeNAS setup?

THANK YOU very much in advance!
Tobi

Some Screenshots of the web console:

 

[Scareh]

missing some info on your first build: the exact cpu/amount of ram.

Generally speaking cifs is cpu based, so if your cpu is lower "quality" with the I3 that might be part of the problem.
If you look around on the forums for speed optimalized builds, you'll notice they'll use mirrors instead of raidz's.
And if you really want to max out your connection, change to ftp, multiple connections possible there.

Thats about it for what i can offer you ;-)

 

[Tobi]

Hi Scareh,
thanks for your reply: The older built was with the same cpu and 8GB ram. Also as shown in the Screenshots the cpu is not at 'max'. In my opinion the cpu is not the problem.
I need Z2 for data security reasons. I know that stripes are faster in general. The pool was tested with around 200 MBytes/s (!!) with Z2 internal (as I wrote above)...

1. dd if=/dev/zero of=testfile bs=1024 count=50000
2. 50000+0 records in
3. 50000+0 records out
4. 51200000 bytes transferred in 0.248443 secs (206083581 bytes/sec)

 

[cyberjock]

Writes to the zpool will always be to RAM. So writes, will always be limited to other things (source hard drive), etc.

So therefore, all writes should be equal to or faster than reads, at all times. So there doesn't seem to be much to say about your read speed being slower than your write speed.

That being said, your iperf tests are only 0.83Gb/sec. That's a pretty poor test. You should be able to do 930Mb/sec at the minimum, with speeds more like 980+ very likely. Here's an iperf test between my TrueNAS and laptop...

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[ 4] local 192.168.88.123 port 5001 connected with 192.168.88.120 port 57436
[ ID] Interval Transfer Bandwidth
[ 4] 0.0-10.0 sec 1.09 GBytes 934 Mbits/sec
[ 5] local 192.168.88.123 port 5001 connected with 192.168.88.120 port 56115

The system was "in service" but should have basically been idle.

So it seems like your networking could use a little work. You're definitely missing out on some performance based on the iperf tests. Your tests show that according to iperf you've already lost 17% of the rated network speed. That's before protocol overhead, latencies from things like ZFS, the hard drives, etc.

 

[Tobi]

Thank you a lot cyberjock!
So it looks like my network is definitly the bottleneck. I will take a look what changed in the network since the time I got >110 MByte/s.
Good to have your stats to compare it that makes it easier to know what speeds i practical could expect.
Thanks again for your helpful answer! :)

 

[Tobi]

After a little break I had now time to investigate the problem a little deeper...
I booted the same computer system (my workstation, not the NAS) from a linux live disk and started iperf again (freeNAS was the same, nothing changed):
The results are as expected for a Gigabit Ethernet-system:
I did several tests (~15 tests) with standard parameters, the worst test was 941 Mbit/s the best 945 Mbit/s.
I also did a test with the onboard Realtek lan chip which gave me something about 937-940 Mbit/s which is still ok for me.
All in all very consistent results which are maxing out the ethernet speed limit. All test are done just with local switch and no internet connection to avoid background downloads, etc.
So far so good. I changed back to Windows. freeNAS-device again without changes since the 'linux tests' had been done.
I got the same poor results as described above. Speeds with a maximum of 830Mbit/s. Average: ~710Mbit/s. Some test dropped down to even 650Mbit/s. [Same conditions - no internet connection]
For me it looked like a driver problem. Thus I tried the Realtek chip, which should have another driver I guess. The results are almost the same.
->> That is not explainable to me...
I also uninstalled and reinstalled the drivers for the ethernet chips (both, Realtek and Intel).
I did this one time with manual installed drivers and one time with Windows automated install (autosearch for drivers).
This didn't change anything too.
The last option was a clean new install of Windows 8.1 on a currently not used HDD, which showed me the same results (+-10Mbit/s).
I have absolutely no idea what the problem is.
Does anyone use a
Intel EXPI9301CT PRO1000
on a Windows 8.1 machine? If yes - which speeds are you getting? Which driver are you using?
Please help me, I have not the slightest idea what it could be... Thank you for your help in advance!