Freenas 9.1 - Slow read performance over LAN

[TenzoR]

I've been trying to figure out why my read performance over LAN degraded after installing FreeNas 9.1

My server is dedicated to Freenas, nothing else is running (no jails). There's really only one device accessing the Freenas server as well. I've tried both onboard NIC ports. Here's the specification of the server

Processor: Intel i3-3220
Motherboard: Intel S1200KPR
Memory: 16GB ECC DDR3-1600
Zpool: Raid1 with 2x2TB

The only tuning I've done is enabling autotune. My write performance over LAN from PC to FreeNas is about 80-100MB/s+. That did not change from going from 8.3 to 9.1.

My read performance over LAN from Freenas to PC went from 80-90MB/s down to 20-30MB/s going from 8.3 to 9.1. I'm using a 3GB ISO image and copying to a SSD drive.

I am almost certain this nothing to do with CIFS as I did a test using FTP and got similar performance.

Here's the autotuned generated settings:
kern.ipc.maxsockbuf 2097152
net.inet.tcp.recvbuf_max 2097152
net.inet.tcp.sendbuf_max 2097152

vfs.zfs.arc_max 10539046102
vm.kmem_size 11710051225
vm.kmem_size_max 14637564032

Any suggestion would be helpful. I can't go back to 8.3 as I've updated the Zpool (oops) :(

 

[joeschmuck]

Yea, that is too bad you updated the zpool, I've been running 9.1.1 since it came out and I still haven't updated the zpool. I likely never will since it locks me into FreeNAS 9.x. The good thing is you have a small amount of data so you could save the data off on a different drive and then recreate the pool under FreeNAS 8.3 if you had to. It gets harder when you have 7 to 10TB of data to do something like that.

Just a stupid question, are you positive the read performance only occurred after the conversion from 8.3 to 9.1 ? Meaning you didn't do anything else?

Could you post the results of "zpool list" and "zpool status", want to see if your pool is healthy.

Here is a link to some testing I did, I'm really directing to you perform Test #1 only and request you post your results.
http://forums.freenas.org/threads/intel-nic-vs-realtek-nic-performance-testing.10325/

 

[TenzoR]

Thanks for your help. I've only tested the 8.3 when I first installed it. I did the same test on 9.1 when I installed it. In addition I did not perform an upgrade because well there's really nothing I setup except for CIFS and autotune. Pretty easy to replicate. Really this server is meant to serve media and backup important files. I just like it to perform as best as it can.

Here's the Zpool List:
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
xxxxxx 1.81T 1.48T 341G 81% 1.00x ONLINE /mnt

Here's the Zpool status
pool: xxxxxxx
state: ONLINE
status: The pool is formatted using a legacy on-disk format. The pool can
still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the
pool will no longer be accessible on software that does not support feature
flags.
scan: scrub repaired 0 in 3h11m with 0 errors on Sun Oct 27 06:11:59 2013
config:

NAME STATE READ WRITE CKSUM
xxxxxx ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
ada1 ONLINE 0 0 0
ada0 ONLINE 0 0 0

test#1 Result
dd if=/dev/zero of=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 987.627114 secs (108719355 bytes/sec)

dd of=/dev/zero if=tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 614.891204 secs (174623058 bytes/sec)

 

[joeschmuck]

So it looks like your pool is in good health but data transfer speeds seem a bit low in the test#1 results, however that is still faster than a Gigabit connection. I would say it's due to file fragmentation but ZFS fragments files just by the way it works so that shouldn't be a consideration. I know once you hit the 10% or less in capacity ZFS goes into a safe mode where it slows things down considerably but you are at 81% so that doesn't make sense either.

You could go back to 8.3 and verify the speeds once again. Of course if you cannot figure out what is happening then you may end up at 8.3 anyway.

Another thing you could try, of course this is if you feel like destroying your pool... Destroy your pool and create a single drive ZFS pool. Try to write the large file to the NAS, rename it, then copy it back to the computer. Check what your speeds are.

The reason your write performance is so good, it's your 16GB of RAM. Your data flows into the RAM and then starts writing the data.

Also one other thing you could try, with 8.3 running, create the pool you want, maybe your RAID1, then load up 9.1 but do not upgrade the pool. See if that works.

I know there is a lot you may need to try but I do not have an answer to exactly what is causing your problem. I really would like to know the cause so if you do find something, please post it.

 

[TenzoR]

Thanks again for your input. I suppose the only major difference is the pool is actually fuller compare to when I tested in 8.3 (e.g. fresh pool with no data). However, I seem to be remembering testing NAS4free with FreeBSD 9.1 and I had a very similar performance issue. One of the reason I went with Freenas.

As for your suggestion of further testing. I will definitely run them once I get some new HDs. I'm waiting for HDs to go on sale during Black Friday. Once I get 2 additional drives, I will run the additional tests.

Right now, the speed from 9.1 is still usable. I can stream 720p video without any issue so I rather figure out the issue before tossing in the towel.

Also, something I noticed from the zpool status

The pool is formatted using a legacy on-disk format. The pool can still be used, but some features are unavailable.
action: Upgrade the pool using 'zpool upgrade'. Once this is done, the pool will no longer be accessible on software that does not support feature flags.

Does this mean I can go back to 8.3? I could've swear I did the zpool upgrade.

 

[joeschmuck]

I'm waiting for HDs to go on sale during Black Friday. Once I get 2 additional drives, I will run the additional tests.

Me too, I set the calendar reminder up 2 weeks ago. I also picked a price that I would purchase the drives at without waiting any longer since there are sales flooding the market everyday. For instance, I'm looking to purchase a 2 TB hard drive as a spare for my FreeNAS system, could be the WD Red or the Seagate ST2000VN000, either one will be fine and both offer a 3 year warranty. The Seagate drive has a little higher rotational speed so latency is improved a little bit but not enough to say they are any better than the WD drives for my purposes but I do like the idea of a mixed drive set. I may purchase a 3TB drive if the price is better, for instance two days ago a WD Red 3TB drive was $109.00 delivered but that was after a $30.00 mail-in rebate. Very good price but I hate mail-in rebates, I'd rather have an instant rebate. So my low price is $80.00, if I see a drive for that price delivered to my door, I'll be purchasing it. I also need a 1TB 7200 RPM drive for my main computer for Virtual Machine data. When I see one go on sale that I want, I'll pick one up.

Also, something I noticed from the zpool status

Does this mean I can go back to 8.3? I could've swear I did the zpool upgrade.

I believe I read somewhere that you see stuff like this once you have upgraded to version 5000 but try this first...
Type "zpool get version" and if it says it's '28' or less then you are fine. If it's a "-" (dash) then you are at 5000 and you cannot go back directly to 8.3, you would have to destroy the pool.

And to let you know, even if you do use FreeNAS 9.1, you can leave your pool at version 28 and all be fine. I just believe that if you create your pool under 9.1 the it will be version 5000 so I'd create my pool under 8.3 first and then move over to 9.1 and auto-import the pool. What this does is gives me options to use other software like NAS4Free if I ever wanted to or go back to 8.3. Let's face it, 8.3 is a fine version and if it works, there is really no "NEED" to upgrade. I think it depends on what someone is using it for. I use mine as a backup device and to stream movies. I may open an FTP port periodically for one of my sons to grab a file from me, otherwise that port is fire-walled.

Not sure if you have used virtual machines much but VMWare Player is free and you can create a virtual machine to install FreeNAS on and play with it. You can configure the hell out of it, add as many hard drives as you want, create any variety of pools, upgrade, change versions, whatever you want and not do any harm to your real system. Whenever a new version of FreeNAS hits the street, it's tested on a VM first and I'll run it for days or longer to test it out. You cannot test everything out such as compatibility with your NAS hardware but it gives you a very good idea on how it's going to work. You might want to give it a try if you haven't already. I have VMWare Workstation and I can save my system at any point in time and revert back to it so this makes it handy when testing. VMWare Player doesn't have that feature but for free it's still a very nice tool. It's just a thought.

Take care,
Mark

 

[TenzoR]

Alright, I picked up my two HDs, Toshiba 2x3TB 7200RPM 64MB Cache. Should be very similar in performance to my current Seagate 2x2TB 7200RPM 64MB Cache. Not exactly the same but should give me a rough idea of the difference between the two version of Freenas.

I've downloaded the latest Freenas 8.3.2 and redownloaded Freenas 9.1.1 Release. For testing purpose, should I be picking up two of the same USB flash drive to be consistent?

I'm planning to do a fresh install, setup CIFS and enable autotune. I'll run the same dd tests on both to compare. Finally I'll do a large file transfer back and forth using CIFS, FTP. Should I try anything else?

 

[joeschmuck]

When copying files back and forth, change the name of each file before each transfer so it isn't cached in RAM. Also if you really want to do a comprehensive test I'd personally run the Intel NAS Performance Tests (reference my link in post #2 above) but if you are just concerned about CIFS, well copying a folder of many small files each direction will give you stats for small files and then one large file each direction should do it. Small files is where you should see the worst throughput which is normal for all systems.

Good luck.

 

[TenzoR]

Well, I was wrong. It turns out, I had to turn on AIO in 8.3.2 to obtain those speed. If AIO was off, speed drops drastically (10~Mbps). So this means 9.1.1 was actually an improvement!

However, I don't really get why AIO made such a big impact ... it's only one client (my PC) ?

Also, I'm happy to report my old pool was not upgraded yet :)

 

[joeschmuck]

Keep your results in mind when/if you upgrade to 9.2.0 because AIO I don't believe is even an option and it's turned off. Maybe it's available under Advanced settings, I'd have to look but just keep that in mind.

Glad you figured it out.

 

[TenzoR]

The more I think of it, the more I think it's not freenas but something else is crapping out.

I did an iperf test and the results are somewhat troubling

NAS iperf server|PC iperf client
bin/iperf.exe -c 192.168.x.x -P 1 -i 1 -p 5001 -f k -t 10 -T 1
------------------------------------------------------------
Client connecting to 192.168.x.x, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[156] local 192.168.x.x port 59299 connected with 192.168.x.x port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 103792 KBytes 850264 Kbits/sec
[156] 1.0- 2.0 sec 112448 KBytes 921174 Kbits/sec
[156] 2.0- 3.0 sec 113368 KBytes 928711 Kbits/sec
[156] 3.0- 4.0 sec 113904 KBytes 933102 Kbits/sec
[156] 4.0- 5.0 sec 111776 KBytes 915669 Kbits/sec
[156] 5.0- 6.0 sec 112552 KBytes 922026 Kbits/sec
[156] 6.0- 7.0 sec 113864 KBytes 932774 Kbits/sec
[156] 7.0- 8.0 sec 113528 KBytes 930021 Kbits/sec
[156] 8.0- 9.0 sec 108608 KBytes 889717 Kbits/sec
[156] 9.0-10.0 sec 113128 KBytes 926745 Kbits/sec

PC iperf server|NAS iperf client
bin/iperf.exe -s -P 0 -i 1 -p 5001 -f k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
OpenSCManager failed - Access is denied. (0x5)
[248] local 192.168.x.x port 5001 connected with 192.168.x.x port 40178
[ ID] Interval Transfer Bandwidth
[248] 0.0- 1.0 sec 37314 KBytes 305677 Kbits/sec
[248] 1.0- 2.0 sec 17337 KBytes 142027 Kbits/sec
[248] 2.0- 3.0 sec 28752 KBytes 235537 Kbits/sec
[248] 3.0- 4.0 sec 42375 KBytes 347137 Kbits/sec
[248] 4.0- 5.0 sec 21943 KBytes 179758 Kbits/sec
[248] 5.0- 6.0 sec 9771 KBytes 80044 Kbits/sec
[248] 6.0- 7.0 sec 12544 KBytes 102760 Kbits/sec
[248] 7.0- 8.0 sec 8131 KBytes 66607 Kbits/sec
[248] 8.0- 9.0 sec 48313 KBytes 395784 Kbits/sec
[248] 9.0-10.0 sec 63553 KBytes 520628 Kbits/sec
[248] 0.0-10.0 sec 290048 KBytes 237322 Kbits/sec
Done.

Granted, my PC is fairly old, core2duo, realtek nic .. I wonder if that's the issue ...

 

[TenzoR]

Updated my LAN driver on the PC side ....

[248] local 192.168.x.x port 5001 connected with 192.168.x.x port 42841
[ ID] Interval Transfer Bandwidth
[248] 0.0- 1.0 sec 104819 KBytes 858676 Kbits/sec
[248] 1.0- 2.0 sec 86729 KBytes 710487 Kbits/sec
[248] 2.0- 3.0 sec 64766 KBytes 530559 Kbits/sec
[248] 3.0- 4.0 sec 103699 KBytes 849502 Kbits/sec
[248] 4.0- 5.0 sec 97579 KBytes 799365 Kbits/sec
[248] 5.0- 6.0 sec 82296 KBytes 674170 Kbits/sec
[248] 6.0- 7.0 sec 87554 KBytes 717243 Kbits/sec
[248] 7.0- 8.0 sec 111857 KBytes 916336 Kbits/sec
[248] 8.0- 9.0 sec 44162 KBytes 361773 Kbits/sec
[248] 9.0-10.0 sec 112789 KBytes 923965 Kbits/sec
[248] 0.0-10.0 sec 896256 KBytes 733438 Kbits/sec

 

[cyberjock]

If you have a c2d it could be that when CPU loading is low it can do full throughput, but background tasks that popup and consume resources are just enough to put you "over the top", thereby hurting network performance.

What I'd do if I were you is get a Linux Mint CD, boot it up via liveCD, then do some iperf testing with that.

 

[TenzoR]

If you have a c2d it could be that when CPU loading is low it can do full throughput, but background tasks that popup and consume resources are just enough to put you "over the top", thereby hurting network performance.

What I'd do if I were you is get a Linux Mint CD, boot it up via liveCD, then do some iperf testing with that.

Or ... I can use this excuse to get a brand new i7 with all the trimmings .... :evil:

But I'll give that suggestion a try before opening the wallet :)

 

[TenzoR]

So just wanted to update everyone. The problem was definitely my old core 2 duo. I've upgraded to a new i7 with the Intel i217v nic card. My read speed are more in line with the write speed. I guess the only other factor is the new 9.2.x with Samba4, but I doubt it.

 

[joeschmuck]

Glad you're happy. An I7 is a lot of overkill for this project but maybe you have other motives like upgrading your personal computer. You didn't post your speed results.

 

[TenzoR]

Updated iPerf

NAS iperf server|PC iperf client
------------------------------------------------------------
Client connecting to 192.168.x.x, TCP port 5001
TCP window size: 63.0 KByte (default)
------------------------------------------------------------
[156] local 192.168.x.x port 50466 connected with 192.168.x.x port 5001
[ ID] Interval Transfer Bandwidth
[156] 0.0- 1.0 sec 95072 KBytes 778830 Kbits/sec
[156] 1.0- 2.0 sec 93472 KBytes 765723 Kbits/sec
[156] 2.0- 3.0 sec 92608 KBytes 758645 Kbits/sec
[156] 3.0- 4.0 sec 93400 KBytes 765133 Kbits/sec
[156] 4.0- 5.0 sec 93264 KBytes 764019 Kbits/sec
[156] 5.0- 6.0 sec 96224 KBytes 788267 Kbits/sec
[156] 6.0- 7.0 sec 96472 KBytes 790299 Kbits/sec
[156] 7.0- 8.0 sec 96392 KBytes 789643 Kbits/sec
[156] 8.0- 9.0 sec 95088 KBytes 778961 Kbits/sec
[156] 9.0-10.0 sec 92648 KBytes 758972 Kbits/sec

PC iperf server|NAS iperf client
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 8.00 KByte (default)
------------------------------------------------------------
OpenSCManager failed - Access is denied. (0x5)
[296] local 192.168.x.x port 5001 connected with 192.168.x.x port 21411
[ ID] Interval Transfer Bandwidth
[296] 0.0- 1.0 sec 115050 KBytes 942488 Kbits/sec
[296] 1.0- 2.0 sec 114285 KBytes 936225 Kbits/sec
[296] 2.0- 3.0 sec 114447 KBytes 937553 Kbits/sec
[296] 3.0- 4.0 sec 114336 KBytes 936640 Kbits/sec
[296] 4.0- 5.0 sec 114306 KBytes 936397 Kbits/sec
[296] 5.0- 6.0 sec 114288 KBytes 936247 Kbits/sec
[296] 6.0- 7.0 sec 114363 KBytes 936859 Kbits/sec
[296] 7.0- 8.0 sec 114359 KBytes 936828 Kbits/sec
[296] 8.0- 9.0 sec 114341 KBytes 936681 Kbits/sec
[296] 0.0-10.0 sec 1144704 KBytes 937777 Kbits/sec

It seems my write to the NAS is a bit slower. But the read speed is definitely much more consistent and faster. I will run it again to see if enabling auto-tune makes a difference.