Slow transfer speed, but good hardware. Need some advice.

[HHawk]

Okay, I have been using FreeNAS for some time now.
Started with FreeNAS 7 almost 2 years ago and have been using FreeNAS 8 ever since it came out.

The problem is, is that my transfer speeds are quite low, or I am expecting to much from it.

I am currently running: reeNAS-8.0.4-RELEASE-p2-x64

The NAS is build from the following hardware:
- AMD Phenom(tm) II X4 955 Processor
- 16 GB RAM
- 6x WD 2 TB (WD20EARS) in Raidz2
- 4x 1000 MBit Intel Network cards in LACP

Connected through a LACP compatible switch (D-Link DGS-1210-16)

Anyways, I know there are many, many topics relating to bad transfer speeds, but I find it slow. Yesterday it only did about 5 MB/s, after some fooling around and rebooting it did 30 MB/s.

I really have no clue what I should do. I know it was better a past few weeks ago (doing about 50 MB/s or so), but still it seems to me it's on the low side, given the current hardware and connections.

I tested it several times with my game PC, which is really fast enough (and has 2x SSD's in RAID0) and has 2x 1000 MBit connections teamed (dynamic).

I tried transfering files through NFS and CIFS, but both show slow speed.

Maybe I should do some ZFS tweaking? Or could it be something else.

I really have no clue and searching through the many posts on this forum, it didn't provide me with a solution. :(

 

[paleoN]

First lose all the LACP. I doubt you need it, you are not seeing a benefit from it as is and it gets in the way of testing. Remove the teaming from your gaming rig as well. Then try the CIFs settings in particular in this [post=28793]thread[/post]. Some of the kernel tunings may help as well. Try the CIFs settings first though.

What's slow uploading to the NAS or downloading from it? Have you tried FTP? I get near line speed FTP transfers using FileZilla on my desktop. You could use FTP as your control assuming it works well for you.

 

[Joshua Parker Ruehlig]

I doubt it's your ram or cpu, so I'd think it's either your disks, network/client.

Test disks ~ Use dd from /dev/zero to your zpool with a good block size, there's some nice thread on this stickied. Should get around 4xspeed of 1 drive.
Test network ~ transfer file from ram in client computer to freenas, and vise versa. I know how to do this in linux with tmpfs, not sure about windows. Make sure to use a client with a decent cpu, and b connected through only 1 switch. Like paleoN said you might wanna ditch LACP till you can isolate where the problem is. Also make sure to test multiple protocols, CIFS/NFS.

Good luck

 

[HHawk]

Thanks for the great answer guys. I already looked through the thread and will make some changes, especially regarding CIFS.

I also found drawing, which I made a few months ago, about how everything currently is configured. It should be still up-to-date except for the fact both routers are now on Tomato USB firmware: http://www.mvdl.info/network_config.jpg

But I will check and apply the settings from the mentioned thread. I really appreciate the help, especially considering the many threads around there, but that is also the problem. There are to many threads about slow perfomance and transfer speeds that it's hard to find an useful one. Maybe some moderator should sticky the best tweaks?

I did a write bench:

WRITE:
dd if=/dev/zero of=/mnt/storage/benchtest/bench.mark bs=2048k count=50k
107374182400 bytes transferred in 68.041833 secs (1578061281 bytes/sec)
= ~1505 MB/s

READ:
dd if=/mnt/storage/benchtest/bench.mark of=/dev/null bs=2048k count=50k
107374182400 bytes transferred in 45.101823 secs (2380706047 bytes/sec)
= ~2270 MB/s

I have enabled ZFS compression (lzjb), so probably the results are not real-world performance?

Anyways, I will check the tweaks and apply them where needed.

Thanks once again for your input guys! Highly appreciated.

 

[survive]

Hi HHawk,

FreeNAS includes a program called "iperf" whose sole purpose is to blast packets out through a network interface to another iperf client & report the speed. Grab a copy of the iperf windows client, put it on the gaming PC & see how the connection works.

Your benchmark results look real nice, but that's because you are writing out a bunch of zeros which compress real well. Turn off compression and re-run to get a more representative number.

-Will

 

[HHawk]

Okay I will re-test it during the weekend without compression.

On the other side I just tested iperf.

NAS with iperf -s
And gamepc with iperf -c 192.168.0.100
Results in: ~473 Mbits/sec

When I run iperf -c 192.168.0.100 -r
Results in:
~557 Mbits/sec
~886 Mbits/sec

The first value seems a bit low?


The other way around (gamingpc = iperf -s)
Results in:
~880 Mbits/sec
No second value because my gamingpc gives me an exception "STATUS_ACCESS_VIOLATION" for some reason.

//update

I ran another test with NAS using iperf -s
And my gaming PC iperf -c 192.168.0.100 -d
Results in:
56.6 Mbits/sec
894 Mbits/sec

56.6 Mbits/sec ?????

Another try results in:
71.4 Mbits/sec
902 Mbits/sec

 

[paleoN]

56.6 Mbits/sec ?????

Another try results in:
71.4 Mbits/sec
902 Mbits/sec

Have you disabled LACP in FreeNAS, your gaming PC and the switch? Try running iperf after you do.

 

[cyberjock]

Honestly, if you are only running a few users, LACP won't do anything to increase your performance. I learned this after spending a few hundred dollars on hardware at home. It turns out that one link can only go to one ethernet port for source or destination. If you had 2 dual port NICs on 2 different machines, you cannot use LACP to get 2Gb. You would have to make 2 separate connections. Of course, you run a chance that both connections will randomly get assigned to the same NIC on one end, in which case you STILL have only 1Gb connection. If you want to maximize your performance, setup the ethernet ports independently and have different machines access different IPs on your server.

Anyway, here's what I did for my dual Intel NIC.

One port for the home LAN, and one for my main desktop.

My main desktop was directly connected to the second NIC on my FreeNAS server.

The FreeNAS NIC had the following settings: mtu 9000 -rxcsum -txcsum

I also made the following settings to CIFS service:

socket options = TCP_NODELAY
socket options = IPTOS_LOWDELAY
socket options = SO_RCVBUF=1048576
socket options = SO_SNDBUF=1048576

I have no idea why disabling the transmit and receive checksums increased performance, but they did. I get a consistent 120MB/sec+ to and from the server to my desktop.

 

[HHawk]

Sorry didn't forget about the topic, but had some bad news recently and that is taking up most of my time at the moment. Will report back as soon as I have some time to do some more testing. Sorry and I hope I didn't offend anyone here who provided help.

 

[paleoN]

No worries HHawk. Just let us know when you get back to it.

 

[tmstone835]

I noticed your comment on the checksums. This has been a problem since this feature started becoming available many years ago. It sounds good in theory but, in practice, offloading checksums to the NIC rarely works properly. Both Intel and Broadcom have only recently fixed most of the problems in their current drivers and chipsets so it isn't worth messing with. The original reasoning was to remove the interrupt load from the main CPU of the computer. That might have been a noticeable problem on old slower processors from 10 years ago but it is barely detected on modern fast CPUs. I have seen 25% throughput gains on some systems by disabling checksum offloading.

 

[HHawk]

No worries HHawk. Just let us know when you get back to it.

Well sorry for the really late feedback. Like mentioned had some really bad news. Oh well...

Anyways, recently I noticed a problem with one of the network cards. Watchdog was giving me some errors / timeouts. Finally found out which network card it was. I must admit it was difficult to tell which one it was, since they are all the same brand and type, but unplugging the cable helped to indentify the one who was causing problems and replaced it.

I tested it a few times lately while copying files (FreeNAS is still using LACP, my PC is not anymore). Seem to have improved things.

I recently ordered a Quad Port ethernet card for my FreeNAS to replace the (1x Dual Port and 2x Single Port cards). The Dual Port network card will go into my personal PC (using this one, instead of onboard controller crap). Will probably improve things as well.

Still running for the time being on 8.0.4 (p3), until it's easier (for me) to use plugins like Sabnzbd, Couchpotato and Sickbeard. I looked into it, but I find it (still) a bit difficult to understand it completely.

When I going to upgrade to the latest FreeNAS, I will also low-level format all harddisks (after backing up my stuff) and redo several things and I am thinking of doing 2 hardware upgrades (maybe).

I want to increase my memory from 16 GB to 32 GB RAM. Why? For ZFS and RAM isn't that expensive anymore nowadays. Don't know if I will get a performance increase.

Another thing I am currently considering is adding a Solid State Disk or a PCIE based SSD (if it's supported?) for ZFS caching and such and maybe move the ZIL to it? And would it also be possible to install FreeNAS on the SSD as well? Or does this have serious flaws? I only scanned through the information.

Anyways, just 2 things I am currently considering.

And sorry once again, for the late reply back. :(

 

[paleoN]

Seem to have improved things.

A failing network card would certainly slow things down. I take it you are happy with the speeds now?

When I going to upgrade to the latest FreeNAS, I will also low-level format all harddisks (after backing up my stuff) and redo several things and I am thinking of doing 2 hardware upgrades (maybe).

Are you planning on changing your zpool config?

I want to increase my memory from 16 GB to 32 GB RAM. Why? For ZFS and RAM isn't that expensive anymore nowadays. Don't know if I will get a performance increase.

You may not get much of one depending on how hard you are hitting your NAS, but like you said RAM is fairly cheap.

Another thing I am currently considering is adding a Solid State Disk or a PCIE based SSD (if it's supported?) for ZFS caching and such and maybe move the ZIL to it?

You could always get a Fusion-I/O card. Of course they are rather expensive and pointless until you upgrade to 10GigE networking.

More importantly I wouldn't run a ZIL until 8.3.0 is released with ZFS version 28. Otherwise if you lose the ZIL drive(s) you lose the entire pool.

And would it also be possible to install FreeNAS on the SSD as well? Or does this have serious flaws? I only scanned through the information.

You could, but what's the point? The main "flaw" is that it takes up a SATA port that could be used for something else. Personally, I keep a couple of USB thumbdrives on hand, regularly backup the FreeNAS config and rotate them on upgrades. Plus if one did fail I would just swap it with the other I have.

 

[HHawk]

A failing network card would certainly slow things down. I take it you are happy with the speeds now?

Are you planning on changing your zpool config?

You may not get much of one depending on how hard you are hitting your NAS, but like you said RAM is fairly cheap.

You could always get a Fusion-I/O card. Of course they are rather expensive and pointless until you upgrade to 10GigE networking.

More importantly I wouldn't run a ZIL until 8.3.0 is released with ZFS version 28. Otherwise if you lose the ZIL drive(s) you lose the entire pool.

You could, but what's the point? The main "flaw" is that it takes up a SATA port that could be used for something else. Personally, I keep a couple of USB thumbdrives on hand, regularly backup the FreeNAS config and rotate them on upgrades. Plus if one did fail I would just swap it with the other I have.

Yeah I am happy enough with the current speeds. I will give LACP, on my PC, another go, when I have received the Quad Port and my PC will have the Dual Port. Can always try, right?

Nah, I am not going to make any changes to my zpool, but since I know more about Windows than FreeNAS (or FreeBSD), I think it would be easier for me to low-level format the harddisk and check them on their current condition. Yeah, may sound stupid, but for me it's more understandable than FreeNAS / FreeBSD. ;)

Okay I will take the RAM-advice into consideration.

I was thinking about a OCZ RevoDrive X2 PCI-Express SSD 100GB in first place. But I really have no clue on what amount of space will be used by FreeNAS and ZFS. But apparently it's useless since I don't have (a) 10 GigE or (b) it's currently dangerous with ZIL. Uhmz... Maybe I will better wait.

That's why I was investigating the option for a PCIE based SSD, so it won't take up the SATA port, but I don't know if I can run FreeNAS on the same PCIE SSD in combination with ZIL (in the future), along with caching.

Regards,
HHawk

 

[paleoN]

Nah, I am not going to make any changes to my zpool, but since I know more about Windows than FreeNAS (or FreeBSD), I think it would be easier for me to low-level format the harddisk and check them on their current condition. Yeah, may sound stupid, but for me it's more understandable than FreeNAS / FreeBSD. ;)

IMHO, smartmontools is the best for checking the harddisks. It doesn't have any incentive to pass the drives. However, writing to as well as reading the drives is a more thorough test. What Windows tool do you low-level format with anyway?

Okay I will take the RAM-advice into consideration.

Which advice? I thought I came down on upgrading the RAM? Another way to say this is you will see a performance increase by upgrading your RAM. The question is how much will you notice it? Depending on your workload it might be quite a bit or just a little.

That's why I was investigating the option for a PCIE based SSD, so it won't take up the SATA port, but I don't know if I can run FreeNAS on the same PCIE SSD in combination with ZIL (in the future), along with caching.

FreeNAS uses 4 primary partitions. So, nothing else can use the rest of the drive. Not to mention I don't think anything boots off of a PCIe SSD, but I could be wrong. PCIe SSD's don't have a normal hard drive interface and require special drivers to use.