Slow Gigabit Transfers

[AndrewPr]

Good Evening freenas forum

Before anyone flames me for not reading the FAQ's and countless other posts regarding this issue, I've read them and need help beyond what google can tell me.

Now I'm running a gigabit router, All cat 6 cables. I'm running a freenas 8.3 (latest stable version) on an AMD A4-5300 Chip on a MSI FM2-A85XA-G65 with 2xSeagate 3TB and 3xHitachi 3TB drives in a ZFS RaidZ pool. All drives on SATA 6gbps cables/ports with 16GB of 1600mhz ddr3. My data is currently on a DLink DNS-323 running Raid-0 2x 1.5WD black setup. I'm attempting to transfer files from the Dlink to the Freenas box over CIFS/Samba Shares on via My Win7 Machine. I had the Freenas server relegated to a 100BaseT and moved it to gigabit (I've checked the root console with ifconfig to have it at 1000BaseT w/ full duplex now) I was getting 9.7-10 MB/s previously now that its on gigabit I'm seeing 14 MB/s when i should realistically see 60-80-100MB/s I've been over all the CIFS settings designed to increase network/filetransfer speeds recomended in previous posts without positive results. I'm not a freebsd guru, so I haven't started really really digging into the core the OS. I ran a JPerf on TCP and UDP and seeing 500-600 Mbits/sec and 70 MB/s (tcp/udp) I've tried fooling with Jumbo frames/High MTU on client, NAS, and Freenas with no positive results either.

I'm contemplating on getting another set of 16 gigs of ram and seeing if that helps, but I'm looking for the free fix before I shell out more money for ram.

I'm really not sure what to do at this point, so any suggestions would be happily welcome.

I can provide logs, info, or anything else that may help evaluate the problem I'm happing

Thanks

-Andrew

 

[HolyK]

What kind of "Now I'm running a gigabit router" do you have? Are the ports "switched" or "bridged"? Thing is, if they are "bridged", that crazy data transfer is going through router CPU and it could burn it to 100% .

Basically ... check your router stats for CPU load during transfer

 

[cyberjock]

Well, I'll definitely give you credit. You tried all the easy stuff without success. While not likely to solve your problems, an Intel NIC performs FAR better than anything else. Judging from your 500-600Mbit/sec with iperf I'm going to guess you are using Realtek. Everyone that has a Realtek NIC gets those speeds. Of course, that's not your problem(as you'd be much happier with 50MB/sec) so lets get to work.

Comments of stuff off the top of my head in no particular order:

1. Do you have compression or dedup on? If you do those can be performance killers.
2. Can you provide system specs? You didn't provide much of anything or even mention if you are using ZFS or UFS or what vdevs you have.
3. What is the output of a dd if=/dev/zero of=/mnt/(zpool name here)/temp bs=2048k count=10000?
4. Have you run SMART tests on your hard disks? A single failing disk can cripple a zpool.
5. Are you trying to drag and drop straight from the DNS-323 network shares to FreeNAS? There can be performance issues going from a network share to a network share. If this is your situation you may have to tough it out and just let it copy slowly or copy the files locally and then to FreeNAS. What happens if you copy a file from your C drive to FreeNAS or vice versa.
6. For troubleshooting from this point on out I'd recommend you remove any tweaks, jumbo packets, etc you have done. I would expect that with your RAM and CPU and enable autotune you should get at least 50MB/sec, and most likely much more. Of course, without an Intel NIC you may have issues getting more than 60-70MB/sec(which is typically the max for Realtek).
7. This is a stupid question, but are you using the 64 bit version? Quite a few people have been surprised to find that they have this great system with 8GB+ of RAM but then installed the 32 bit version. Of course, if you are using the 32 bit version you are only using 4GB of RAM, which also disable ZFS prefetch(which kills ZFS performance).

I've found jumbo packets really don't add much value, but they can cause serious complications if all of your network hardware doesn't support them. They're really not fun to use except in very specific situations where you know for a fact that for your situation it will provide great performance and all of your network hardware supports it. Years ago they were nice because it could really cut down on CPU usage doing network checksums and stuff, but now CPUs are so powerful(and many NICs do the calculations on the NIC processor) it really doesn't matter much.

Just as an example a similarly powered system with only 12GB of RAM I have seen can do over 100MB/sec. So you should be able to do pretty darn well. Something is definitely wrong and I wouldn't say more RAM is your solution.

Edit: I'm a dumbass.. so I had to do some edits.

 

[pete_c20]

I've had pretty much the same problem trying to initially fill the FreeNAS system from a windows system. I played around with this on and off for 4 days trying to speed it up. That 14MB/s seems to be a magic number. Here's my setup:
FreeNAS (FreeNAS-8.3.0-RELEASE-p1-x64 (r12825)) end:
Supermicro x9SCM-F v2.0b BIOS, E3-1220Lv2 2.3GHz Xeon, 16GB ECC DDR3 RAM, 10 x 500GB RAIDZ2. LSI9211 HBA.
Windows end:
WinXP Pro 32 SP3. Intel DP55KG motherboard. 12GB ram but only 4GB good to XP. 1x 120GB SSD system & pagefile. 2x 2TB HDDs on SATA storage.

Usually connected with a HP Procurve 1810G-8 J9449A switch (1GB/s on all ports). All leads are Cat5e, some are bought in, some have been made by me from known spec cable. Removing the switch (its all static IP) and connecting the Win PC to the FreeNAS back to back (and swapping / changing leads) had no effect on speed. Putting in an Intel 1000PT dual port 1GBit/sec NIC and disabling the onboard motherboard NIC on the windows PC had no effect. The Supermicro NIC on the FreeNAS end seems to behave fine on other systems. Delivering files from the SSD made no difference. Delivering very large 1Gbyte+ files made no difference. I've never seen the FreeNAS CPU come even close to bottlenecking, usually even under load, it's below about 10% CPU.

I think this is possibly windows problem but I can't prove it. Here's my thinking -
Using a windows RSYNC client -
The magic number of 14MByte/s transfer rate appears all the time for me running an Rsync client (in this case QTDSync v0.6.18 beta). Only on extremely rare (maybe 0.01% of the time) occasions will it burst faster (about 70MByte/s) on large files, and why it does it then I have no idea.
Using a windows FTP client.
Filezilla version 3.6.0.2 - This will transfer faster, upto about 50-70Mbytes/sec but has a problem where queueing a lot of files (some say 5000+, or is it 50000+?) leads to it severely slowing down. So I saw 50-70Mbytes/sec to start with while it queued more files in the background, then the speed dived.

Using Explorer or SyncToy with CIFS -
I found CIFS to be the fastest of the the lot for a single service. It will usually managed 50-70MBytes/sec with either a straight file copy or using SyncToy. Microsoft's Synctoy does work very well and its very easy to set up. Using explorer was okay until it aborted for some reason and trying to tell what was copied and what wasn't, wasn't easy. The aborting was usually down to a windows resource issue but what underlay that error I'm not sure, See below for possible work rounds.

There is a combination that I found to be the fastest of all, providing you have effectively 2 sources of data (ie Data on at least 2 HDDs needing to head towards the FreeNAS system, and thats to run an rsync service to transfer one HDD and then use CIFS to handle the other. Doing this worked very well. The FreeNAS system took it all in its stride. Again I saw the magic 14MBytes/sec figure appearing. If you look at the graph of the interface traffic attached for the FreeNAS NIC , that 100Mbits/sec baseline is the RSync client running. All the Peaks above that are the CIFS transfers. Running CIFS at the same time doesn't slow the Rsync down at all. Click on it to see it bigger.



A couple of other notes -
Sometimes ultimately unexplainable (to me anyway!) improvements in speed could be had by increasing the windows pagefile to silly sizes, especially if you're trying to transfer large files. Normally this system runs with no pagefile at all. Allowing the page file to grow to about 50GB didn't speed it up but often stopped things grinding to a halt or crashing.

The DLink DN-323 is not known for its speed. I have an unused one here aswell as a QNAP TS-110 and the FreeNAS system has walked all over the QNAP. Okay it's a bottom end QNAP but there's been many software issues with it. It seems half-baked. I think you'll like the FreeNAS when it gets going.

I used both CIFS and RSYNC at the same time to fill the FreeNAS system, This is doable due to having 2 sources of data in the source system, otherwise the head seek in a single HDD system would just slow it all up again, and then RSYNC to check the system out as a whole. Rsync is much faster in check mode than transfer mode.

Hope that gives you some food for thought.

If you solve it please do post a solution!! ;)

 

[AndrewPr]

Upon further testing last night I transferred a 1+ Gig file from my Win7 Rig (Raid-0 840pro's) at about 110MB/s So looks like the bottleneck is the DNS-323. Since i'm running a raid-0 on the DNS-323 I don't think i can pull the drives and stuff them into my Win7 rig.

Looks like its the Windows Share -> Windows Share issue described. The DNS-323 has a built in FTP server, so I'm going to try to use an FTP server to copy the last of the files on the DNS and see if that lets me get the files off in a more expedient manner.

Output from Freenas DD Test

[root@freenas ~]# dd if=/dev/zero of=/mnt/Server1/temp bs=2048k count=10000
10000+0 records in
10000+0 records out
20971520000 bytes transferred in 44.404461 secs (472,284,078 bytes/sec)​

I picked up a Intel Nic from amazon that i'll have tomorrow just to ensure super speeds.

Thanks for all the help.
-Andrew Prokopiak

 

[cyberjock]

Well, looks like you identified your issue. Congrats!