Help with network troubleshooting

[Daisuke]

Hi guys,

I posted a while ago my FreeNAS setup, where I had amazing speeds with 8.0.2 versions.
http://forums.freenas.org/showthread.php?1310-My-new-NAS-box-running-on-8-0-2

We are talking 115MB/sec UP's and 75MB/sec DOWN's through CIFS:

I have a Cisco SA520W-K9, hooked to a SG 300-10 switch.
Here it is what my RAID1 disks do in Windows:

Code:

> Disk  Sequential 64.0 Read                   96.18 MB/s        6.5
> Disk  Random 16.0 Read                       2.35 MB/s         4.3
> Responsiveness: Average IO Rate              3.16 ms/IO        6.3
> Responsiveness: Grouped IOs                  8.77 units        7.4
> Responsiveness: Long IOs                     16.91 units       6.7
> Responsiveness: Overall                      148.27 units      6.6
> Responsiveness: PenaltyFactor                                  0.0
> Disk  Sequential 64.0 Write                  123.03 MB/s       6.9
> Average Read Time with Sequential Writes     5.771 ms          5.7
> Latency: 95th Percentile                     18.573 ms         4.7
> Latency: Maximum                             42.131 ms         7.9
> Average Read Time with Random Writes         6.894 ms          5.4
> Total Run Time 00:01:53.96

So the bottleneck is not there. Same goes with my RaidZ2:

Code:

$ dd if=/dev/zero of=/mnt/nas/media/tmp.dat bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 661.295446 secs (154.84758091 MB/sec)

$ dd if=/mnt/nas/media/tmp.dat of=/dev/null bs=2048k count=50k
51200+0 records in
51200+0 records out
107374182400 bytes transferred in 393.194915 secs (260.430631638 MB/sec)

Running iperf from my Windows box to NAS, I get those results:

Code:

Client connecting to 192.168.2.6, TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  3] local 192.168.2.4 port 56956 connected with 192.168.2.6 port 5001
[ ID] Interval      Transfer      Bandwidth
[  3] 0.0-39.2 sec  3.00 GBytes   82.125 MB/sec

Does this means something is wrong on my network? I would appreciate any feedback and see how I can get the old speeds back, as nothing was changed on my network. I'm running everything through Intel NIC's.

 

[paleoN]

TECK I get less than stellar CIFs speed myself, 75MB/sec UP's and 65MB/sec DOWN's, compared to FTP in particular. I think I might play around with iperf myself tonight. In my case I know the network is fine as my FTP speeds are nearly at line speed UP & DOWN. I would try transferring the same files via FTP as a comparison.

 

[Daisuke]

That's what I get now also, 75MB/sec UP (instead of 115MB/sec) and 50MB/sec down (instead of 75MB/sec).
I used to get the high numbers on 8.0.2 release, once I moved to 8.0.4 the speed was cut in half.

Any advice from the network experts?

 

[paleoN]

Edited: I originally had some bad client tuning.

Some iperf results. All tests with matched TCP window size, 64K & 128K, on client & server.

PC - client
NAS - server

Code:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.06 MByte
------------------------------------------------------------
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.2 port 1155
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-100.0 sec  10935 MBytes   109 MBytes/sec

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.12 MByte (default)
------------------------------------------------------------
[  4] local 192.168.1.1 port 5001 connected with 192.168.1.2 port 1028
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-100.3 sec  11307 MBytes   113 MBytes/sec

PC - server
NAS - client

Code:

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.06 MByte
------------------------------------------------------------
[1872] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 53879
[ ID] Interval       Transfer     Bandwidth
[1872]  0.0-100.0 sec  11223 MBytes   112 MBytes/sec

------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 0.13 MByte
------------------------------------------------------------
[1872] local 192.168.1.2 port 5001 connected with 192.168.1.1 port 18900
[ ID] Interval       Transfer     Bandwidth
[1872]  0.0-100.0 sec  11223 MBytes   112 MBytes/sec

10GB file transfer via CIFs: UP - 75MB/s, DOWN - 60MB/s. :(

 

[Daisuke]

Use CIFS and transfer some large files, like 10GB. Let me know the speed you get.

 

[cyberjock]

Keep in mind that your source or destination may not be able to keep up with LAN speeds. If you are copying to a FreeNAS server from a slow hard disk, getting 75MB/sec may be all you will ever get because that is the limiting factor. When I do tests I try to use RAMdrives or SSDs that I know can far exceed LAN speed.

 

[Daisuke]

I understand... so even if I have decent disk results like the ones posted above, I could get bottlenecks?
From what I see, is my network that can push only 80MB/sec. I used to get easy 115-120MB/sec while using FreeNAS 8.0.2. The funny thing is that other people reported this issue also, so I was hoping that we can all work together and find out what is wrong.

 

[cyberjock]

If there was a bottleneck it may be hard to identify. For instance, if Samba has applied an update responsible it may be difficult to even identify since it is likely nobody working on FreeNAS has worked on Samba. Additionally, their fix might have been some kind of security fix and it just so happens your CPU just can't cut the mustard with the fix applied.

There could be a hardware issue resulting in the low performance or a hardware limitation.

My windows desktop at home connected to a Windows server get about 75MB/sec when it's antivirus is running. Disable the Symantec Endpoint Protection Network Protection and it will instantly go up over 100MB/sec.

The easiest answer to increase performance is to throw more powerful hardware at it. LOL. My friend's server uses an i3-530(first gen i7 at 2.93Ghz) and I can max out it's 3 NICs simultaneously. Being that the 13-530 is about 2-3 years old and can be had for $120 or so I always go that route when looking for a CPU for a FreeNAS server or something else in the same price range. I'm not an Intel fanboi but I've been using i7s since they came out and they just...rock. Really it is alot of "each man for his own" when it comes to optimizing your server. This is one of the advantages to Unix/Linux systems. Instead of being optimized for a "one size fits all" (aka Windows) you can optimize your server for your exact type of load. If you know what your server will be used for you should be able to tune it for your needs. Of course, there is always the possibility that your hardware just can't cut it.

Keep in mind that you are using Windows too. There is alot of "one size fits all" optimizations that could be hurting your performance with the latest version of FreeNAS versus an older version. I don't think I made a server with 8.0.2 aside from a VM to experiment with. 8.0.4 is when I started building servers with FreeNAS.

 

[Daisuke]

Honestly, that answer does not help with absolutely nothing. We are here to discuss FreeNAS network troubleshooting, not hardware performance or Windows Server (especially when I mentioned clearly that on previous software version I was getting proper numbers). Can anyone with FreeBSD networking experience chip in and post some advice what I could do to restore the previous numbers? Thanks.

Personally, I'm a Linux guy, so I don't know much about Windows.

 

[cyberjock]

You missed the point of my post. I was trying to say that small mundane things that someone might immediately exclude as a possible cause can make a big difference.

Your screenshots were Windows.. so my post was directed towards Windows users and some of the more mundane things(such as antivirus on Windows) that could be causing your performance to degrade. The only thing you really know for certain is that your performance is slower. Correlation is not causation. Assuming FreeNAS is the culprit for your slower performance isn't a bad place to start, but you can't rule out other possibilities. There's been alot of people complaining about low performance on all versions. Figuring out what YOUR limitation is harder than it may look. My limitation was a darn Firewall for Windows. Yours might be your crappy NICs. Someone else's could be the networking cable(yes, not all cables that are cat6 are created equal). Still someone else's could be that one machine is starved for RAM.

What YOU need to do is figure out where the weakest link is and start there.

You do know that there are 4 year degrees you can get specializing in Networking. It's amazing how many 'zomg my performance isn't line speed' threads there are, and how few of them actually get resolved. The few that do seem to be people that upgrade hardware or figure out one of their pieces of hardware has failed.

You may be able to fix it with a few tweaks. (Definition of "tweak": To adjust; fine-tune.) Fine tuning is for your exact situation. If it was broader and worked for everyone it would be the default. It's tweaking because its something you do on a case-by-case basis.

But continue to ignore me because I mentioned Windows and you're a Linux guy. The fact that you AREN'T a Windows guy is precisely why you have an uphill battle trying to optimize your LAN speeds. I wouldn't ask a Windows guy to optimize my Linux box.

Your first post showed screenshot of Windows so feel free to flog me for guessing your system from a ss. I did a few tweaks on one my of systems, and those settings are somewhere in one of my posts. Feel free to go through my posts and find it and see if it works for you. Good luck and feel free to post back with what worked for you.

 

[Daisuke]

What I'm interested (as well other users) is to look at the actual FreeNAS sysctl/Windows tuning and other Cisco network environment settings, not HUGE posts that contain no technical troubleshooting points. If you have any input on that, feel free to post some real suggestions.

 

[paleoN]

I updated my [post=34553]iperf post[/post] with the new, correct results. I was playing around with some client tuning and 8.3.0-BETA1 which threw off my initial results. Now iperf matches what I see with FTP transfers. Have you tried out any FTP transfers and are they maxed at 82MB/sec?

Hmm, I might have to try out 8.0.2 just to see if I get faster CIFs transfers as well.

Keep in mind that your source or destination may not be able to keep up with LAN speeds.

No shit? Do me a favor noobsauce80 and delete your posts in this thread. They are just cluttering it up. Both TECK and I do understand the basics here. If you actually read through his other post you would have known that his speeds were cut in half to & from the same machines. The only difference was the upgrade. In my case transferring the same files via FTP to the same destinations I get close to line speed transfers.

 

[Daisuke]

Hmm, I might have to try out 8.0.2 just to see if I get faster CIFs transfers as well.

If you can, that would be much appreciated. In this way we know for sure if is related to software. Here are the settings I have in CIFS:
Large RW support: Checked
Send files with sendfile(2): Checked
EA Support: Checked
Support DOS File Attributes: Checked
Homes auxiliary parameters: socket options = TCP_NODELAY IPTOS_LOWDELAY IPTOS_THROUGHPUT

Anything else is disabled. With the above settings, I was getting 115MB/sec solid in 8.0.2.

 

[cyberjock]

Here's my settings for 115MB/sec in 8.2

socket options = TCP_NODELAY IPTOS_LOWDELAY SO_KEEPALIVE SO_RCVBUF=2097152 SO_SNDBUF=2097152 IPTOS_THROUGHPUT
read raw = yes
write raw = yes
max xmit = 262144
getwd cache = yes
write cache size = 262144

Some are duplicates because 8.2 has different defaults, but I keep them anyway.

 

[paleoN]

Anything else is disabled. With the above settings, I was getting 115MB/sec solid in 8.0.2.

Tried with 8.0.2 with your settings and same results, UP - 75MB/s, DOWN - 60MB/s.

Here's my settings for 115MB/sec in 8.2

I wish it was that easy. I have similar settings in 8.2, but who knows maybe it will work this time. Nope, no difference for me. :(

 

[sunflashx]

I can do 1-2 gig file at 90+ MB/s. As soon as I jump up to a bigger file, like 4+ gig, transfers start out fast and then taper down to like 65-70 MB/s.

That seems to be pretty common, what settings would impact a longer/larger transfer like that?