Inconsistent LAN performance on FreeNAS 9.1

[draetheus]

Just recently built out my FreeNAS server and was doing some performance testing before copying over everything from my old drives and noticed that my writes were nearly half of my reads. I thought this was a bit strange for WD Black drives, so I ran a DD test and theoretical performance is great: 125MB/s read and write. Should be able to max out a gigabit connection just fine. So I figure its a Samba issue and start doing some iperf testing:

From server to desktop: 940 megabits per sec in every test
From desktop to server: 500-800 megabits per sec, changing in every test

That doesn't seem right. It must be my consumer grade gigabit gear then. I tried:
- Changing gigabit switches
- Changing CAT6 cables on both ends
- Purchasing an Intel Gigabit CT adapter and changing on both desktop and server
- Forgoing the switch entirely and setting up a direct crossover between desktop and server
- Booting Linux on my desktop, maybe its Windows?

Still got the same iperf results every time. Finally I decide to test out FreeNAS 8.3.2: 940 megabits per sec in both directions

What the heck is up with 9.1? I'm running server grade hardware that I thought was more than capable:

CPU: Xeon E5430
Mobo: Supermicro X7DCL-3
RAM: 16GB DDR2 ECC
Mirror: 2 x 1TB WD Black
RAID-Z1: 3 x 2TB WD Green

 

[cyberjock]

I don't have any obvious glaring reason why things aren't working right. You should be doing faster transfers than the 500-800megabit. I will tell you that firewall software sometimes has a hard time going from desktop to server. Might want to disable your antivirus and firewall temporarily and do a test. I had this problem myself and there was no "solution" except to disable it permanently or switch to a different program. I can't explain why sometimes the firewall has a problem and seems to have heavy CPU loading, but other times it doesn't seem to care. I couldn't even figure out what its doing. The only thing I could prove easily was that mid-transfer I could disable the firewall and within 3 seconds speeds would skyrocket to saturation speeds on my LAN.

I will say that I have had far better network performance with Linux Mint than I ever had with Windows.

You tried all the common stuff though. Are you sure the issue isn't your hard drives not being able to keep up? Have you tried using an SSD or a Ramdrive? In Windows you can get a free open source ramdrive program called imdisk http://www.ltr-data.se/opencode.html/#ImDisk

 

[draetheus]

I have no antivirus or firewall on my Windows 7 desktop other than built in one. Given FreeNAS 8.3.2 can max gigabit in both directions that doesn't seem like the issue.

I thought iperf was an in memory test? How would my drives affect its 500-800 megabit results?

 

[cyberjock]

I have no antivirus or firewall on my Windows 7 desktop other than built in one. Given FreeNAS 8.3.2 can max gigabit in both directions that doesn't seem like the issue.

I thought iperf was an in memory test? How would my drives affect its 500-800 megabit results?

Umm.. because I'm an idiot? Does that work? I was thinking that was actual file transfer speeds. I beg forgiveness from the FreeNAS gods!!!

Yes, it should be in-memory performance(basically).

So you are running Intel NICs on both sides and getting only 500-800mbit/sec? I assume you aren't trying to use PCI NICs.. right?

 

[draetheus]

Yep, PCIe Intel NICs on both sides, direct CAT6 cable crossover with static IPs configured, same TCP window size set in iperf (128k). 8.3.2 is consistently 940 megabit in both directions, 9.1.1 is only 940 megabit from server to desktop. From desktop to server is extremely dodgy. I feel like it might be a sysctl or something but I have little experience with FreeBSD in general.

 

[cyberjock]

Honestly, my first guess is some power management is coming into play. I know power management has affected my speeds in weird ways. Did you enable powerd?

 

[draetheus]

I never enabled powerd (unless its enabled on default install settings).

Ok so this is crazy. I was doing some more testing on 8.3.2, dropped back to using my integrated realtek on my desktop, still getting over 900 megabit in both directions. So just for the hell of it I reflash 9.1.1 back on the USB drive and test things out again. Now my iperf tests in 9.1.1 are over 900 megabits in both directions! So, I guess there was something weirdly corrupt with my initial flash of 9.1.1?