SOLVED Strange, low iPerf speeds

[mxve]

Hello Everyone.

I have a problem regarding low transfer speed but it doesn’t look like it’s something that have already been posted. And since it’s my first FreeNAS install, I have found about it, of course, only after testing transfer speeds using SMB share. But when I dug deeper more peculiar things have appeared:

(MBP - MacBook Pro I used for testing with Thunderbolt to 1GbE adapter (I don’t know what chip is this but it’s supposed to be great; besides I don’t think it matters)
SV - server I am using as a FreeNAS platform with 2x Intel 82574L GbE NICs)

• initial test (MBP->SV via SMB) show slower than possible transfer speeds: ~60MB/s. I thought it was odd since I have created stripe volume on SSD directly plugged to motherboard.
• then I confirmed this behavior by testing AFP share with the same connection between MPB and SV
• about that time I started to check everything I could. I used iPerf 3 to check cables, switch, router and my other computers. I was constantly getting ~940Mb/s (~110MB/s) between my other machines in both ways
• I returned to testing connection between MBP and SV by using installed on FreeNAS iPerf 2. My results were strange, at least for me:

(for connection I used UTP 5e twisted pair cable directly plugged to MBP using manually configured second NIC on SV (later I also checked the other one). I chose 128KB as a window size in iPerf because it gave me most stable and best results in other tests)

MBP: iperf -s
SV: iperf -c 192.168.1.3 -i 1 -w 128KB
Speed: ~940Mb/s

SV: iperf -s
MBP: iperf -c 192.168.1.1 -i 1 -w 128KB
Speed: ~600Mb/s

That was on FreeNAS 11.1 U5. On FreeNAS 11.2 Beta2 I was getting same results. In both occasions FreeNAS was installed on SSD. I have also installed FreeBSD 11.2 and results were almost the same as in FreeNAS: SV(-c)->MBP(-s): ~640Mb/s, SV(-s)<-MBP(-c): ~940Mb/s.

Because I have already checked the cable in other tests and confirmed it is ok and capable of saturating 1GbE both ways I didn’t know where to turn to… so I turned to CentOS. Between MBP and CentOS 7.3 on SV I was getting constant ~940MB/s for SV as a client and ~840MB/s when SV was acting as iPerf 2.0.12 server. Still not getting 1GbE but much better than while using FreeNAS/FreeBSD. So I went and confirmed this by transferring file using SCP and results were the same: ~110MB/s down and ~100MB/s up.

So I fired up FreeNAS 11.1 U5 again and tested file transfer with SCP. ~30MB/s down, ~20MB/s up.

Also, cyberjock wrote in one of his posts that he uses/used same NICs and they should work OOTB…

I am stumped now. What’s going on? Those are the same machines, disks, NICs, cable, etc. Why there is such big discrepancy?

Maybe someone has seen something like that and would like to shed some light on what might be the culprit here. Possibly it might not be a problem in respect only to FreeNAS but it is more apparent there.

I am a noob when it comes to networking so any help would be appreciated.

SV:
Supermicro X8DTE-F
2x Intel E5620
48GB ECC Registered
2x Intel 82574L GbE onboard
2x Samsung 830 SSD plugged directly into motherboard (Intel ICH10R SATA 3.0Gbps Controller)
I have also Adaptec 8505 RAID card with 6 disks attached to it but let’s say I won’t be using it for now… I don’t want to make PROs angry when I need them the most ;)

Final note: SV is headless and I’m using IPMIView from Supermicro as a not very comfortable KVM and I don’t think it’s possible to copy an output, but using SSH I can provide any type of results that might help.

FInal final note: I am trying to set up FreeNAS as a main NAS at home and I did some tests regarding different volumes’ types but no matter what type I chose I couldn’t get past that ~60MB/s (stripe, mirror, RAIDZ1; SSD, HDD) so I think resolving problem described above is paramount.

 

[RickH]

Have you tried both of the on-board NIC's on your server motherboard? Is there any difference between them?

The fact that CentOS wasn't able to saturate the connection leads me to believe that there may be an issue with the on-board NIC's, have you tried updating the motherboard BIOS?

Check the MTU settings - and disable Jumbo-Frames if they're enabled (set all MTU settings to 1500)

Is FreeNAS correctly detecting the NIC's? Run the following and post the output:

Code:

lspci | grep Ethernet

The 82574l silicon is support by the EM network driver - are the interfaces showing correctly as em0 and em1?

Gigabit NIC"s are pretty cheap, have you tried an add-in card to see if that improves things?

 

[mxve]

Thank you for your reply RickH.

Have you tried both of the on-board NIC's on your server motherboard? Is there any difference between them?

Yes and no.

Actually the problem was indeed with onboard NICs or more precisely with inability of Intel 85574L chip to work with Active-State Power Management (ASPM) or at least to work properly. This looks like rather popular problem with 85574Ls and I found info on that on different forums (CentOS, RedHat, VMWare) and even it's in Supermicro's FAQ.

On CentOS there is a file (/sys/module/pcie_aspm/parameters/policy) where one can switch ASPM profile but it haven't had any impact in my case. Fortunately, disabling previously enabled ASPM switch in BIOS solved the case... and increased server's power consumption by additional 40W.

And just for fun: I did try using a different NIC. It was based on, hated on this forum, RTL8168E. And it did manage to give me stable 941mbps... both ways...