File transfers to freenas drop midway (10gbe)

[shaf]

Hi,

I'm running freenas under exsi 6.7 and this is for home use.

I've recently upgraded to 10gbe network using a mikrotik switch and Intel x520 da2 nics.

When I transfer from PC to Freenas I get speeds close to 500MB/s at the beginning but then this drops to 100MB/s and remains until completion.

I don't think it's a network issue as I transfer to another Windows VM and I get a constant 300-400MB/s.

I've done some digging and disabled autotune and removed all autotune parameters as suggested in some posts. But this has not helped.

Please can you let me know what other information I can provide and I will gladly do so.

Your help is appreciated.


Hardware:
Supermicro X10SrI
80GB Ram
IBM m1015
x520 da2
Intel xeon 8 Core processor

Freenas
48GB Ram
4x6TB WD Red (2 disk redundancy)
2x6TB WD Red (mirror)

 

[shaf]

ifconfig output on pfsense.

 

[Elliot Dierksen]

What are the IP addresses of all the hosts involved, and do you have different IP networks? If yes on the latter, is pfsense doing the routing between the networks? That can cause a serious problem. You best bet is it to use iperf/iperf2 to test network throughput without involving storage. That way you can validate the network performance without an extraneous factors. Also, what version of FreeNAS, and please tell us about the exact CPU model and how many cores are assigned to the FreeNAS VM.

 

[shaf]

Thank you for your reply.

CPU is: Xeon ES QEYX E5 2630LV3 CPU 8-core 1.8GHZ 20M LGA2011-3 and 4 cores are assigned to freenas.
Freenas version is: FreeNAS-11.2-RELEASE

ipaddress of freenas.
192.168.1.111

ipaddress of windows machine:
192.168.1.237

Pfsense is running but on it's own dedicated machine.

I'll do some iperf tests and report back.

 

[shaf]

iperf3 results.

 

[shaf]

iperf3 results seem quite low which is odd.

 

[Elliot Dierksen]

iperf3 results seem quite low which is odd.

You said you cleared all tunables and disabled autotune, right? I would suggest you re-enable autotune, reboot FreeNAS, and test again.

 

[shaf]

I have turned on autotune and rebooted.

 

[Elliot Dierksen]

How many 10G hosts do you have, and how many ports on each? What media are you using for the connection (TP, twinax/DAC cable, fiber)?

 

[shaf]

How many 10G hosts do you have, and how many ports on each?
I'm not sure what you mean.
I have only one esxi server and it has 2 ports intel adapter.
It's running 4 VMs one of which is freenas

What media are you using for the connection?
I'm using the following:
10Gtek 10Gb/s SFP+ DAC Cable 2m - 10GBASE-CU Passive Direct Attach Copper Twinax SFP+

 

[Elliot Dierksen]

Does FreeNAS have a NIC that is passed through directly from ESXi, or does it have a virtual NIC from ESXi? If it is the latter, what kind vnic is it (Flexible, VMXNET2, VMXNET3)?

 

[shaf]

It's passed through and its using vnic VMXNET3.

 

[shaf]

Really appreciate you help by the way.

I've done some iperf3 tests from.
Windows machine to Windows VM on esxi server.

Something seems wrong as I can transfer with high speed but iperf3 is showing lower speeds. Or am I reading this wrong?

I'm using iperf3 for windows.

 

[Elliot Dierksen]

Do you have vmware tools of some sort installed? I probably need to defer to others who are more knowledgeable on virtualizating FreeNAS. There are quite a few threads on this if you search the forums. Both my FreeNAS systems are bare metal.

 

[Elliot Dierksen]

Really appreciate you help by the way.

You are welcome.

Something seems wrong as I can transfer with high speed but iperf3 is showing lower speeds. Or am I reading this wrong?

That is odd because normally it is the other way around. iperf is testing with synthetic traffic so you don't have any dependencies on storage that could slow your results. It is only testing the network stack and underlying network. I am just guessing here, but it could be that since SMB/?CIFS transfer are not normally done with synch on, the writes could be getting buffered giving you higher speeds. That is just a guess, and not what I would normally expect to see.

You really ought to look at this and other forum posts on virtualization and FreeNAS. This one is long, but is kind of the grand daddy of all of them.
https://forums.freenas.org/index.ph...ide-to-not-completely-losing-your-data.12714/

 

[Mlovelace]

Don't use autotune, delete any of the autotune settings and reboot. Here are some of the tunables I've done on one of our system.

Code:

Variable                            Value            Type
cc_htcp_load                        YES              loader
net.inet.tcp.cc.algorithm           htcp             sysctl
net.inet.tcp.htcp.adaptive_backoff  1                sysctl
net.inet.tcp.cc.htcp.rtt_scaling    1                sysctl
net.inet.tcp.recvbuf_auto           1                sysctl
net.inet.tcp.recvbuf_inc            6291456          sysctl
net.inet.tcp.recvbuf_max            33554432         sysctl
net.inet.tcp.recvspace              6291456          sysctl
net.inet.tcp.sendbuf_auto           1                sysctl
net.inet.tcp.sendbuf_inc            6291456          sysctl
net.inet.tcp.sendbuf_max            33554432         sysctl
net.inet.tcp.sendspace              6291456          sysctl
kern.ipc.maxsocbuf                  33554432         sysctl

Typically if it's a sysctl your system doesn't require a reboot, if it's a loader the system will require a reboot. I've found the htcp congestion control to be better than the default newreno, which is the top loader cc_htcp_load. Start with these setting, though the window sizes might be bigger than you need. The value for the window sizes are in bytes (33554432 = 32MB) just fyi.

 

[shaf]

Tried deleting/editing the tuneables and now system has crashed.

I've rebooted but gui will not load. I've had to reset the configuration.

 

[shaf]

Adding this autotune parameter crashes my system:

net.inet.tcp.recvspace
or
net.inet.tcp.sendspace

 

[shaf]

With the above settings from Mlovelace I get the following results.

These show 1GBs speeds.

 

[shaf]

Without above settings I get these results.

Please note that I have the following three settings in autotune.
vmmemctl_load YES loader
vmxnet3_load YES loader
vmxnet_load YES loader

Some help appreciated please.