FreeNAS/ESXi 10GbE slow SINGLE transfers

[adx442]

Hi there,

I've recently put in a cheap 10GbE backbone in my SMB server room, consisting of a Netgear ProSAFE Plus XS708E, connected to two ESXi servers with 10GbE Intel NICs built in (SuperMicro boards), and my FreeNAS 9.3 box has an Intel X540T1 card added to it. The switch reports a full duplex 10GbE link between all devices. It should be quite the infrastructure upgrade for $1200.

My FreeNAS server has 16GB of ECC RAM, and 10 Western Digital SE (server grade SATA) 3GB drives with a 256GB SSD SLOG and 256GB SSD ZIL device. The hard drives are configured as mirror VDEVs, leaving a little under 10TB of storage for the array.

My "problem" isn't huge. I definitely see the 10GbE benefit on multiple streams of data going to and from the FreeNAS box, but a single file transfer never goes much past 1GbE speeds (a little higher, but only like 133MB/s). However, if I send multiple files simultaneously from multiple VMs, the aggregated speed seems to be more around 300-400MB/s ... about what I would expect from my RAID10. All VMs are using the VMX3 network card driver, and the VMs state they have a 10GbE connection. The processors have miles of idle time on them while doing the transfers, so it's not a CPU bound issue.

Most of my shares from FreeNAS are CIFS/SMB3, and the VMs involved are mostly Server 2012R2, so it should be using SMB3 which should obviate some of the inherent SMB slowness. I did try an NFS mount with about the same speed results, though.

Is there anything I can check on or modify to increase my transfer rate from my ESXi boxes to FreeNAS (and vice-versa)?

Thanks for any pointers you can provide. I searched some other threads, but didn't find anything quite like this.


EDIT: Oh my, my array seems mighty slow internally. I just ran a dd if=/dev/zero of=testfile bs=1024 count=100000 and got a result of 103MB/s (a poor result from a single drive, much less all 5 VDEVs working in concert). The read wasn't much better. I repeated with a 10GB file and it was consistently bad.

 

[zambanini]

you have slog and zol on a seperate drive? ,, )

if you mean l2arc...you need much more ram.
with your dd test, of parameter was on the zpool of your disks?

 

[jgreco]

SLOG and ZIL are the same thing. If you meant to say L2ARC, your L2ARC is crushing the **$*@&@ out of your system. You need probably 64GB of RAM before you can think about L2ARC. This will ruin performance.

 

[jgreco]

you have slog and zol on a seperate drive? ,, )

if you mean l2arc...you need much more ram.

Damn, beat me.

And I should have said "In this context, SLOG and ZIL are the same thing."

https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

 

[adx442]

Okay, I must have totally misunderstood L2ARC when I set up the system. Is it possible to disable/remove it after the fact without having to restore my data from backups? Thanks for your help!

 

[jgreco]

Indeed, it is entirely safe to remove the L2ARC.

The process is a little non-obvious. Log into the GUI. Right hand column, Storage, Volumes, View Volumes.

Click on your pool. At the bottom, four buttons appear. Click "Volume Status."

Click on the L2ARC device (should say "cache" I believe). Click "Remove."

Should do it.

 

[adx442]

Indeed, it is entirely safe to remove the L2ARC.

The process is a little non-obvious. Log into the GUI. Right hand column, Storage, Volumes, View Volumes.

Click on your pool. At the bottom, four buttons appear. Click "Volume Status."

Click on the L2ARC device (should say "cache" I believe). Click "Remove."

Should do it.

Thank you very much, I did that, rebooted, and all is intact. Write speeds immediately went to 225MB/s and Read went to 666MB/s. Definitely a huge improvement. My CIFS transfers remain the same speed, though, so I've still got that to figure out.

Wow, L2ARC is not to be toyed with. I noticed that my system had consumed all of its memory, but I just thought that was precaching as Linux/FreeBSD like to do. Now it's got 8GB free after boot.

 

[jgreco]

The 8GB will plummet as the ARC fills, that's actually a good thing. The problem with L2ARC is that it requires space in the normal ARC to act as an index, so it is robbing you of the normal functionality of the ARC. This could be impacting how the system is caching metadata that tracks free blocks (my offhand guess), meaning lots more I/O.

We usually don't suggest L2ARC until at least 64GB. Further, the ratio of L2ARC-to-RAM is best off around 5:1, at least until you validate that your workload can safely handle a better ratio, which probably still isn't more than 10:1.

 

[Elliot Dierksen]

I can add one thing that is based strictly on my own observations. Vcenter must have some sort of throttling on how much bandwidth it will use for a single operation. I didn't get very good utilization on my system when moving a single VM. It will only move two at a time, but I get periods of time where I can sustain 75% utilization of a 10G NIC port while two operations are going. I rarely see more than 10-20% with a single operation in progress.

 

[adx442]

I had another, completely non-FreeNAS related issue that was also impacting this. There were two subnets involved, and some types of traffic were being routed rather than switched, and crossing a gigabit port in doing so. Getting some 300MB/sec copy operations now, which is much better than it was before.

 

[jgreco]

I can add one thing that is based strictly on my own observations. Vcenter must have some sort of throttling on how much bandwidth it will use for a single operation. I didn't get very good utilization on my system when moving a single VM. It will only move two at a time, but I get periods of time where I can sustain 75% utilization of a 10G NIC port while two operations are going. I rarely see more than 10-20% with a single operation in progress.

I don't think there's throttling, but, the number of operations is tunable. One of the more readable explanations: http://frankdenneman.nl/2013/01/14/limiting-the-number-of-concurrent-vmotions/