Slow NFS Reads, fast NFS writes, fast iSCSI read/write

[ser_rhaegar]

On my main system, I have a FreeNAS VM, running with 12GB of RAM, pass through M1015, 10Gb vmxnet3 and 1Gb e1000 vNICs. This has 8 drives connected, 6 2TB WD Red/Green drives and 2 480GB Samsung SSDs. The drives are in two pools, a RAIDZ2 with the spinners and a mirror with the SSDs. My intention was to use the SSD pool for VMs via NFS shares and leave my RAIDZ2 alone.

The 10Gb vNIC on FreeNAS is set to mtu 9000 and is on a vlan with only the ESXi host (mtu 9000), hosting it. I am able to ping with a 9000byte packet across, good replies. I set up an NFS share with the SSD mirror and connected it to my ESXi host. Moving a VM over was quick. However when reading from the share, it was almost non-existent in speed (i.e. <50Mb, couldn't see on the reports tab graph as the writes were 3Gb making the read line invisible).

So I setup a few tests. I did a dd in and out of both pools, raw on the FreeNAS VM via SSH. I also setup an NFS share on each pool with sync=disabled. Then I also setup an iSCSI share on each pool. For the NFS/iSCSI tests, I moved a FreeBSD VM between each share and added an 80GB disk for testing. I wrote 30GB for each test, then read the 30GB back and wrote it again. The test was using /dev/zero, compression off on the NFS datasets.

The results... RAW/iSCSI were great. 150MB/s+ (I'm just looking for 100+ as this is a home system). NFS writes were good but reads... I cancelled after 20min of no network activity on the report tab, no CPU usage either (other tests pegged my CPU which was possibly the limiting factor on some of them).

What might cause NFS reads to be slower than a turtle? I'm talking so slow they don't register on the network graphs.

Tests results:

Code:

alpha - 6 2TB drives, RAIDZ2
epsilon - 2 480GB SSDs, mirror

RAW, pool alpha, no compression
freenas# dd if=/dev/zero of=/mnt/alpha/demo3/tmp.bin bs=2048k count=15k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 127.782688 secs (252086220 bytes/sec)
freenas# dd if=/mnt/alpha/demo3/tmp.bin of=/mnt/alpha/demo3/tmp2.bin bs=2048k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 235.061685 secs (137037453 bytes/sec)


RAW, pool epsilon, no compression
freenas# dd if=/dev/zero of=/mnt/epsilon/bench/tmp.bin bs=2048k count=5k
5120+0 records in
5120+0 records out
10737418240 bytes transferred in 32.638861 secs (328976500 bytes/sec)
freenas# dd if=/dev/zero of=/mnt/epsilon/bench/tmp.bin bs=2048k count=10k
10240+0 records in
10240+0 records out
21474836480 bytes transferred in 59.492930 secs (360964512 bytes/sec)



iSCSI, pool alpha/demo2
root@testfreebsdvm:/bench # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 284.915308 secs (113059052 bytes/sec)
root@testfreebsdvm:/bench # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 226.766500 secs (142050324 bytes/sec)


NFS, pool alpha/demo3, no compression, sync=disabled
root@testfreebsdvm:~ # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 113.788534 secs (283088757 bytes/sec)
root@testfreebsdvm:~ # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k
Too slow, waited 20min, network/cpu on freenas were near 0% in usage


iSCSI, pool epsilon/demo
root@testfreebsdvm:~ # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 172.761371 secs (186455193 bytes/sec)
root@testfreebsdvm:~ # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 210.780767 secs (152823501 bytes/sec)


NFS, pool epsilon/performance6, no compression, sync=disabled
root@testfreebsdvm:/build # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k
15360+0 records in
15360+0 records out
32212254720 bytes transferred in 103.755059 secs (310464425 bytes/sec)
root@testfreebsdvm:/build # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k
Too slow, waited 20min, network/cpu on freenas were near 0% in usage

 

[cyberjock]

You will find that nobody here will help troubleshoot issues that even look like they might be related to FreeNAS being run in a VM. We don't support or condone the activity. And most of us find that trying to help people with a problem in a VM is more often than not taken as our endorsement.

 

[ser_rhaegar]

You will find that nobody here will help troubleshoot issues that even look like they might be related to FreeNAS being run in a VM. We don't support or condone the activity. And most of us find that trying to help people with a problem in a VM is more often than not taken as our endorsement.

Even if you can't help with it as is, I appreciate the reply CJ.

When I get some time today, I'll backup my config, load up FreeNAS on a USB drive, boot on the bare metal, restore the config and retest using 1GbE, my other ESXi host and 64GB writes (host has 32GB of RAM). I'll report the results later today.

 

[cyberjock]

That's probably a good test.

Just to clarify, it's not a personal thing.The problem is that because of the extra virualization layer gives the propensity for normally working things to be broken, and then we spend huge amounts of time trying to troubleshoot problems that only exist for your machine.

So the thought process is:

1. It's not worth our time for a one-off problem.
2. It's not worth the risk of giving the impression we endorse virtualization.(and this happens every damn time.. someone later will make the accusation)
3. If you are virtualizing FreeNAS, then you *should* be capable of solving your own problems anyway because you should be an ESXi expert.
4. We just don't like people with 2 word in their name(ok.. I'm just kidding about this one).

There was talk of a section of the forums specifically for people that virtualize. But, the big red stop sign was put on that very rapidly and while the section exists it's not available to the public(it's empty.. so you aren't missing out on anything). We didn't want to give the impression that virtualizing FreeNAS is remotely smart.

One thing.. dd tests must be at least 2x your system RAM for the results to mean anything. Otherwise it's all cached and you get a false high or false low value. Also compression must be disabled(and the default is enabled for 9.2.1+).

 

[pbucher]

Even if you can't help with it as is, I appreciate the reply CJ.

When I get some time today, I'll backup my config, load up FreeNAS on a USB drive, boot on the bare metal, restore the config and retest using 1GbE, my other ESXi host and 64GB writes (host has 32GB of RAM). I'll report the results later today.

Save your self some time and just debug your virtualized setup. While it will help you eliminate some possible issues, you are probably better off chasing down the cause of your issue. CJ needs to get over it, virtualized FreeNAS just simply rocks when done right and it looks like your doing a pretty good job of it from your post.

I'd suggest making you've got jumbo frames enabled across everything, you need to enabled it on the vSwitch, and the vmware kernel, along with any physical switches that might be in the mix and off course on the FreeNAS interface. I'd suggest installing the vmxnet3 driver and switching over to that. It really sounds like you have something in the stack that doesn't have jumbo frames enabled probably the vmware kernel possibly. Since it can write that data without problem but can't read it(so it's probably writing regular sized frames and they are passing through everything ok and then FreeNAS is trying to push out jumbo frames and it's not getting through something.

For what it's worth I manage 7 VMed installs of FreeNAS with over 100 spinners between them all and haven't had any issues due to it being virtualized, it's a road less traveled so sometimes you need to work a little harder and it requires better understanding of the architecture of everything but it's worth it if you have the knowledge & time to do it. I recently had some hardware compatibility issues with ESXi 5.5 and I dropped one of my big SANs down to the bare metal and moved all the VMs off to another server that is connected to this SAN via a 10GBe network and I definitely lost performance vs having the VMs on the same box as the SAN, the latency of going over the physical network was without question noticeable(while not end of the world unlivable - I was just surprised that by moving FreeNAS to the bare metal that I didn't pick up any performance).

Feel free to PM me if you need some help, though my time is pretty limited at the moment I'll do what I can.

 

[ser_rhaegar]

Thanks for the reply pbucher, really appreciate it. I've already dropped it to baremetal though, just haven't had time to run the tests yet today.

I'd suggest making you've got jumbo frames enabled across everything, you need to enabled it on the vSwitch, and the vmware kernel, along with any physical switches that might be in the mix and off course on the FreeNAS interface. I'd suggest installing the vmxnet3 driver and switching over to that. It really sounds like you have something in the stack that doesn't have jumbo frames enabled probably the vmware kernel possibly. Since it can write that data without problem but can't read it(so it's probably writing regular sized frames and they are passing through everything ok and then FreeNAS is trying to push out jumbo frames and it's not getting through something.

The vSwitch and the vmkernel NIC both have 9000 MTU set, along with the 10Gb NIC in FreeNAS. I'll give it a shot with default MTU settings when I return to virtualizing FreeNAS as another test. Though only NFS has any issues with reads, iSCSI performed well in both directions.

I'll post back with my test results later in the day. Thanks again for the reply.

 

[ser_rhaegar]

Haven't forgotten this, still running tests when I have time. Rerunning the original ones as well with 1GbE across a physical switch to another ESXi host to bring it closer to apples to apples. Also letting the NFS reads finish.

 

[HoneyBadger]

CJ needs to get over it, virtualized FreeNAS just simply rocks when done right and it looks like your doing a pretty good job of it from your post.

Right, but when done wrong, which is easy to do, it can result in a torched pool. jgreco has a thread for "if you absolutely MUST virtualize" that should be required reading.

 

[cyberjock]

And I could have sworn we banned you for your piss poor behavior before pbutcher... but whatever.

 

[pbucher]

Right, but when done wrong, which is easy to do, it can result in a torched pool. jgreco has a thread for "if you absolutely MUST virtualize" that should be required reading.

Hence the point is to do it right......and we all know very well that it's really easy to do it wrong without any virtualization being involved. Yes virtualization makes it easier to do it wrong and too many folks try use to use virtualization to cut some corners, but I think it's time to move beyond this blank condemnation of virtualizaiton. Look you scared the original poster into thinking it was better to run FreeBSD then FreeNAS because you can't virtualize FreeNAS, now that wasn't productive. You guys gotta learn to tell the difference between folks who can learn and build a working system vs folks who are going to toast their pools with or without virtualization, I'm not saying I can either I'm just saying a little lighter handed approach on some folks might be a nice change.

 

[ser_rhaegar]

Mods please lock this thread. Not looking cause arguments about virtualizing or not. Was just wondering if something stood out with NFS writes. I'll resolve this on my own when I have time.

Thanks for the input guys.

 

[toadman]

I too had run into this issue. After some testing (read a file using "dd") I have narrowed my issue down to a specific combination... ESXi (5.1) reading from an NFS share served over an LACP lagg (2 nics) on freenas (9.2.1.3). This combination reads at <40 mpbs on 1G hardware. And there is no issue at all with writes, this is only a read problem.

Specific config:
esxi 5.1, distributed switch w/2 uplinks (Intel nics), vmkernel teaming w/route by physical nic load, vlan defined for nfstraffic
switch: netgear GS108T, vlan for nfstraffic
freenas: 9.2.1.3, lagg configured with 2 nics (Intel) and LACP, vlan & interface defined for nfstraffic w/vlan physical interface being the lagg

Note, what does work as expected:
(1) a linux VM on the esxi box reading from the same NFS share. This reads at >900 mpbs.
(2) the VM reading from a CIFS share over that LACP lagg. Also >900 mbps.
(3) ESXi reading from the NFS share with the vlan physical interface moved to a separate single nic (realtek spare nic). >500 mpbs (yea, realtek nics suck)
(4) ESXi reading from an mpio ISCSI share with it's 2 vlans using the LACP lagg for their physical interface. >600 mpbs. (This was a testing config - not the normal setup for iscsi.)

(3) was a surprise! So the LACP itself is working (tested with other clients over NFS and CIFS in addition to the above, and with ISCSI). The ESXi box itself can read the NFS share at expected speed as shown by case 3. But obviously something with ESXi and the Freenas LACP config is throttling NFS reads. Any ideas?