ser_rhaegar
Patron
- Joined
- Feb 2, 2014
- Messages
- 358
On my main system, I have a FreeNAS VM, running with 12GB of RAM, pass through M1015, 10Gb vmxnet3 and 1Gb e1000 vNICs. This has 8 drives connected, 6 2TB WD Red/Green drives and 2 480GB Samsung SSDs. The drives are in two pools, a RAIDZ2 with the spinners and a mirror with the SSDs. My intention was to use the SSD pool for VMs via NFS shares and leave my RAIDZ2 alone.
The 10Gb vNIC on FreeNAS is set to mtu 9000 and is on a vlan with only the ESXi host (mtu 9000), hosting it. I am able to ping with a 9000byte packet across, good replies. I set up an NFS share with the SSD mirror and connected it to my ESXi host. Moving a VM over was quick. However when reading from the share, it was almost non-existent in speed (i.e. <50Mb, couldn't see on the reports tab graph as the writes were 3Gb making the read line invisible).
So I setup a few tests. I did a dd in and out of both pools, raw on the FreeNAS VM via SSH. I also setup an NFS share on each pool with sync=disabled. Then I also setup an iSCSI share on each pool. For the NFS/iSCSI tests, I moved a FreeBSD VM between each share and added an 80GB disk for testing. I wrote 30GB for each test, then read the 30GB back and wrote it again. The test was using /dev/zero, compression off on the NFS datasets.
The results... RAW/iSCSI were great. 150MB/s+ (I'm just looking for 100+ as this is a home system). NFS writes were good but reads... I cancelled after 20min of no network activity on the report tab, no CPU usage either (other tests pegged my CPU which was possibly the limiting factor on some of them).
What might cause NFS reads to be slower than a turtle? I'm talking so slow they don't register on the network graphs.
Tests results:
The 10Gb vNIC on FreeNAS is set to mtu 9000 and is on a vlan with only the ESXi host (mtu 9000), hosting it. I am able to ping with a 9000byte packet across, good replies. I set up an NFS share with the SSD mirror and connected it to my ESXi host. Moving a VM over was quick. However when reading from the share, it was almost non-existent in speed (i.e. <50Mb, couldn't see on the reports tab graph as the writes were 3Gb making the read line invisible).
So I setup a few tests. I did a dd in and out of both pools, raw on the FreeNAS VM via SSH. I also setup an NFS share on each pool with sync=disabled. Then I also setup an iSCSI share on each pool. For the NFS/iSCSI tests, I moved a FreeBSD VM between each share and added an 80GB disk for testing. I wrote 30GB for each test, then read the 30GB back and wrote it again. The test was using /dev/zero, compression off on the NFS datasets.
The results... RAW/iSCSI were great. 150MB/s+ (I'm just looking for 100+ as this is a home system). NFS writes were good but reads... I cancelled after 20min of no network activity on the report tab, no CPU usage either (other tests pegged my CPU which was possibly the limiting factor on some of them).
What might cause NFS reads to be slower than a turtle? I'm talking so slow they don't register on the network graphs.
Tests results:
Code:
alpha - 6 2TB drives, RAIDZ2 epsilon - 2 480GB SSDs, mirror RAW, pool alpha, no compression freenas# dd if=/dev/zero of=/mnt/alpha/demo3/tmp.bin bs=2048k count=15k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 127.782688 secs (252086220 bytes/sec) freenas# dd if=/mnt/alpha/demo3/tmp.bin of=/mnt/alpha/demo3/tmp2.bin bs=2048k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 235.061685 secs (137037453 bytes/sec) RAW, pool epsilon, no compression freenas# dd if=/dev/zero of=/mnt/epsilon/bench/tmp.bin bs=2048k count=5k 5120+0 records in 5120+0 records out 10737418240 bytes transferred in 32.638861 secs (328976500 bytes/sec) freenas# dd if=/dev/zero of=/mnt/epsilon/bench/tmp.bin bs=2048k count=10k 10240+0 records in 10240+0 records out 21474836480 bytes transferred in 59.492930 secs (360964512 bytes/sec) iSCSI, pool alpha/demo2 root@testfreebsdvm:/bench # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 284.915308 secs (113059052 bytes/sec) root@testfreebsdvm:/bench # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 226.766500 secs (142050324 bytes/sec) NFS, pool alpha/demo3, no compression, sync=disabled root@testfreebsdvm:~ # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 113.788534 secs (283088757 bytes/sec) root@testfreebsdvm:~ # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k Too slow, waited 20min, network/cpu on freenas were near 0% in usage iSCSI, pool epsilon/demo root@testfreebsdvm:~ # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 172.761371 secs (186455193 bytes/sec) root@testfreebsdvm:~ # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 210.780767 secs (152823501 bytes/sec) NFS, pool epsilon/performance6, no compression, sync=disabled root@testfreebsdvm:/build # dd if=/dev/zero of=/bench/tmp.bin bs=2048k count=15k 15360+0 records in 15360+0 records out 32212254720 bytes transferred in 103.755059 secs (310464425 bytes/sec) root@testfreebsdvm:/build # dd if=/bench/tmp.bin of=/bench/tmp2.bin bs=2048k Too slow, waited 20min, network/cpu on freenas were near 0% in usage