Another FreeNAS+NFS+Vmware Speed

[tahoo]

Hi FreeNAS forum

Another FreeNAS+NFS+ESXI+Datastore performance problem.

Thanks for a great forum.
(sorry for my english, hope that you understand)

First of all, I have read the many threads about NFS Performance with VMWare, and I know and get that it is all related to the SYNC writes requested by VMWARE's NFS.

I have the following hardware

FreeNas-9.10 hardware
- Supermicro X9SRL-F
- Intel Xeon E5-2603 V2 - Box
- 96 GB DDR3-1866 ECC REG
- 8 x HGST Ultrastar 7K6000 4TB HDD 7200rpm SAS 128MB cache 24x7 12 Gb/s
- Intel DC P3500 PCI-e SSD 400 GB
- 10 Gbit Intel X520-DA2 E10G42BTDA - PCI Express 2.0 x8
- IBM M1015 (Cross flashing to a LSI9211-8i in IT Mode P20.00.07.00)

VMware Server
- HP DL385p gen8
- Esxi 6.5
- 10 Gbit Intel X520-DA2 E10G42BTDA

10 Gbe Nic with direct cable from FreeNAS to 10 Gbe Vmware Nic.
(Works great - about 9400 mbit with iperf3 test)


All test with:
NIC:
MTU=9000 (FreeNAS + VMware)

ZFS:
Sync=standard
Compression=Off
Atime = Off


Test 1: - Performance test direct on the FreeNAS Server.
Raid Type: Mirror / Raid-10
HDD: 8 x HGST Ultrastar 7K6000 4TB HDD 7200rpm SAS 128MB
SLOG: N/A
CMD: dd if=/dev/zero of=temp.dat bs=4M count=50000
Write: 793 mb/s
CMD: dd if=temp.dat of=/dev/null bs=4M
Read : 790 mb/s

Test 2: - Performance test direct on the FreeNAS Server.
Raid Type: No raid only one SSD
HDD: 1 x Intel DC P3500 PCI-e SSD 400 GB (Speed 1000 mb/s write, 2200 Mb/S Read)
SLOG: N/A
CMD: dd if=/dev/zero of=temp.dat bs=4M count=50000
Write: 907 mb/s
CMD: dd if=temp.dat of=/dev/null bs=4M
Read : 1149 mb/s

Test 1 and Test 2 - It's the speed I expect. - What do you think ?


Test 3 - Performance test - FreeBSD Server on VMware ESXI - "NFS FreeNAS Datastore"
Raid Type: Mirror / Raid-10
HDD: 8 x HGST Ultrastar 7K6000 4TB HDD 7200rpm SAS 128MB
SLOG: 1 x Intel DC P3500 PCI-e SSD 400 GB (Speed 1000 mb/s write, 2200 Mb/S Read)
CMD: dd if=/dev/zero of=temp.dat bs=4M count=50000
Write: 365 mb/s
CMD: dd if=temp.dat of=/dev/null bs=4M
Read : 490 mb/s


Test 4 - Performance test - FreeBSD Server on VMware ESXI - "NFS FreeNAS Datastore"
Raid Type: No raid only one SSD
HDD: 1 x Intel DC P3500 PCI-e SSD 400 GB (Speed 1000 mb/s write, 2200 Mb/S Read)
SLOG: N/A
CMD: dd if=/dev/zero of=temp.dat bs=4M count=50000
Write: 490 mb/s
CMD: dd if=temp.dat of=/dev/null bs=4M
Read : 493 mb/s


Test 3+4
- Does the performance look right to you ?
- Is it as you would expect ?
- Ideas for performance improvement ?
- Do you think my CPU is the problem ?

If I disable ZFS Sync everyting is great - But that not an OPTION !!!

Thanks for your help.!!

/Simon

 

[mav@]

VMware requires every single NFS operation to be synchronously written on stable storage (SLOG). If there is no SLOG, ZFS writes it to main pool (that's why SSD-only pool is not much better).

I think it is all about SSD write/flush latency. Declared SSD write speeds unfortunately tell nothing about its flush latency. Desktop Intel 530 SSDs in my lab, for example, for this reason show absolutely horrible performance, while even Intel 520 are better.

Plus there is CPU overhead. What exactly version of FreeNAS are you using? FreeNAS 9.10.2 got some optimizations on the log latency/performance front.

 

[tahoo]

Hi Mav

Thanks for your reply.

I understand the synchronously written and why it is necessary to have a SLOG. I thought the Intel P3500 PCI-E was a very fast SSD - but maybe you are right about the "write/flush latency"

But what about my read performence, from 1149 mb/s to 493 mb/s on ESXI NFS.

Is that as you would expect ?

My FreeNAS version is 9.10.2-U2

Thanks

 

[mav@]

Sorry, I haven't noticed the read speed. The easiest start would be to look on CPU and disk usage: `top -SHIz` and `gstat -I 1s -p`. Also I would try to read the same file several times from NAS cache to see whether it is limited by disks at all, or just by link/protocol. Your CPU is probably lowest in its line.

 

[Ericloewe]

I understand the synchronously written and why it is necessary to have a SLOG. I thought the Intel P3500 PCI-E was a very fast SSD :) - but maybe you are right about the "write/flush latency"

It's reasonably fast. The P3700 is the one designed for this kind of application.

3D Xpoint shows promise for SLOG, assuming the marketing isn't pure crap.

 

[tahoo]

Thanks Mav

NP. I will test that

I will also try with a new CPU - Intel Xeon E5-1620 V2 - 3,7 Ghz
I can just borrow one

Again Thanks

 

[tahoo]

Hi Ericloewe

Great - So there must still be hope :)

Do you have a link to that report ? Thanks