Intel NAS performance toolkit - slow file read benchmark

[matram]

I am evaluating various storage solutions for an ESXi 5 host and was surprised by the slow read performance with FreeNAS in the Intel NAS benchmarks.

"FileCopyFromNAS"
FreeNAS-CIFS share: 26.3 MB/s
QNAP-iscsi: 42.4 MB/s
QNAP-NFS: 55.8 MB/s
VMDK: 137.5 MB/s

"FileCopyToNAS"
FreeNAS-CIFS share: 21.2 MB/s
QNAP-iscsi: 23.7 MB/s
QNAP-NFS: 23.7 MB/s
VMDK: 7.4 MB/s

The configurations are:
FreeNAS, running as a VM with 8 GB ram, ZFS filesystem, benchmark run on a CIFS share, VM running the tests or on the same host connected through a virtual vSwitch, physical disk volume used is the same as for VMDK
QNAP TS-659 PRO, 4 WD-RE4 2 TB disks in RAID5 configuration connected via 1 GB network and HP1910 switch, mounted as iSCSI or NFS datastore on the host
VMDK, 2 WD-RE4 2 TB disks in RAID 0 configuration, connected through a Dell PERC H200 controller locally on the host, as this is the underlying disk used by FreeNAS it represents what should be achievable.
The host is a Dell T710 with 32 GB of ram and dual four core Xeon processors (2.4 GHz). The systems was running no other significant load at the time of the tests.

The write performance is in line with the QNAP and much better than the direct connected disk (the H200 controller lacks a cache and disables the disk cache which makes for slow writes).

What surprises me is the read performance which only reaches 20% of the underlying disk performance (VMDK). The other benchmarks in the Intel suite points to a bottleneck in the read performance at about 30 MB/s for FreeNAS which I find surprising given the hardware it is running on.

I have searched the forum for suggestions and tried some of the aux settings for CIFS with almost no effect. Anyone have any suggestions?

 

[cyberjock]

Don't run FreeNAS as a virtual machine and expect stellar performance.

Have you tried doing a DD test for reads and writes? If you get horribly low numbers during the DD test then something is wrong.. either hardware or possibly related to the virtualization. FreeNAS really performs and is designed to have direct access to the hardware. Running in a virtual machine is somewhat contractory to the design of FreeNAS.

I'm a little confused on your post. Are you saying the VMDK is on a RAID0? If so it's a hardware RAID, which is not recommended for FreeNAS(but probably wouldn't be your performance problem).

 

[matram]

The T710 system has two hardware RAID 0 volumes. On one of these a VMware virtual disk is created, the VMDK. This volume is used by no other VM.

In the VMDK test this volume is mounted directly in the Windows test client as a D disk and benchmarks run on that disk. So the VMDK figure of 137.5 reflects the penalty of disk, network and test VM virtualization.

In the FreeNAS test the same virtual disk is mounted in FreeNAS a ZFS volume is created and a test directory shared out to the Windows test client using CIFS. Adding FreeNAS, ZFS and CIFS in between so to say drops the performance from 137.5 to 21.2 MB/s.

Please note that this is not VMWare workstation, this is the bare metal ESXi hypervisor. The pundits seem to think that presenting a disk a raw device to the VM (which you can do) provides no performance benefits, it is only recommended if you need to access low level features of your RAID controller. Monitoring host load during the benchmark also indicates lots of spare capacity on the host side.

There are "video playback tests" in the Intel suite which simulates longer reads, similar to a DD. The results are as follows:

VMDK (RAID0 volume directly mounted in test client)
x1 play: 130.9
x2 play: 168.6
x4 play: 144.9

FreeNAS (RAID0 volume shared out through FreeNAS using CIFS)
x1 play: 28.0
x2 play: 26.2
x4 play: 26.2

FreeNAS is currently used only to hold a replica of the primary NAS, the QNAP, so as long as FreeNAS is used only for that purpose the performance may be OK.

The picture of a bottleneck at 30 MB/s and the LACK OF the asymmetric behavior that i see in the other solutions (reads faster that writes and larger files give higher performance than smaller) makes me think there may be some configuration issue?

 

[cyberjock]

You still should do a DD from the CLI in FreeNAS. I would do a DD of 100GB or so inside FreeNAS to determine your performance. If it shows 28MB/sec then at least the problem isn't CIFS related. Also keep in mind that CIFS is single threaded, so more Ghz is better than more cores.

The main reason why FreeNAS strongly recommends against using RAID and virtualization is because FreeNAS uses it's own analysis to ascertain the server hardware status. If using a hardware RAID or virtualized disk it is not possible for the OS to determine if you have actually lost a parity/mirror disk or not since the hardware layer is hiding the true condition.

All that virtualization really does aside from allowing you to multi-purpose the hardware is complicate the configuration and maintenance of FreeNAS. The forum has alot of posts of people that ran ESXi for months, then they realized that FreeNAS doesn't quite operate as expected in a virtualized environment. In the last 2 weeks or so someone had a post because getting away from a virtual machine that is already configured with users and whatnot was not easy for him.

 

[matram]

DD on the 1.2 GB video file used in the "x1 play test" from the FreeNAS command line (login via SSH) gives a 33.3 MB/s transfer rate.

Attached is a picture of the disk performance from the vSphere server during DD (right peek, see also max values) again READ seems to peek at about 30 MB/s (write has a higher peek) while the same file on the same disk can be read at 130 MB/s under Windows XP.



Any basic FreeBSD IO configuration I should be aware of?

 

[matram]

Some more benchmark data 8.3beta1 and Nexenta comparison

Curios about 8.3 beta1 I re-run the benchmark on 8.3 and also on Nexenta with the same basic setup. The data is here F82 = FreeNAS 8.2, F83 = FreeNAS 8.3beta1, NEX=Nexenta store, VMDK=performance with RAID0 volume directly mounted on XP test client.

Intel tests (MB/s)..: (F82 | F83 | NEX | VMDK)
FileCopyToNAS.....: 21.2 | 04.0 | 15.1 | 07.4
FileCopyFromNAS.: 26.3 | 24.9 | 39.9 | 137.5
DirCopyToNAS.....: 04.4 | 03.7 | 12.1 | 07.1
DirCopyFromNAS.: 12.0 | 10.0 | 26.2 | 49.7

Any ideas?

 

[cyberjock]

I'm not to familiar with Nexenta so I'll ignore those results for now.

The fact that your DD is only 33.3MB/sec tells me something is wrong outside of FreeNAS. 2 drives in a RAID0 should be able to hit a conservatively low 100MB/sec. The picture is downsized to the point where I can't read it at all, but I don't think it would have provided much info. There's something going on either with the drives themselves, some other hardware issue, or the virtualization.

I've never used ESXi 5 but perhaps it is limiting performance to prevent other virtualized machines from being choked by the high disk reads/writes and throttles it accordingly?

You should focus on getting the DD test up MUCH higher. Obviously if the DD test can't give you good speeds there's no way you'll ever get higher speeds across the LAN in FTP, CIFS, etc. I'd first try running tests on the hard disk benchmarks on the disks indiviually. All of your write speeds are just horrible and all of your read speeds are bad except the file copy from NAS on VMDK.

This could be yet another example of why virtualization is bad. You'll have to experiment to figure out what is actually going on. If you don't have anything else on those 2 drives you could pull them out and put them in another machine with a copy of FreeNAS installed on a USB key and try to setup a RAID0 on the drives using ZFS to see what you get for a DD speed. Anyway, at this point its a game of ruling out various causes until you find the bottleneck.

This is the game of cat and mouse. Since you are using virtualization you have an extra layer of potential problems to go through. :(

Good luck! Be sure to post back what the problem is if you find it in case someone else has the same problem you do.

 

[matram]

More input

Sorry I made a mistake in the previous "dd" tests. I copied the test file to a new file on the same disk which was the reason for the low speed.

Rerunning (dd if=tesfile of=/dev/null) I get the following raw speeds from the virtual machine accessing the virtual disk.

FreeNAS: 112 MB/s (ZFS + FreeBSD)
Nexenta: 138 MB/s (ZFS + OpenSolaris)

Testing also other VM (Debian and Windows) basically confirms that read speed form a virtual machine is around 110 MB/s.

Performance when the test is run on a Windows client against a CIFS share is

FreeNAS: 28 MB/s (25% of raw performance)
Nexenta: 50 MB/s (36%)

It seems the issue is with the CIFS sharing? Although there is a bit of difference between the FreeNAS and Nexenta both show good performance on the raw zfs / os level but have a big drop when sharing out via CIFS.

Anyone have any ideas?

 

[paleoN]

It seems the issue is with the CIFS sharing? Although there is a bit of difference between the FreeNAS and Nexenta both show good performance on the raw zfs / os level but have a big drop when sharing out via CIFS.

Anyone have any ideas?

Have you tried using NFS? I don't even know why you were comparing QNAP NFS to FN8 CIFs given that FN8 has NFS.

A number of people are less than pleased with their Samba on FreeNAS performance. It works well for some people though. You can try some CIFs tuning and see if that helps.

 

[matram]

Update ...

I don't even know why you were comparing QNAP NFS to FN8 CIFs given that FN8 has NFS.

The QNAP is used as a datastore in VMware vSphere. A datastore has to be local disk, NFS or iSCSI. The intent of the benchmark on the QNAP was to decide between NFS and iSCSI. The decision was NFS based on the benchmark results.

Secondly I was concerned about the performance of the local disk array on the Dell T710 server, a vSphere datastore which is used by FreeNAS (only).

Disappointed with the benchmark results I have now upgraded the local array to:
- Dell PERC H700 high performance RAID controller with 1 GB NV RAM write back cache
- Created a RAID5 array with three WD RE4 2 TB enterprise SATA drives

Performance as measured by HD Tune Pro in a Windows VM is now
average read: 209 MB/s,
access time: 7 ms access time,
random access: 83 IOPS
I am looking at putting a Intel 520 SSD as a second level cache in the array using DELL's CacheCade firmware.

From a benchmarking perspective I am already having problems getting reasonable results due to ZFS caching and the write back cache in the RAID controller.The caches are simply becoming so big that a benchmark may fit completely in cache and you get very high (unrealistic) performance. A real life example would be my build server which after a reboot builds in about 110 seconds (previously 160 seconds). Subsequent builds now take 15 seconds. :)

FreeNAS is used for three purposes.
- Several VM rsync in critical folders and databases nightly and FreeNAS then rsync them to offsite storage (rsync.net)
- Some public ftp folders on the QNAP are rsynced to FreeeNAS that acts as a backup (internal and) ftp server.
- Backup of VM's are stored on FreeNAS (original VM's on QNAP and VM replicas for critical systems on local server storage)
This all came out of a failure mode analysis and wanting to have no single point of failure in the whole system. Functionally this all works great and it is not really performance critical.

The reason for sharing out the FreeNAS (public ftp) folders with SAMBA/CIFS was simply that all internal clients could mount those folders without additional configuration or software in the same way as they are mounted from QNAP.

I understand your comment that I am comparing "apples and oranges", you are right in this.

I will rerun benchmarks in a more "symmetric" way. Keep you posted ...

 

[matram]

Test results

OK, tests re-run. I tried to attach a PDF with results but that was not allowed, it seems 40k is too much?

In summary. Raw performance of the local datastore is around 180 MB/s for both read and write (using Intel NAS performance toolkit).

If the local datastore is shared using FreeNAS and NFS performance drops to, read: 45, write: 6

If FreeNas uses CIFS instead performance is read: 27, write: 16.

Nexenta perfromance is about 30% better (OpenSolaris effect I guess as both are using ZFS).

QNAP performance is in line with Nexenta.