FreeNAS in VMware Performance (NFS) Benchmark

[dmnt3d]

I recently built a FreeNAS system with the following configuration:

CPU: i3 2120
Mem : 16GB
HBA: supermicro sas2lp-h8ir
4 x SAS Disk 450GB 15K RPM (RAIDZ)
1 x 120GB (configured as ZIL)
4 x Intel NIC (2 x LAG, 1 x mgmt)

I set it up as NFS target for my ESXi machine running 5.5.

My previous setup was a Synology DS412+ (3 x 2TB WD RED, ext4).

I've run benchmark to compare both and i'm suprised the Synology (with EXT4) is performing better in 100%Read/100%Random tests.

Here's a quick rundown of the result (i use io_analyzer; with the secondary disk in the NFS DS only)
IOPS TEST
OLD:
297.63 4k_50%Read_100%Random
127.94 4k_0read_100rand
1631.95 4k_100read_100rand

NEW (FreeNAS)
2346.06 4k_50%Read_100%Random
531.58 4k_0read_100rand
774.4 4k_100read_100rand

Anyone have tips in improving the result?

I'll primarily use the FreeNAS as a Shared Storage for my ESXi home lab.

Also - anyone know a good benchmarking for FreeNAS/Disk Subsystem. I'm from Virtualization world so i only know IOMETER/ IO-Analyzer.

Thank a lot.

 

[zambanini]

read the noob guide, the manual and the sticky posts. you are doing it wrong. short answer, because you did not understand raidz, nfs and vsphere synch writes.

 

[eraser]

Welcome to the forums!

If you are running FreeNAS 9.3, first step is to check for and install any available updates. Many things have been tweaked/improved since the release.

=-=-=

From your test results, your FreeNAS server completely destroys your Synology in two out of three tests. Cool!

Why is your Synology faster in the 100% read/100% random test? My guess is that you are seeing the difference between RAIDZ and whatever disk layout you used in your Synology as well as the difference between ZFS and EXT4.

I wouldn't worry about it. When is a virtualized environment ever going to be 100% read?

=-=-=

Since this is your home lab, I am going to guess you have one ESXi host. In this case, setting up a LAG won't make any difference because only a single TCP connection is used for NFS traffic between your ESXi host and FreeNAS server.

If you want a learning project you could get rid of the LAG and move from NFS to multipath iSCSI (MPIO). But with only four disks this won't make a performance difference and it will greatly complicate your setup. It also requires multiple NICs in your ESXi server.

=-=-=

IOPS is what people usually care about with VMware.

You will get increased IOPS if you switch from one 4-disk RAIDZ vdev to two MIRROR vdevs (at the cost of another disk's worth of storage). Probabaly not worth deleting/recreating your volumes but something to consider in the future.

=-=-=

In the grand scheme of things though, once your FreeNAS server has been up for a few hours/days then its huge 16 GB ARC will be warm and you will enjoy robust performance. I sometimes like to open up a terminal and run the following command to watch the ARC work while I reboot my VMs:

arcstat.py -f time,read,hits,miss,hit%,arcsz,c 5

 

[dmnt3d]

@eraser,
Thanks for the warm welcome.

You are correct on the disk layout that is causing the performance difference. I did some reading and it looks like for ZFS, the READ IOPS is always equal to a single DISK per VDEVS, which in this case, is a single SAS 15K RPM.

I'll recreate the VDEV from RAIDZ to RAIDZ2 and perform the test again. Will report back the result here.

I'm currently using: 3 x ESXi hosts and one FreeNAS server. I just reinstalled everything so it's fine to recreate the NFS.

QUESTION:
would partitioning my 120GB SSD and using it for both ZIL and L2ARC make a performance difference? Specifically for VMware purpose where 100% of the time it's doing random workloads.

 

[mav@]

I'll recreate the VDEV from RAIDZ to RAIDZ2 and perform the test again.

It is pointless to switch from RAIDZ to RAIDZ2 for performance -- it will be only lower. If you need random read performance -- you need stripe of mirrors (aka RAID10).

 

[jgreco]

I'll recreate the VDEV from RAIDZ to RAIDZ2 and perform the test again. Will report back the result here.

RAIDZ2 will be a bit slower. The mirror vdevs previously suggested will be rather faster.

would partitioning my 120GB SSD and using it for both ZIL and L2ARC make a performance difference? Specifically for VMware purpose where 100% of the time it's doing random workloads.

You always have a ZIL. I think you mean "SLOG".

At 16GB, you have insufficient RAM to make good use of L2ARC. We don't suggest L2ARC until 64GB, because it crushes the ARC, which has to provide the index into the L2ARC.

There seem to be a lot of people coming in lately with 16GB systems trying to use L2ARC. Where'd you get this idea from?

 

[Ericloewe]

There seem to be a lot of people coming in lately with 16GB systems trying to use L2ARC. Where'd you get this idea from?

Must be some random blogger spreading crap, like the German Technical Monkey website that led to countless Atom J1900 builds in the German section of the forum.

 

[jgreco]

The Atom J1900 looks like it'd be a fine platform for a pfSense NAT gateway...

 

[cyberjock]

@eraser,
QUESTION:
would partitioning my 120GB SSD and using it for both ZIL and L2ARC make a performance difference? Specifically for VMware purpose where 100% of the time it's doing random workloads.

It is a fail to even consider putting the slog and L2ARC on the same device. They should not ever be put on the same device as they will compete for I/O and both will suck... horribly.

 

[jgreco]

It is a fail to even consider putting the slog and L2ARC on the same device. They should not ever be put on the same device as they will compete for I/O and both will suck... horribly.

Yes, but it's even more of a fail to consider doing it on a 16GB system. One fail at a time. ;-)

 

[cyberjock]

Yes, but it's even more of a fail to consider doing it on a 16GB system. One fail at a time. ;-)

But if I do 4 fails, is that like not failing at all?

I run FreeNAS on a 2GB USB drive, on a system with 4GB of RAM, with a hardware RAID, on bad non-ECC RAM. This should be an amazing experience!

 

[norbs]

Yes, but it's even more of a fail to consider doing it on a 16GB system. One fail at a time. ;-)

How sure are you about that? I added an L2ARC on an 16GB system that hosts iSCSI for VMWare and I've noticed a pretty significant response in the VM's since I made the change. Is there something I'm not understanding or simply over looking? It takes some time to build up the cache but it seemed to have helped. I was also considering adding SLOG SSD since I have one more channel left on the LSI card and an extra SSD, seems like a bad idea after reading this. If anything I was hoping that the L2ARC would at least reduce hits to the hard drives since it's 6 drives working vs 1 ssd.

Config:
Six 4TB drives in a RAID10 (~11TB Pool) with 1TB iSCSI file extent on top of that. I added a 120GB ssd for the L2ARC.

Keep in mind these are parts I already have laying around and I am not buying stuff, so it's not like I'm investing money into it. I also can't go over 16GB on the particular system.

 

[norbs]

Just wanted to share a few benchmarks.

I ran HD Tune in a VM that and ran it repeatedly. Now although the overall transfer speeds did not rise in any significant way you can clearly see a significant improvement in the average access times (which is important to me in VMWare). The only thing that actually dropped was the burst rate. But look how nicely the access time (yellow dots) plot after adding an L2ARC. Either way, what I missing here?

L2ARC removed (same results after multiple runs):

L2ARC Enabled (after 4 benchmark runs):

L2ARC enabled (after 7 benchmark runs):

 

[HoneyBadger]

I was also considering adding SLOG SSD since I have one more channel left on the LSI card and an extra SSD, seems like a bad idea after reading this.

SLOG doesn't lean on RAM, so it's worthwhile; however it should be noted that by default the VMware iSCSI initiator writes all data async. That's why iSCSI seems so much faster than NFS - it doesn't give you write assurance by default. Enable sync=always on the dataset and it will come to a screeching halt.

What's the extra SSD? Not all of them are good SLOG devices, in fact most of them aren't.

Re: the whole "16GB + L2ARC = Fail" it's because you'll generally see better options by increasing the RAM before adding L2ARC. In addition, having an L2ARC consumes RAM, so it's robbing you of primary cache. In your case you can't go beyond 16GB though.

 

[norbs]

The drive I was going considering for SLOG is a 120GB Intel 530 which I under-provisioned in the firmware for 8GB. Pretty sure it's an MLC drive and I put an in-line sata power capacitor (don't know how well it actually works but it's there and the whole system is on a UPS).

Is the size of the L2ARC in direct proportion to how much memory is uses? If so, any idea roughly what that ratio would be?

BTW, I appreciate you taking the time to explain all that.

 

[HoneyBadger]

Underprovisioned to 8GB it should definitely be a lot better performer than full-sized. I'm assuming you did this via a secure erase and setting the Host Protected Area on the drive per Intel's instructions which will let it use the whole drive for wear-leveling.

Re: the L2ARC stuff, I'm going off memory at this point, it's late, and I work in Solaris as well so there might be errors below. Someone feel free to correct me as needed. All of the below assumes no tuning of ZFS parameters since that often does more harm than good. ;)

The rule of thumb for L2ARC is around a 1:4 or 1:5 ratio of RAM to L2ARC.

Each entry in the L2ARC index uses roughly 180 bytes, and iSCSI works on 4KB recordsize. So a 120GB L2ARC, completely filled with 4KB, would consume:

120GB = 125829120 KB, divide by 4 KB = 31457280 records, times 180 bytes = 5662310400 bytes or about 5400MB of RAM.

Assuming your max ARC is the default of 7/8ths your RAM, you've got 14336MB available. That should work, right?

Well, not quite. The L2ARC index is considered metadata, which by default is capped to 1/4 your total ARC size, or 3584MB. And we're trying to stuff 5400MB in there. Whoops.

So let's assume the 3584MB limit. Reversing the math above, you get a target L2ARC size of about 79.5GB, but realistically less because there's other metadata that will want to live in RAM as well.

Hey, wait a second.

16GB (of RAM) * 5 = 80GB (of L2ARC)

Well, would you look at that. ;)

Now, if you look at this in the case of purely performance, the question is which is better:

1) Have 14GB of really fast reads, and then hit the spinning rust immediately
2) Have 10.5GB of really fast reads, 75GB or so of "fast" data - once it's warmed up, though - and then hit the spinning rust

And that's something only you can answer, since you know your workload better.

The usual question though isn't "To L2ARC or not to L2ARC?" it's usually "Do I buy more RAM or an SSD?" and in that case the better performing solution is almost always "add more RAM" since ZFS eats it like popcorn.

Think I've rambled for long enough.

 

[norbs]

"Under-provisioned to 8GB it should definitely be a lot better performer than full-sized. I'm assuming you did this via a secure erase and setting the Host Protected Area on the drive per Intel's instructions which will let it use the whole drive for wear-leveling."

I may have done the under provisioning incorrectly...

I'll have to redo it, I used seatools for dos to under-provision. I'll have to reprovision back to 120gb, secure erase and then under-provision back to 8gb.

Wow that is pretty significant memory usage for the L2ARC... I still like the idea of less hits to the physical hard drives. Now to simply pick an L2ARC size which i have to carefully think about. I did have a 480GB ssd that I was considering to sacrifice for the L2ARC but I'm most likely just going to go with another 120GB Intel 530.

Thanks again.

 

[cyberjock]

Now, if you look at this in the case of purely performance, the question is which is better:

1) Have 14GB of really fast reads, and then hit the spinning rust immediately
2) Have 10.5GB of really fast reads, 75GB or so of "fast" data - once it's warmed up, though - and then hit the spinning rust

Actually, that assessment is much too simplistic. You have to account for the fact that you'll also have higher turnover of the ARC, which creates all sorts of new problems that makes the L2ARC cache total crap useless data, so now you've made things worse than before.

No, this is stupid, We've said again and again to have lots of RAM *before* you add an L2ARC. The L2ARC is supposed to supplement the ARC. It is not intended to, it is not designed to, and it will not replace, actual ARC. Anyone that wants to think or argue otherwise is smoking something really good and is selfish for not sharing.

It doesn't matter how much you want the L2ARC to help. It doesn't matter that you are too poor to afford more RAM. It doesn't matter that you can't put more RAM on your board. It is what it is. Either accept it, don't use ZFS, or be stupid. Take your pick.

 

[norbs]

Weird, why would I see such a difference in the responsiveness of my VM's with an L2ARC (even without a SLOG)?

I chose to do the following stupid setup for now:
16GB RAM
8GB SLOG (120gb under-provisioned)
60GB L2ARC (120GB under-provisioned)

I'll continue to "be stupid" for a while and report back with what I see with this setup in case anyone is actually interested and maybe just scared to say they are.

 

[HoneyBadger]

I'll have to redo it, I used seatools for dos to under-provision. I'll have to reprovision back to 120gb, secure erase and then under-provision back to 8gb.

There's a white paper from Intel that details this. Not sure if the commands are supported on the 530 though. Which brings up another point, don't use a 530. ;) It's lacking the power-failure write protection of a supercapacitor or battery. Look at one of the Intel DC series SSDs if you can. Even the cheapest one (S3500 or S3610 I think) should be sufficient.

But again, remember that by default you aren't using an SLOG at all right now which means performance is very fast but unsafe. Even with a fast SLOG device, performance will take a big hit when you turn on sync.