FreeNAS in VMware Performance (NFS) Benchmark

[jgreco]

Weird, why would I see such a difference in the responsiveness of my VM's with an L2ARC (even without a SLOG)?

The SLOG has nothing to do with it.

Since you're maxxed out on RAM, you can see some benefit from L2ARC, but the problem is that it will tend to be somewhat unpredictable and variable. iSCSI is the worst sort of workload. You can read up on some of that at this link here, which will help explain some of why iSCSI typically requires more resources for what seems like a simple task.

As @HoneyBadger noted, there's a rule of thumb that suggests about 1:4 or 1:5 ARC:L2ARC, and we often trivialize that by saying it's RAM:L2ARC, but that fails at small quantities of RAM. On a warmed-up busy 16GB system here, for example, the ARC size is only 8GB. The FreeNAS middleware and other resource requirements can have a tendency to push down the ARC size a bit more than might be expected (I was expecting to find it around 10GB!) and as noted, ARC meta is limited to 1/4th that. So doing @HoneyBadger 's math again, we come up with something more like 40GB L2ARC. This might vary for non-iSCSI uses.

And that'll be very likely to work and take some load off the pool. However, the rate at which data is being evicted from ARC is going to be relatively high, and this will cause it to be difficult for ZFS to identify what would be good candidates for L2ARC. The window into your dataset is relatively tiny, and that means ZFS won't be making the best selections for eviction to L2ARC. It is sending to L2ARC the blocks that have the most activity that it can no longer keep in ARC, but, if you have a 30GB file that you're sequentially reading totally hundreds of times a day, all the blocks appear to have approximately equal levels of activity. It won't easily identify that the whole thing should be sent to L2ARC...

But overall, L2ARC helps bigger busier systems a lot more.

 

[dmnt3d]

Hi All,
OP here and just got back from re-configuring the drives based on the recommendation to use RAID10.
So far - i'm happy with the performance and will be using this config.

Here are some benchmark I did:

NEW: ZFS RAIDZ with SLOG
2346.06 4k_50%Read_100%Random
531.58 4k_0read_100rand
774.4 4k_100read_100rand
3384.37 4k_70%Read_100%Random

NEW: RAID10 no SLOG:
1692.28 4k_50%Read_100%Random
643.05 4k_0%Read_100%Random
2518.75 4k_100%Read_100%Random
1068.68 4k_70%Read_100%Random

NEW RAID10 with SLOG
4102.94 4k_50%Read_100%Random
1223.79 4k_0%Read_100%Random
1916.41 4k_100%Read_100%Random
1450.82 4k_70%Read_100%Random

Consistent 1k+ IOPS!! wee

Thank you all.

Last question: Anyone can point me on how to underprovision SSD's? I'm concerned on the wear in it as i'm using a consumer grade MLC (Crucial M4 64GB).
My plan is to underprovision it then split it up into two partition.
64GB -> Underprovision to 30 GB,
Partition into 15GB's
1 x 15GB parition as SLOG for the RAID10
1x 15GB partitionas SLOG for another vDEV i'll be adding soon.

A quick guide on how to do this will be appreciated.

Thanks again.

 

[HoneyBadger]

Glad to see the performance boost there!

The Crucial M4 you're using is probably not a good SLOG choice, as it has poor reliability overall and is very small in size. Despite most people needing only about 10GB of SLOG space, the larger drives still offer better performance due to increased parallelism from multiple NAND chips.

Regarding underprovisioning it - I don't believe the M4 supports the HPA command set for drive-level underprovisioning, so you will have to resort to a BIOS-level size limitation if possible, or manually creating a smaller partition from the command line. Whichever one you do, you should perform a secure erase first using the Crucial SSD toolbox.

SLOG is assigned on a per-pool basis, not per-vdev, so you will only not need to attach a second one for any additional vdevs you add. If you instead meant that you'll share this SLOG between pools, my advice is "don't" because then you're in a situation where both of the different workloads can impact one another.

Command-line underprovisioning commands follow. Make sure to do this from a completely clean, secure-erased SSD, and substitute your SSD's device identifier if "ada0" isn't accurate. This creates an 8GB SLOG which is more than large enough for folks who aren't on 10GbE.

Code:

gpart create -s GPT ada0
gpart add -t freebsd-zfs -a 4k -s 8G ada0
zpool add YOURPOOLNAME log ada0p1

 

[Ericloewe]

Glad to see the performance boost there!

The Crucial M4 you're using is probably not a good SLOG choice, as it has poor reliability overall and is very small in size. Despite most people needing only about 10GB of SLOG space, the larger drives still offer better performance due to increased parallelism from multiple NAND chips.

Regarding underprovisioning it - I don't believe the M4 supports the HPA command set for drive-level underprovisioning, so you will have to resort to a BIOS-level size limitation if possible, or manually creating a smaller partition from the command line. Whichever one you do, you should perform a secure erase first using the Crucial SSD toolbox.

SLOG is assigned on a per-pool basis, not per-vdev, so you will only not need to attach a second one for any additional vdevs you add. If you instead meant that you'll share this SLOG between pools, my advice is "don't" because then you're in a situation where both of the different workloads can impact one another.

Command-line underprovisioning commands follow. Make sure to do this from a completely clean, secure-erased SSD, and substitute your SSD's device identifier if "ada0" isn't accurate. This creates an 8GB SLOG which is more than large enough for folks who aren't on 10GbE.

Code:

gpart create -s GPT ada0
gpart add -t freebsd-zfs -a 4k -s 8G ada0
zpool add YOURPOOLNAME log ada0p1

And it has no power failure protection, which means it really can't do its job as a ZIL device in most cases where the ZIL is actually read.

 

[jgreco]

Agree that it is a poor SLOG device - no power loss protection. However, ignoring that, if the OP is sufficiently familiar with his traffic demands to understand that sharing of the device wouldn't adversely impact performance, having two partitions for SLOG for two different pools might be acceptable.

It is certainly a hell of a lot better than trying to make it both SLOG and L2ARC.