All SSD vdev OR SSD L2ARC & Cache for shared ESXi SAN

[HalfJawElite]

Hello Everyone,


First off I want to say that I’ve long browsed the online forums for help and ideas on setting up a FreeNAS system and figured it’s now time for me to join in on the discussions and gain some knowledge. I’m fairly new to FreeNAS, but look forward to learning what its full potential can be for my home lab from the experts here online.

And without further adieu, the million dollar question:

1. All SSD drive vdev’s with maxed out RAM? (no L2ARC or Cache)

OR

2. WD RE4 vdev with SSD ZIL and L2ARC and maxed out RAM?

I’m looking to build this as a shared SAN for my VMware ESXi cluster, which I hope to be adding more hosts to in the future, along with replacing aging ones. Currently I’m running two VMware 5.5 u3b hosts with local SSD storage on them for my VM’s, and an NFS “backup” volume on a QNAP NAS for copying the VM’s every time I take snapshots of them. I’m looking to add a fast SAN which I can provision and store all of the VM’s on without the need for local storage on the hosts. This is to give me better flexibility when I need to spin up VM’s or migrate them between hosts for maintenance. I currently run some VM’s for study purposes, but also have a few running for some friends as well. I’ll be running either 40 Gb InfiniBand or 10 Gb Ethernet on the networking side, connected directly to each ESXi host, until the time comes to add a switch.

I’ve done some digging through the forums and the FreeNAS primer document and found some useful information, but nothing concrete that can help assess which option is better suited for my scenario.

Links to similar forum posts:

ESXi NFS storage SSD RAIDZ3

I'm building a high performance ZFS SAN for education

An All SSD Rig and FreeNAS

Successful SAN build for ESXi

High Performance All SSD Array?

I’m looking to get some more details on which FreeNAS drive setup would provide the fastest speed and higher IOPS. This is considering that the number of VM’s running on between all the hosts at any time can be between 6-7 minimum, with potentially more running as needed by my other users. One of these VM’s will also be running a small email server, with another two running a few websites. Any suggestions to hardware changes for the SAN would be welcome as well, since the system has not been purchased yet.


The bare essentials for the SAN configuration will be as follows:

Motherboard: MBD-X11SSL-F-O

CPU: Xeon E3-1220 v5 OR i3-6100

RAM: 64 GB DDR4 ECC UDIMMs (4 x 16 GB sticks)

PSU: SeaSonic 400 watt 2U PSU

Case: Norco RPC-250

Networking: Dual port Mellanox Connectx-2 InfiniBand OR Dual port Intel 10 Gb copper PCI-e card


The two ESXi hosts are currently configured with the following setup:

Motherboard: Tyan S5512GM4NR

CPU: Xeon E3-1230 v2

RAM: 32 GB DDR3 ECC UDIMMs (4 x 8 GB sticks)

PSU: SeaSonic 400 watt 2U PSU

Case: Norco RPC-250

Networking: Dual port Intel ET gigabit PCI-e card for vMotion network & single port Mellanox Connectx-2 InfiniBand OR single port Intel 10 Gb copper PCI-e card

Storage: 1 x Curcial M500 960 GB SSD

 

[Dice]

Hello,
If your budget allows for SSDs with enough space, that's the given choice. The vdev configuration will be mirrors.
However, you'll still want a SLOG device if running sync=always (SSD, preferrably NVMe, Power loss protected, with high write durability).

 

[HalfJawElite]

Hello,
If your budget allows for SSDs with enough space, that's the given choice. The vdev configuration will be mirrors.
However, you'll still want a SLOG device if running sync=always (SSD, preferrably NVMe, Power loss protected, with high write durability).

Firstly, the SSD's I'd be looking at would be something close to enterprise grade (sadly my Crucial M500's are no longer in circulation). Suggestions on specific drives are welcome.

As for the sync=always part, I'm guessing having the SSD for SLOG will depend on if I'm using it enabled with NFS or iSCSI correct? I'm still fairly new to how ZFS and FreeNAS itself work with storing data and how the file system works as a whole (though the primer doc is quite explanatory).

On a side note: what size is recommended for the SLOG on an NVMe PCI-e SSD? Given I'll be using this to run VM's off of for the hosts, is there a general rule-of-thumb that dictates the size based on the ZFS array?

 

[Dice]

As for the sync=always part, I'm guessing having the SSD for SLOG will depend on if I'm using it enabled with NFS or iSCSI correct?

The sync flag refers to NFS, correct.
A SLOG is advisable in the iSCSI scenario too.

what size is recommended for the SLOG on an NVMe PCI-e SSD? Given I'll be using this to run VM's off of for the hosts, is there a general rule-of-thumb that dictates the size based on the ZFS array?

Typically quite small. It is not dictated by pool size. It is rather influenced by amounts of "incoming data" that needs to be committed to the pool. This is due to the intervals in which ZFS flushes transactions groups onto the pool. So the amount of data that end up in transaction groups between each flush is the required size of the SLOG. So a 40GBit would need a 40x larger SLOG device, theoretically.

Some other things come to mind in your scenario. I'll skate out on thin ice and suggest something along these lines (which would need to be verified or corrected):
In your case with a pool consisting of more or less unlimited IOPS (multiple mirrored vdevs of SSDs). In articles you'll read about performance percs of adding a SLOG. Typically this refers to rotating rust, not SSDs (HDDs becomes penalized harder as they are hit twice, compared to SSDs). But - since ZFS uses transactions groups to cache data before committing data to the pool even 'random writes' is treated sequentially. From this follows, the benefit of insane IOPS is not ...as clearly cut as one would think. What I'm getting at is that I'm not too sure the SLOG device would carry as much benefit to an all SSD pool, as it would to rotating rust due to the already low latency and low seek times. Put differently - a SLOG device capable of handling 40Gbit without tanking performance ...is not cheap. Disregarding performance - the data loss protection aspect of adding a SLOG is probably viable. At some point I think the ZIL on pool would be quick enough to write out data. That tiny gap of time is what you are taking precautions to secure.

There are a boatload of intricacies that dictate the performance of ZFS.
Sizing RAM, L2ARC, enough vdevs and ...extremely important - maintaining a lot of free space is PIVOTAL to performance.
That is sort of the beauty of ZFS - cheap hardware can output a lot more value than anticipated. A boatload of cheap HDDs, eons of free space and well selected SLOG and L2ARC has the real potential to shit on the performance of enterprise SSDs. Notably when lacking RAM, free space and poorly selecting a SLOG.

I've made an effort for you to dig up some good reads for you to better wrap your head around SLOGs in relation to ZFS.
This link covers some broad basics:
https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/
http://www.freenas.org/blog/zfs-zil-and-slog-demystified/

These are particularly good reads (unfortunately deeply hidden in the forum)
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/
https://forums.freenas.org/index.ph...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

cheers

 

[HalfJawElite]

The sync flag refers to NFS, correct.
A SLOG is advisable in the iSCSI scenario too.


Typically quite small. It is not dictated by pool size. It is rather influenced by amounts of "incoming data" that needs to be committed to the pool. This is due to the intervals in which ZFS flushes transactions groups onto the pool. So the amount of data that end up in transaction groups between each flush is the required size of the SLOG. So a 40GBit would need a 40x larger SLOG device, theoretically.

Some other things come to mind in your scenario. I'll skate out on thin ice and suggest something along these lines (which would need to be verified or corrected):
In your case with a pool consisting of more or less unlimited IOPS (multiple mirrored vdevs of SSDs). In articles you'll read about performance percs of adding a SLOG. Typically this refers to rotating rust, not SSDs (HDDs becomes penalized harder as they are hit twice, compared to SSDs). But - since ZFS uses transactions groups to cache data before committing data to the pool even 'random writes' is treated sequentially. From this follows, the benefit of insane IOPS is not ...as clearly cut as one would think. What I'm getting at is that I'm not too sure the SLOG device would carry as much benefit to an all SSD pool, as it would to rotating rust due to the already low latency and low seek times. Put differently - a SLOG device capable of handling 40Gbit without tanking performance ...is not cheap. Disregarding performance - the data loss protection aspect of adding a SLOG is probably viable. At some point I think the ZIL on pool would be quick enough to write out data. That tiny gap of time is what you are taking precautions to secure.

There are a boatload of intricacies that dictate the performance of ZFS.
Sizing RAM, L2ARC, enough vdevs and ...extremely important - maintaining a lot of free space is PIVOTAL to performance.
That is sort of the beauty of ZFS - cheap hardware can output a lot more value than anticipated. A boatload of cheap HDDs, eons of free space and well selected SLOG and L2ARC has the real potential to crap on the performance of enterprise SSDs. Notably when lacking RAM, free space and poorly selecting a SLOG.

I've made an effort for you to dig up some good reads for you to better wrap your head around SLOGs in relation to ZFS.
This link covers some broad basics:
https://www.ixsystems.com/blog/o-slog-not-slog-best-configure-zfs-intent-log/
http://www.freenas.org/blog/zfs-zil-and-slog-demystified/

These are particularly good reads (unfortunately deeply hidden in the forum)
https://forums.freenas.org/index.php?threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/
https://forums.freenas.org/index.ph...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

cheers

Many thanks for the detailed information. I'll dig through it and let you know what I end up doing based on answers I've got now.