ESXI iSCSI Data Store

[CookiesLikeWhoa]

Hey all!

First things first, here's the current system:
Case - Mercury S8, CaseLabs
PSU - EVGA SuperNova G850
Motherboard - Supermicro X10SRL-F
CPU - Xeon E5-1650 V3
Memory - 128GB (4X32) Samsung ECC R-DIMM
Boot Device - 2X Crucial 16GB USB Drives
Hard Drives - 12 HGST Deskstar NAS 4TB Mirrored VDevs
SSD's - 4 Samsung 850 EVO 500GB Mirrored VDevs
NIC - X520-DA2 10Gb, Intel
FreeNAS-9.10-STABLE-201606270534 (dd17351)

I'm planning on making my own ESXI system and am looking to use my Freenas as an iSCSI target for data store(s) for it. The ESXI system will have around 15 VM's going, 1 VM will be for Windows running Blue Iris which will likely be the busiest. I would like to set sync=always for these targets so...

I've read in a couple of places that a good SLOG device and a large L2ARC are helpful for running iSCSI data stores for ESXI. The first question I have is, should I invest in an L2ARC right now or invest in maxing out my ram (another 128GBs) then going for the L2ARC? The second question is, would a SLOG device help performance if I plan to set it to sync=always?

For L2ARC devices I was leaning towards an Intel 750 400GB SSD. Should provide a decent amount of L2ARC at good speeds unless there is something else I should look into.

For a SLOG device I was looking at a 200 GB Intel S3700. The question I have is, would I be loosing performance with a SLOG device like that? Is it possible to stripe a device like that or should I just invest in a single Intel 750?

Many thanks ahead of time!

 

[Spearfoot]

Hey all!

First things first, here's the current system:
Case - Mercury S8, CaseLabs
PSU - EVGA SuperNova G850
Motherboard - Supermicro X10SRL-F
CPU - Xeon E5-1650 V3
Memory - 128GB (4X32) Samsung ECC R-DIMM
Boot Device - 2X Crucial 16GB USB Drives
Hard Drives - 12 HGST Deskstar NAS 4TB Mirrored VDevs
SSD's - 4 Samsung 850 EVO 500GB Mirrored VDevs
NIC - X520-DA2 10Gb, Intel
FreeNAS-9.10-STABLE-201606270534 (dd17351)

I'm planning on making my own ESXI system and am looking to use my Freenas as an iSCSI target for data store(s) for it. The ESXI system will have around 15 VM's going, 1 VM will be for Windows running Blue Iris which will likely be the busiest. I would like to set sync=always for these targets so...

I've read in a couple of places that a good SLOG device and a large L2ARC are helpful for running iSCSI data stores for ESXI. The first question I have is, should I invest in an L2ARC right now or invest in maxing out my ram (another 128GBs) then going for the L2ARC? The second question is, would a SLOG device help performance if I plan to set it to sync=always?

For L2ARC devices I was leaning towards an Intel 750 400GB SSD. Should provide a decent amount of L2ARC at good speeds unless there is something else I should look into.

For a SLOG device I was looking at a 200 GB Intel S3700. The question I have is, would I be loosing performance with a SLOG device like that? Is it possible to stripe a device like that or should I just invest in a single Intel 750?

Many thanks ahead of time!

You may not need an L2ARC at all, and there are numerous discussions about this subject on the forum.

However, a SLOG device is a necessity for an ESXi datastore with synchronous writes enabled, and the faster the better. You can stripe two SSDs, as you mentioned, and the Intel S3700 is a good entry-level choice. For better performance, choose the 750.

 

[diehard]

SLOG will essentially be a requirement for forcing sync.

Striping will not help performance, and would probably just hinder it. Get the 750 unless you are going to be doing a lot of writes.. then get a DC P3700.

 

[Spearfoot]

SLOG will essentially be a requirement for forcing sync

Agreed!

Striping will not help performance, and would probably just hinder it.

iXsystems disagrees: "You can even use multiple SLOG devices, which OpenZFS will stripe across for improved performance."

Get the 750 unless you are going to be doing a lot of writes.. then get a DC P3700.

Virtualization by its very nature involves a lot of writes... :)

The Intel DC P3700 is an excellent choice for a SLOG device, albeit more expensive than a DC S3700/S3710 (entry-level) or 750 (mid-level).

This forum thread has an excellent discussion on the subject: "Some insights into SLOG/ZIL with ZFS on FreeNAS"

 

[diehard]

iXsystems disagrees: "You can even use multiple SLOG devices, which OpenZFS will stripe across for improved performance."

I'm about 95% sure that in practice, this is wrong.

Virtualization by its very nature involves a lot of writes... :)

VM's can still vary greatly in their amount of writes committed. The amount of VM's and usage scenario can help determine the amount of endurance of an SLOG that will be required.

 

[CookiesLikeWhoa]

Thanks for the information!

Looks like I'll need to find out if I can stripe two devices for the SLOG.

As far as the L2ARC goes, I'm personally leaning towards maxing the ram out then doing the L2ARC, which should be fun.

 

[diehard]

send jgreco a PM about the striping of the SLOG if you want a 100% sure answer and not a 95% one.

 

[CookiesLikeWhoa]

Haven't seen jgreco around in a while. Worst comes to worst I'll shoot him a PM.

 

[Mirfster]

I've been using an Intel DC S3500 (160 GB) for SLOG and been pretty satisfied. While I do have S3710s (200GB) I have not compared them (yet). Of course this is all in an experimental AiO ESXi/FreeNAS test box.

I suppose that stripping SLOGs would provide better performance, but not too sure if it is truly worth the investment. Not to mention it would take an extra port/slot.

Paging @jgreco and @cyberjock for their input. ;)

 

[cyberjock]

Sorry, I don't comment on systems with FreeNAS virtualized. As our stickied threads say, if you need assistance, you shouldn't be virtualizing.

/exits the thread.

 

[CookiesLikeWhoa]

Eventually I want to get around to an aio esxi box but I need to learn a lot more before that.

I'm not to worried about ports. Right now I have everything running off an expander for the 9207 so I still have 8 Sata ports on the mobo and another 12 via the expander.

Sorry, I don't comment on systems with FreeNAS virtualized. As our stickied threads say, if you need assistance, you shouldn't be virtualizing.

/exits the thread.

Freenas isn't virtualized. It's just going to be a iscsi target for an esxi host. Just looking for advice on how to ensure it runs well.


Sent from my iPhone using Tapatalk

 

[Mirfster]

Sorry, I don't comment on systems with FreeNAS virtualized. As our stickied threads say, if you need assistance, you shouldn't be virtualizing.

/exits the thread.

OP isn't running a Virtual FreeNAS (that is me, but I am not asking for assistance). I was just commenting on my setup. ;)

 

[HoneyBadger]

Striping SLOGs does indeed increase performance; however, the nature of an SLOG being "stable storage" means that the more appropriate setup (and default in FreeNAS, I believe) for two SLOG devices is a mirror.

This has no positive effect on performance, but further reduces the risk window for incoming data, since you now need to lose two devices and have a power loss at the same time.

 

[CookiesLikeWhoa]

Striping SLOGs does indeed increase performance; however, the nature of an SLOG being "stable storage" means that the more appropriate setup (and default in FreeNAS, I believe) for two SLOG devices is a mirror.

This has no positive effect on performance, but further reduces the risk window for incoming data, since you now need to lose two devices and have a power loss at the same time.

I take it you could stripe mirrors of slog devices if needed? Seems a bit extreme and at that point would be cheaper/easier to just get an Intel 750.


Sent from my iPhone using Tapatalk

 

[Spearfoot]

I take it you could stripe mirrors of slog devices if needed? Seems a bit extreme and at that point would be cheaper/easier to just get an Intel 750.

Yes, you could. But, to be honest, I've never seen or heard of anyone doing that. Most users configure a single device, or a pair of mirrored devices. In your particular case, you would probably be best served using the 750 (good) or an Intel DC P3700 (better); mirrored or not, as your budget and HDD port constraints allow.

 

[maglin]

Stripped mirrors on SLOG would be very expensive to replace when they wear out.


Sent from my iPhone using Tapatalk

 

[HoneyBadger]

I take it you could stripe mirrors of slog devices if needed? Seems a bit extreme and at that point would be cheaper/easier to just get an Intel 750.


Sent from my iPhone using Tapatalk

Well, cheap NVMe devices didn't always exist. At this point, a single Intel NVMe device is faster than a 10Gbps network link, so a mirror is fine. But once faster speeds become commonplace, or someone bonds/MPIO's multiple 10Gbps links, you may end up needing striped mirrors of NVMe if throughput can't keep up on the SLOG side.

 

[HoneyBadger]

Stripped mirrors on SLOG would be very expensive to replace when they wear out.

"Speed costs money. How fast do you want to go?"

 

[CookiesLikeWhoa]

Thanks for the information guys! I think it would be the safest route to just go with a intel 750 for the L2Arc and SLOG for now.

 

[Nick2253]

Thanks for the information guys! I think it would be the safest route to just go with a intel 750 for the L2Arc and SLOG for now.

Just to clarify, we're talking about two separate devices, right? You shouldn't use the same device for both functions.