FreeNAS for storing Proxmox VMs - help with SLOG / L2ARC device selection?

[victorhooi]

Hi,

We're putting together a FreeNAS server to store VMs for ProxMox.

Our motherboard is a SuperMicro X11SSH-F - this does have M.2 PCIE x2 slot, as well as PCIE slots. We'll be loading it with 32 GB of RAM (max for motherboard is 64GB).

Storage - we'll be using 4 x Crucial M500 512GB SSDs.

Networking will be provided with Intel X520-DA2 cards, connected via DAC to a 10Gbe switch. We'll be serving data over iSCSI.

I've done a lot of reading and I'm still confused. My question is - what sort of SLOG and/or L2ARC device should I be looking at?

1. iSCSI should be sync writes. So a SLOG should help here, correct?
2. Is it still recommended that you not use the same device for both SLOG and L2ARC? Or would a decent enough PCIE NVMe drive with power-loss protection work? What's the main danger with using one device for both?
3. For the SLOG - the Intel DC P3700 400GB is apparently the best budget option, or the Optane 900p - is this still the case?for the ZIL.
4. For the L2ARC - should I get another Optane 900p, or something else?
5. The fact my main storage is SSDs - will a SLOG/L2ARC still make a difference?

Regards,
Victor

 

[victorhooi]

Oh - and do the 16GB/32GB Optane M.2 memory cards work for a SLOG? (I assume they'd be too small for a L2ARC).

 

[Chris Moore]

iSCSI should be sync writes. So a SLOG should help here, correct?

Yes, absolutely necessary. The Intel SSD DC P3700 Series PCIe 3.0 NVMe drive makes a good one:
https://www.newegg.com/Product/Product.aspx?Item=N82E16820167232

Is it still recommended that you not use the same device for both SLOG and L2ARC?

Yes. Whether you need L2ARC or not is not certain. You might want to test performance without to see if it is good enough and only add one if needed.
The Intel SSD DC P3600 Series PCIe 3.0 NVMe drive is supposed to be good for it because of being more read optimized:
https://www.amazon.com/Intel-P3600-SSDPEDME016T401-1-6TB-Height/dp/B00L0LFI3M

Or would a decent enough PCIE NVMe drive with power-loss protection work?

We have at least one user that used the Intel SSD DC P3700 Series PCIe 3.0 NVMe drive for both when they built a home lab. In a business environment, I am not sure that is how you would want to do it but there is a link to his "AIO" build in my signature under the "Useful Links" button.

For the SLOG - the Intel DC P3700 400GB is apparently the best budget option, or the Optane 900p - is this still the case?

The Optane drive is a possibility, but I understand it is a bit more expensive.
Here is a link to a video where someone puts one through it's paces with FreeNAS in the position of L2ARC:
https://www.youtube.com/watch?v=oDbGj4YJXDw&t

The fact my main storage is SSDs - will a SLOG/L2ARC still make a difference?

Yes. A PCIe 3.0 NVMe drive is so much faster... Check thc video, great illustration.

 

[Chris Moore]

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

 

[Chris Moore]

Optane M.2 memory cards work for a SLOG?

I would say no, because they have no power loss protection.

 

[dturner71.dt]

Optane 900p 480GB:
https://www.amazon.com/Intel-Optane-900P-280GB-XPoint/dp/B0772T4BVZ?th=1
$573

vs
Intel DC P3700 400GB
https://www.newegg.com/Product/Prod...Series_SSDPEDMD400G401-_-20-167-232-_-Product
$650

Newer 900P is cheaper and faster. Does the 900p have all the same power loss protection the 3700 does? If it's the same, the 900P is a lot faster for SLOG, IMHO.

 

[Chris Moore]

Newer 900P is cheaper and faster. Does the 900p have all the same power loss protection the 3700 does? If it's the same, the 900P is a lot faster for SLOG, IMHO.

The 900P is "workstation" class, and might be fine for an L2ARC (like in the video I shared) but it doesn't have the same level of Power Loss Protection as the Intel DC P3700 400GB.

The server we just ordered at work has these two drives in it:

for SLOG: Intel 3D XPoint DC P4800X 375G PCIe
https://ark.intel.com/products/9716...00X-Series-375GB-2_5in-PCIe-x4-20nm-3D-XPoint

for L2ARC; Intel DC P4600 2TB,NVMe PCIe
https://ark.intel.com/products/122509/Intel-SSD-DC-P4600-Series-2_0TB-2_5in-PCIe-3_1-x4-3D1-TLC

 

[dturner71.dt]

So for SLOG, the more cost effective solution would be the P3700 at this time?

I read the review on the 900P and came away impressed with the performace upgrade over the 3700. :

https://www.servethehome.com/exploring-best-zfs-zil-slog-ssd-intel-optane-nand/

$400 to $600 is doable. $1500 not so much.

 

[Chris Moore]

So for SLOG, the more cost effective solution would be the P3700 at this time?

Cost, based on the links you shared, might be less for the Optane drive, but for SLOG, where power loss protection is indicated, the P3700 is still the way I would go if it were my money and my data.

The server we bought for work includes an Intel 3D XPoint DC P4800X, which is also Optane memory, but in a DC (data center) package and including the power loss protection. It is fast (very fast) but at a price.

It is all down to the performance you want (need) and the amount of money you have to spend. If you really don't have the cash, you might have to survive with a regular SSD or something.

 

[dturner71.dt]

I can purchase either the 3700 or the 900p.

Wish the 900p had the extra protection. Would it work if the APC was set to shutdown before total power loss?

 

[Chris Moore]

I can purchase either the 3700 or the 900p.

Wish the 900p had the extra protection. Would it work if the APC was set to shutdown before total power loss?

The reason for the power protection is for the times when you don't get to shutdown gracefully. It allows you to recover from this when the server comes back up, to complete transaction groups that were not yet sent to disk.

It is a risk, but if you are willing to take it for the additional performance and lower cost. Just be aware that any crash could have catastrophic results. Or not.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Stux]

You should see what sort of perf you get with just an ssd pool.

Then you can do some simulations and purchase hardware once you know where things can be improved.

 

[dturner71.dt]

Can you have 2 slogs on one device? Split a 3700 to assign to 2 different pools?

 

[Chris Moore]

Can you have 2 slogs on one device? Split a 3700 to assign to 2 different pools?

Technically, this can be done, but it is not a recommendation. If both pools were active at the same time, there would be some division of the performance of the card such that you might have less capability than you need with regard to latency. Your mileage may vary...

 

[sfcredfox]

Can you have 2 slogs on one device? Split a 3700 to assign to 2 different pools?

I just played around with that.
https://forums.freenas.org/index.ph...-partitioned-for-two-pools.62787/#post-452517

In my case, unless I completely maxed out the throughput of the SSD devices, the extra latency of two pools writing to the SSDs did not amount to anything noticable, but in fact reduced the latency of writing since there were too devices eating up the IO.

Situation dependant, but like Chris said, not a recommended/supported config. It did it mostly for fun and because I can in a home lab. I'd rather have the P3700s and am taking donations :)

EDIT: I paid more attention to detail and noticed you meant share one device between two pools. You could test that too. It will likely suck if your work load is throughput intensive, might be OK if it's just small IO intensive. Same comments about it being smart idea or not.