PLP'ted SSD-Pool - SLOG needed?

[dareinelt]

Hey,

I'm relatively new to the TrueNAS-universe. In my all day work I'm a systems administrator in a hospital that uses a NetApp Metro Cluster for general file storage and serving storage to our ESX-farm - so maybe the totally different part of the whole storage world compared to TrueNAS when it comes to configuration and sometimes features.
But I'm someone who likes to educate and introduce myself tho something new, so I want to get some TrueNAS practice in my homelab. The first box is running fine for a while - it's only there for some medium-fast file-access yet, but I want it to serve my virtualization-box some block storage options.
After some research I hang on getting some Intel SSD D3-4610 with 480GB and put at least four of them in striped mirrors. They will be presentated via iSCSI to my virtualization-box.

For better insight; here are my configurations for filer and hypervisor:

• Supermicro X9SRi-F
• Intel Xeon E5-1620 v2
• 64GB DDR3 ECC Reg
• LSI 9207-4i4e IT-Mode
• Supermicro SuperChassis 826TQ-R800LPB
• Backplane replaced with BPN-SAS2-826EL1
• 8x WD RED EFRX 2TB -> 4 vdev striped mirrors
• TrueNAS-12.0-U2.1

• Dell Poweredge R630
• 2x Intel Xeon E5-2630 v3
• 384GB DDR4 ECC Reg
• 2x 120 GB SDD for Proxmox VE 6.3
• 256GB Samsung PM981a for local storage in PCIe->M.2 adapter card

I don't have experience with using block storage in FreeNAS - I've only read the typical and often mentioned threads here in the community. So it seems, that the basics are the same compared to other solutions.
Now to the main-question I'm writing here: Is it really needed to have an SLOG-device for performant sync-writes if the SSDs in the pool to be stored on are already powerlossprotected? Is the acknowledge of the succussfull write transfered "out of the pool" or will I need something like an Optane/RMS-200/SS530 to get fast(er) sync-writes in this configuration?

 

[jgreco]

When you write something in ZFS, it is stored in a transaction group in main memory, and then flushed out to disk (or SSD) at a later time. Without SLOG, the power loss protection on your SSD's is meaningless because nothing is protecting the data while it is in memory. Power loss protection on pool devices is about as useful as a screen door on a submarine.

Please refer to

https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

 

[kdog2]

@jgreco

Without an SLOG, then the ZIL is written to the pool, and thus the PLP for an SSD-backed pool *is* important, isn’t it so?

Also, from an latency perspective, since DRAM-backed SSDs are pretty fast (and more so with NVMe), again using the SSD-backed pool for the ZIL seems that it might be okay.

A reason for not doing this might because writing the ZIL to the pool might reduce its endurance, so having a cheap/disposable SLOG (think: used S3700s as burners) might be a better ROI. Thoughts?

 

[jgreco]

@jgreco

Without an SLOG, then the ZIL is written to the pool, and thus the PLP for an SSD-backed pool *is* important, isn’t it so?

Not really in practice. In-pool ZIL writes are incredibly expensive compared to a proper SLOG, and even if your pool is SSD, the in-pool ZIL writes are a LOT slower. You might be able to argue that this would be useful if you had something that was writing at a low velocity (maybe some sort of audit logs?) where you absolutely couldn't lose it, but why wouldn't you be doing that to hard disk instead? People buying high quality SSD are probably going to be expecting best-of-class performance, and that's just not going to happen with in-pool ZIL. You would basically need something where you had an application that demanded huge read performance, and tepid write performance with a sync write requirement, and then, okay, yes that might make sense.

So I think that's wrong in practice. If you buy an SSD pool and you want sync writes, it is hard to imagine not going for a proper SLOG device. Not impossible, just ... highly unusual.

It's an interesting thought exercise in any case. ;-)

Also, from an latency perspective, since DRAM-backed SSDs are pretty fast (and more so with NVMe), again using the SSD-backed pool for the ZIL seems that it might be okay.

A reason for not doing this might because writing the ZIL to the pool might reduce its endurance, so having a cheap/disposable SLOG (think: used S3700s as burners) might be a better ROI. Thoughts?

SLOG selection typically works out to picking a device suitable to your expected workload. I think the majority of people have been headed towards Optane. If that's too pricey, then, yes, burner S3700's are available cheaply on eBay.

 

[rvassar]

A reason for not doing this might because writing the ZIL to the pool might reduce its endurance, so having a cheap/disposable SLOG (think: used S3700s as burners) might be a better ROI. Thoughts?

On the "burners".... I've been casually experimenting with this. I have a NFS-only pool comprised of a pair of 4Tb HGST 7200 rpm disks. Since it's only 2 disks mirrored, it's not a performance config, but since all the activity is NFS, the SLOG certainly helps. On the used S3700's... Remember, you don't need a lot of capacity. I've been picking up 120gb S3500's with 85% write endurance remaining for ~$20/ea. They're maybe 2/3rds the speed of the S3700's, but faster than my pool devices, and you can't beat the price. I've been watching the write endurance, and seem to be consuming 0.5% per month.

 

[HoneyBadger]

Now to the main-question I'm writing here: Is it really needed to have an SLOG-device for performant sync-writes if the SSDs in the pool to be stored on are already powerlossprotected? Is the acknowledge of the succussfull write transfered "out of the pool" or will I need something like an Optane/RMS-200/SS530 to get fast(er) sync-writes in this configuration?

You'll definitely have better performance with a dedicated SLOG device, although it will vary based on how strong your pool vdevs are. I have a pool of Intel DC S3500 SSDs that I run sync=always on for an iSCSI workload. Without my (Optane) SLOG it's about a -60% hit to random I/O performance; sequential is largely unaffected.

Since block storage almost always lives or dies by random I/O performance, the small investment to add a powerful SLOG device makes sense, especially if you've already laid out the cash for an all-SSD pool. As @jgreco mentioned, it's "possible, but very unusual" - why spend a few thousand dollars on fast NAND only to save a few hundred on a good SLOG that will really let it shine?

(I should do an experiment to see if throwing a couple of the same model SSD in as dedicated SLOGs helps just from a latency reduction perspective as well, just for giggles.)

A reason for not doing this might because writing the ZIL to the pool might reduce its endurance, so having a cheap/disposable SLOG (think: used S3700s as burners) might be a better ROI. Thoughts?

Yes - the small (32K or less) writes to in-pool ZIL would end up being "written twice" from my understanding of how the current code works. A separate SSD would offload some of that and reduce the wearout rate of the main pool SSDs.

On the used S3700's... Remember, you don't need a lot of capacity.

You don't need a lot of usable capacity, but SSD performance often scales with total capacity (eg: increased parallel NAND channels) as well as benefiting endurance.

 

[Etorix]

A reason for not doing this might because writing the ZIL to the pool might reduce its endurance, so having a cheap/disposable SLOG (think: used S3700s as burners) might be a better ROI. Thoughts?

I had the same idea to reduce wear, but went with an Optane NVMe because it doesn't make sense to spare a few hundred euros on the SLOG while spending thousands on SATA SSDs, provides better bandwith to the SLOG and saves all SATA ports for storage.

 

[rvassar]

You don't need a lot of usable capacity, but SSD performance often scales with total capacity (eg: increased parallel NAND channels) as well as benefiting endurance.

That is very true, and the Intel S3500's are probably one of the most noticeable examples of this. There's a nearly 100% swing in performance between the 80gb and 480gb models for the S3500's, and almost as much for the S3700's. I'd link the paper, but I read it at work and can find it publicly.

But at this point those drives are considered "burners" by most everyone reading this.