Are read and write caches recommended at all flash setups?

[Scampicfx]

Hey gents,
when looking at the R series of TrueNAS systems I noticed that the all flash systems do not include any cache devices, see:

TrueNAS R-Series with OpenZFS: Scale-Up & Out Storage

The TrueNAS R-Series combines the powerful features of OpenZFS with budget-friendly, single controller hardware for scale-out storage.

www.truenas.com

My question is: Are read and especially write caches recommended at all flash setups?
My use case would be storage for ESXi with virtual machines; however no databases are hosted on them!

What setting of sync would you recommend at all flash setups? sync=standard or sync=always ?

I would like to highlight that so far I always used sync=always to assure complete data integrity at hybrid pools.
I am not sure what to choose at all flash pools?

 

[jgreco]

Read cache comes in the form of ARC and L2ARC.

ARC is system RAM, so yes, absolutely, hella-lotta-ARC-read-cache is recommended.

L2ARC is flash. If you have some flash that is FASTER than your pool flash, MAYBE that could be useful, but, overall, probably not that much.

Write cache comes exclusively in the form of system memory. iXsystems likes to call SLOG devices "write cache", which they are not. SLOG devices are useful for their own reasons, but in no case can they make a pool faster than sync=disabled, so they are clearly no sort of write cache.

You would want to use sync=always anywhere that you would be afraid of data corruption damaging your datastores of VM's. This does not change in any way for all flash pools, which are simply substituting flash for hard drives.

And here's why I get prickly about the terminology iX is using:

There is no difference between flash and hard drives in the general write path. When you write a block to ZFS, it goes into the write cache -- which is a transaction group in system memory, and then within a few seconds, that gets flushed out to the storage pool.

Pulling the plug on a HDD-backed pool or a fully PLP-protected SSD pool that does not have sync write protection results in the exact same data loss scenario. You lose the stuff that was in the write cache -- i.e. system memory.

Therefore you probably do want some sort of SLOG device, probably Optane something or other.