Slow? ZFS performance

[sretalla]

German magazine c't found in 2016/17 that at least the then current models could bear at least twice the TBW and sometimes much much more than that. The longest lasting one (Samsung 850 Pro, 256 GB) wrote 9.1 PB before dying.

YMMV is always a thing... spinning drives with 4 or 5 year MTBF are routinely seen reported in this forum still spinning after 9 or 10 years.

Expectation is what I was talking about... it should not be less than that TBW number (excepting fault/failure which would be warranty-covered).

 

[HoneyBadger]

Let's wind this back a little bit.

Unless it was manually set at the ZVOL level via sync=always, the OP was never using synchronous writes as ESXi doesn't request those from remote block devices (iSCSI/FC LUNs) - it instead expects the remote array to have non-volatile writes.

OP is also using an HP DL380e Gen8 and these servers don't usually come with HBAs, but rather with RAID cards. ("HBA Mode" is not an acceptable substitute, especially on the ciss driver based HP SmartArray cards.) I'm leaning towards this being an issue as the described scenario where SMB shares drop to 30-40MB/s is far below what you should get with even a single disk.

Regarding SLOGs, PLP, and UPS, if I can digress a little.

Sync writes with your ZIL on spinning platters ("no SLOG") are brutally slow, especially in small record sizes.

ZIL on an SLOG without PLP is a valid device, and will be faster than ZIL on platters - but these are usually not ideal devices because they don't offer enough performance, and often a PLP-less drive doesn't have the endurance of one that does. You also run the risk of your PLP-less device simply lying about its cache flushes - a much lower risk now, but it was non-zero in the early days of SSD's.

ZIL on an SLOG with PLP for in-flight data is the best option, because the device can quickly accept the write, queue it up to the input buffers for the actual NAND, and it's now considered "non-volatile" because of the SSD's PLP circuitry. You don't have to wait for that write to actually complete.

A UPS doesn't change the behaviour above - it just makes you less likely to have to rely on your SLOG if the power gets cut. A UPS also doesn't protect you against failure of the system PSU, motherboard, HBA, or software faults that might cause kernel panics.

It's not that "PLP makes it a good SLOG" but rather "good SLOGs usually have PLP" - and it's a massive difference. There's a community thread where we've been gathering information:

SLOG benchmarking and finding the best SLOG

I'd like to take a few minutes to talk about SLOG devices and what makes good ones versus bad ones. I have no doubt that this will be a controversial topic since this is not well understood by many people. In short, there's 3 things that you need for a "great" SLOG: 1. Fast throughput 2...

www.truenas.com

But the shortened version was summarized well by @Etorix already:

Radian RMS, Optane (DC), or data centre-grade drive with PLP and high endurance.

 

[hamstercnw]

Good tip. Thanks.