very slow performance with PCI-E SLOG, wrong config?

[Lesani]

Hello! I have issues with the performance of my NAS (see signature)

I am currently running 4 spinny disks in Raid-Z2. Those will never be terribly fast, and that is OK.

To speed ingress into my NAS up, I installed my 240GB PCI-E Intel Optane 900p into the server, and set it as slog. My understanding was that this enables the system to ingress data up to the SSDs speed, and then over time write it to the array.

My system does that up to 2GB approx, then it drastically slows down to 65MB/s

This is independent of copying an incompressible file over the network or copying one locally. The incompressible file was created writing 25GB from /dev/random using DD

My intended behaviour would be to ingress data at max speed until the SSD is (nearly) full, and only then slow down to the array speed, which is a scenario I won't encounter often as I don't often (ever?) copy a stack of files >240GB

nvme0n1 is my optane, and the only nvme SSD in the system

copying using rsync with the --progress parameter I see it going to 380mb/s for a few seconds and then dropping down rapidly

ARC goes up to 60GB on my system and is/was sitting at 60GB even before this file transfer started

I know that speeds, especially over network or onto spinny HDDs etc can be affected by record sizes etc, but the behavior and speeds here are so far away from anything this SSD is capable of doing, even on PCIe-Gen2, no matter any block/record size.
Also as I understand with a SLOG in the form of an optane one should keep "sync" on "always" as the slog takes over the sync

For referency my record size is 128kb at this time.
Sync on "always
Compression LZ4
Atime off

 

[Etorix]

To speed ingress into my NAS up, I installed my 240GB PCI-E Intel Optane 900p into the server, and set it as slog. My understanding was that this enables the system to ingress data up to the SSDs speed, and then over time write it to the array.

No! SLOG is NOT a write cache, it is a log.
During sync writes the SLOG fills up to one transaction group (ca. 5s worth of data), not more. Then this amount of data is committed to storage—from the RAM which IS the write cache in ZFS—before more data is taken in.

Which is what you see.

Also as I understand with a SLOG in the form of an optane one should keep "sync" on "always" as the slog takes over the sync

Indeed, without sync writes a SLOG does nothing and serves no purpose. But decision is done the other around: First, decide whether you need sync writes; then, if the answer is "yes", you consider adding a SLOG.

For performance, set "sync=never" (and do not use a SLOG).

 

[sretalla]

You probably should read these:

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

 

[Lesani]

Oh, I did read a few articles about SLOG and the way it was worded or maybe the way I read them it sounded basically like a write cache.

So, even though I have a non-volatile SSD that could without issue be a safe write cache, there is no way for me of using it that way, and the only way to increase the write cache would be to modify how much cache it has in ram, which is (in my case) safe for power outages since I have a UPS, but not safe for crashes.

 

[Etorix]

UPS is no proper substitute for Power-Loss Protection. But even if you had terabytes of RAM, ZFS would still flush writes after each transaction group.
For performance the question is: Does your use case needs sync writes?

 

[Lesani]

No I probably don't need sync writes, but even at the max performance of my 2(+2) drives I won't saturate a 40GBit network link for write, which I have installed on my nas as well as on my home PC. This is why I was looking into an SSD as write cache.

 

[Lesani]

Add: and with sync writes off the situation isn't much different, some 4GB written at (higher than with sync writes) good speeds, then it drops and wildly fluctuates between 40mb/s and 120mb/s. This is locally copying a /dev/random generated file again, not over network. So in this case it's questionable if adding some level of danger witout sync writes is worth it.

 

[Patrick M. Hausen]

This is locally copying a /dev/random generated file again

/dev/random is not a constant speed source of data. Reading will frequently stall when there is not enough entropy.
Use /dev/zero and turn off compression on a test dataset.

 

[Lesani]

Hi, thanks for the suggestion, if you read above I am not using /dev/random directly, I have pre-written a 25GB file from there, and am using that file