SLOG - Mirror or Stripe?

[indivision]

I have 2 nVME devices that I would like to configure as a SLOG.

The GUI gives a warning if I try to stripe them. But, isn't there a performance cost if I mirror them instead?

I am not too concerned about losing the 5-30 seconds of data if they fail.

Just looking for best practices for configuring multi-drive SLOG.

 

[sretalla]

If you're prepared for data loss, why not go without it and just do async IO?

The whole purpose of SLOG is to ensure that no data loss occurs.

I think you can probably find a bunch of posts from @jgreco saying that async IO outperforms any SLOG hands down (and SLOG only ever holds a small number of seconds of data at any given time, so the most important question is can my pool cope with that much IO to flush out the SLOG before it causes pool IO to back off... remembering spinning disks are usually 100-200 IOPS each and NVME are usually 50'000 IOPS each).

I'm really not sure that throwing massive amounts of excess capacity at it in order to win Stripe performance is an advisable path.

 

[jgreco]

I have 2 nVME devices that I would like to configure as a SLOG.

Okay.

I am not too concerned about losing the 5-30 seconds of data if they fail.

Then I suggest that you fail them immediately by not installing them.

Basically what @sretalla said... If you are not worried about losing a small amount of data, what's the point of installing SLOG?

Striping it would basically double the chances for the SLOG to fail. The usual desire is to make sure SLOG doesn't fail, by installing redundant SLOG devices. What kind of devices are these, anyways? Lots of NVMe devices are not qualified for SLOG use as they don't have power loss protection.

 

[indivision]

If you're prepared for data loss, why not go without it and just do async IO?

Thank you. I should have given more context in my original post. I've used FreeNAS/TrueNAS for many years with great performance. But, I came across severe performance issues when working with complex (ie. frequent/random) writes to very large files. I was able to improve performance slightly by using an iSCSI share. But, it was still pretty much unusable for that specific use case (everything else I've ever thrown at it worked fine).

I read that a SLOG can help with performance of sync writes which apparently applies to iSCSI, NFS, etc.

I'm not sure exactly what "async IO" is? Is that changing the sync config for the vdev to "Never"? If so, I tried many combinations of that in my previous tests and couldn't get usable performance.

The whole purpose of SLOG is to ensure that no data loss occurs.

I think you can probably find a bunch of posts from @jgreco saying that async IO outperforms any SLOG hands down (and SLOG only ever holds a small number of seconds of data at any given time, so the most important question is can my pool cope with that much IO to flush out the SLOG before it causes pool IO to back off... remembering spinning disks are usually 100-200 IOPS each and NVME are usually 50'000 IOPS each).

I'm really not sure that throwing massive amounts of excess capacity at it in order to win Stripe performance is an advisable path.

I was able to make it financially efficient by picking up a few older Optane 16GB sticks: https://www.amazon.com/gp/product/B07CGYTXQK/ref=ppx_yo_dt_b_asin_title_o00_s00?ie=UTF8&psc=1

No wasting 250GB, etc. :)

Basically what @sretalla said... If you are not worried about losing a small amount of data, what's the point of installing SLOG?

To improve performance for a specific use-case. Maybe there was a bias somehow in my google searches. But, from what I've been reading so far in various articles about SLOG it's more about performance than data protection.

Striping it would basically double the chances for the SLOG to fail. The usual desire is to make sure SLOG doesn't fail, by installing redundant SLOG devices. What kind of devices are these, anyways? Lots of NVMe devices are not qualified for SLOG use as they don't have power loss protection.

Understood. Though, in this case, it's not critical data. It's actually a one-drive pool to begin with. So, the drive/pool itself is probably more likely to fail long before these new nVME drives.

I linked to the drive above. It does not have PLP. I would assume that could be a problem if I used the space for Virtual Machines... But, I read an article saying that Optane doesn't need PLP because the data isn't stored in "limbo" (or not for as long) compared to NAND technology. Is that true?

I went ahead and set these up as logs and I'm happy to report that they have dramatically improved performance on iSCSI. But, I'm still interested to learn more from you all about the best way to do this (and whether or not I wasted $30 on these drives :) ).

 

[jgreco]

I read that a SLOG can help with performance of sync writes which apparently applies to iSCSI, NFS, etc.

I'm not sure exactly what "async IO" is? Is that changing the sync config for the vdev to "Never"? If so, I tried many combinations of that in my previous tests and couldn't get usable performance.

Async writes are always the fastest that a pool can go. Sync writes are always slower. Sync writes without a SLOG are slowest. However, sync writes with the fastest SLOG on the planet can not be faster than async writes.

If you have a pool design problem where async writes are not performing acceptably, a SLOG and sync writes will only worsen that problem.

 

[indivision]

Async writes are always the fastest that a pool can go.

Isn't there an additional layer of consideration? In other words, if async is always the fastest solution then why is sync ever used at all under any circumstance?

Sync writes are always slower. Sync writes without a SLOG are slowest. However, sync writes with the fastest SLOG on the planet can not be faster than async writes.

In my specific use case the results were a little different:

async via SMB was unusably slow working with the writes/filesize required.
sync via iSCSI was marginally faster but still not usable with the same writes/filesize.
sync via iSCSI with SLOG works quite well with same writes/filesize. BUT, it is, as you suggest still a bit slower than async via SMB using typical writes/filesizes.

If you have a pool design problem where async writes are not performing acceptably, a SLOG and sync writes will only worsen that problem.

I tried many async based variations of the pool in this case before moving on. But, it's definitely possible that I did something wrong. So far, iSCSI with the SLOG is the only variation that has worked.

What would a pool design be that works with good async performance in my particular use-case: 1TB+ sized file(s) with large numbers of small writes to it (not simply transferring the entire file back and forth).

I have the ability to test quickly and would definitely prefer to use regular async for this if it's possible.

 

[danb35]

if async is always the fastest solution then why is sync ever used at all under any circumstance?

Because speed is rarely the only consideration, and often isn't even the most important consideration.

 

[jgreco]

In other words, if async is always the fastest solution then why is sync ever used at all under any circumstance?

Rockets are very fast. We don't use them as our exclusive mode of transportation because sometimes people want to be able to do things like get places safely, steer, park, and be able to return home....

Sync is used for situations where you need to be able to guarantee that data written *is* written. Would you like your bank to commit your payroll deposit to its ledger with async writes, and one day the power fails, and your bank signals that it has "received" the money, but fails to post it to your account because the write was async?

If you have hypervisors running virtual machines, and a VM tells a virtual disk to write a block, let's say critical filesystem metadata for the virtual disk, and the NAS crashes/loses power/wotevr, that write could be lost without a sync write mechanism, which could result in severe damage to the virtual disk because the VM's cache and what it believes to have been written to disk are not the same thing as what is actually out there.

This is pretty much basic compsci filesystem theory stuff, and if you want a much better delve into the topic, I suggest you check out general discussions of POSIX sync writes on the Internet. This is necessary in so many contexts.

 

[indivision]

Why would async not perform to a functional level when doing many writes to a 1TB sized file?

 

[jgreco]

Why would async not perform to a functional level when doing many writes to a 1TB sized file?

Because async doesn't provide the safe write handling that POSIX sync semantics require. The size of the file is irrelevant. If you have a block of data and the filesystem says "OK, written", one of two things can happen:

1) it has actually been written to disk in a manner that guarantees that any future request for that block will reliably retrieve that data, regardless of loss of power, system crash, etc. The operative words are "committed to stable storage."

2) it is merely accepted by the filesystem, cached in memory for future write to disk in the near term, but if someone were to hit the reset or the system were to crash, the data would be lost, because RAM is not stable storage.

For further education, please refer to https://www.truenas.com/community/t...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

 

[indivision]

For further education, please refer to https://www.truenas.com/community/t...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

Thank you. Helpful write-up.

If I am understanding correctly, the likely reason it was slow on SMB was because sync writes were being requested. (Though, I did test it with the sync setting in all positions. Maybe it needed a reset between to see the change?)

Also, according to that, it sounds like using a SLOG device as I am is valid and expected to improve speed for sync writes as I am seeing.

 

[indivision]

(Though, I did test it with the sync setting in all positions. Maybe it needed a reset between to see the change?)

Actually, reading what other people were trying in that thread... I think that I had tried "atime" in different settings. I probably left "sync" alone in those tests. So, that would explain everything.

 

[sretalla]

Actually, reading what other people were trying in that thread... I think that I had tried "atime" in different settings. I probably left "sync" alone in those tests. So, that would explain everything.

If we go back to your original statement that you were prepared for a little loss to get the best speed, maybe you should still investigate sync=never.

 

[indivision]

If we go back to your original statement that you were prepared for a little loss to get the best speed, maybe you should still investigate sync=never.

Yes. You're right. I am going to try that out now.

Though, now that I have the drives, it might be worth keeping that setup since it apparently adds some data integrity improvement and performs at a usable level now.

 

[ChrisRJ]

If you are talking about a stripe for SLOG, that would not be safer than async writes.

What devices did you purchase for SLOG purposes?

 

[indivision]

If you are talking about a stripe for SLOG, that would not be safer than async writes.

Thank you. Noted.

What devices did you purchase for SLOG purposes?

I had 2 unused M.2 spots and have maxed out my SAS/SATA ports. So, I picked up 2 of these: https://www.amazon.com/gp/product/B...dt_b_asin_title_o00_s00?ie=UTF8&psc=1&ozlp-20