Speeding up SLOG on the cheap

[Sirius]

Currently I use 2 x HGST 100gb SLC SSDs as SLOGs. They work well but they're not as fast as I'd like (at least with non-sequential writes) and I can't afford a PCIe NVMe SSD.

I'm wondering, is it possible for me to add 2 more of them and have that speed up the SLOG? Currently I'm underprovisioning the drives to a 10gb partition as that seemed to be the recommended SLOG size for a 10gigE capable network.

The two current ones are in a mirror, so my theory is I just add another 2 in a mirror and then add that mirror as a log somehow... is that even possible or should I just try and save up for an NVMe drive? Also, if I do go the NVMe route... do I need two drives or can I get away with a single one? I've managed to do this system on quite a small budget so far but it's reasonably solid and 2 x NVMe drives would blow the cost out easily.

Thank you for any advice :)

 

[Stux]

Pretty certain you’ll just end up with a 4-way mirror

A battery backup raid card can be used as a slog.

 

[Sirius]

So I could use 2 x battery backup RAID cards, each with 1 or 2 SSDs and that'd be fine? I did read about the RAID card option before... I believe it also means I can use consumer SSDs, eg. an 850 or 860 Pro instead of needing ones with PLP?

Is there a minimum amount of cache I need on one of those cards for it to work or is any amount fine?

Are there any other potential downfalls? Apart from having to replace the battery occasionally, I assume they last a reasonable amount of time though.

 

[Arwen]

Pretty certain you’ll just end up with a 4-way mirror
...

If I remember correctly, you CAN have multiple SLOGs devices, either as Mirrors or stripes, or Mirrored stripes. Just be careful on adding them. One wrong move, and you grew your pool... not added a SLOG.

 

[Mirfster]

Personally, I would avoid consumer grade SSDs for SLOG altogether; even if you were to implement a Battery Raid Controller. I tried that out a long time ago and it really is not worth it IMHO.

Have a read of this past thread, there is plenty of data here by myself and others: https://forums.freenas.org/index.php?threads/slow-writes-on-ixsystems-hardware.46032/

You could get a used Intel DC S3710 200GB for ~$180.00 (may be newer and cheaper models though); that is what I got in the past.

Also, SLOG(s) can be removed and added pretty easily without issues (I showed the steps as well in that thread for the OP). Just be careful as @Arwen mentioned...

 

[Sirius]

If I remember correctly, you CAN have multiple SLOGs devices, either as Mirrors or stripes, or Mirrored stripes. Just be careful on adding them. One wrong move, and you grew your pool... not added a SLOG.

Is there any documentation for how to do that? Most stuff I found to do with SLOGs is just for setting up a single or mirrored SLOG as opposed to a striped mirror.

Also, would there be much benefit in swapping the HGST drives for Intel ones? Is there anyway to benchmark it as I don't really want to invest into 2 x Intel drives for a SLOG and then discover they perform the same as the HGST ones. Unfortunately in Australia the proper Intel DC SSDs are expensive and hard to find :(

 

[Mirfster]

Also, would there be much benefit in swapping the HGST drives for Intel ones? Is there anyway to benchmark it as I don't really want to invest into 2 x Intel drives for a SLOG and then discover they perform the same as the HGST ones.

If you check page 3 post #42 of the link I provided, you will see some actual benchmarks using different SLOGs. Even between using a consumer class SSD (Samsung Evo) and an Intel DC, there is a drastic performance difference.

Also, I was not using mirrored SLOGs either just a single SSD SLOG.

 

[Sirius]

Well the HGST drives aren't consumer ones... they're this model: https://www.servethehome.com/hitachi-hussl4020ass600-200gb-slc-quick-benchmarks/

So PLP, SAS, SLC NAND, huge write endurance, etc... It's just I'm not sure how they'd compare against something a bit more 'modern', say an Intel DC S3710 or Intel DC S3610.

Unfortunately as I'm in Australia, I can only seem to find the 200gb DC S3610 for ~$250 AUD
https://www.megabuy.com.au/intel-ssdsc2bx200g401-drive-p550004.html

I can find the DC S3710 but only in the 800gb size and it's ~$1600 AUD...
https://www.arc.com.au/product/58512

I didn't really want to spend ~$500 on 2 x Intel S3610s if the performance was not really worth it. Also, maybe I'm mistaken but I thought it was a best practice to mirror the SLOG? Or is that not as necessary nowadays? $250 is a bit easier to justify, but again only if it genuinely has a performance impact.

 

[Mirfster]

Yes, it would be considered best practice to mirror the SLOG (always think redundancy). However, I choose not to. I do have a 2nd S3710 actually in the system, but it is not assigned and I can always allocate it as the SLOG if the other one should fail.

I guess the best scenario would be if you were able to borrow one from a colleague to test with; but that may be out of the question.

On a side note, are you running iSCSI or NFS?

 

[Stux]

In the past, if your SLOG failed, your pool failed. That is no longer the case, so mirroring your SLOG is only really to save transactions if your SLOG fails at the same time as your system reboots. If it fails while your system is online, then it’s just removed, your ZIL is written to your pool, and the system continues as if no SLOG was configured in the first place.

I demonstrate a technique of working out the maximum possible SLOG performance in this thread:
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

 

[Mirfster]

Okay, so this other thread you have (What helps with improving Samba speeds?) would lead me to believe you are using CIFS/SMB or are you going to be running iSCSI? Confused since you have benchmarks for both in the other thread...

 

[Sirius]

Okay, so this other thread you have (What helps with improving Samba speeds?) would lead me to believe you are using CIFS/SMB or are you going to be running iSCSI? Confused since you have benchmarks for both in the other thread...

I use a mix of SMB and iSCSI. For the 12 x 4TB pool I use SMB, for the 8 x 3TB pool I use iSCSI. I've set sync=always on both pools. There's no L2ARC.

The iSCSI share is for my steam, origin, battle.net, etc games. From what I can tell due to game access patterns, random reads are the most important. The SMB share is for media and other general purpose stuff. As a result, sequential writes work best for it but I don't mind if I sacrifice sequential writes (and reads) to get better random writes (and reads)

Currently I'm using 1gigE but I do plan to use 10gigE, I have the right switch I just need the NICs and SFPs.

The SLOG for both pools is 2 x 100gb HGST SSDs, each one has been partitioned into 2 x 10gb partitions and the rest left empty. This means each pool has 2 x 10gb mirrored SLOGs. I could theoretically add 2 more HGST drives so each pool gets 2 x SLOG devices but considering I'm usually the only user I thought sharing the SLOGs would be OK. Maybe that was a mistake...

Please let me know if that doesn't make sense or if you need further info. I hope I'm explaining correctly.

 

[Arwen]

Is there any documentation for how to do that? Most stuff I found to do with SLOGs is just for setting up a single or mirrored SLOG as opposed to a striped mirror.
...

From the command line, this is roughly what is done. FreeNAS may have some GUI tool, but it's been too long since I used it for my own SLOG. Note the KEYWORD "log" in the add command line. Without it, you grow your pool by adding the SLOG device, which is basically un-recoverable at present.

Adding a log:
zpool add MY_POOL log sda

Adding a mirrored log:
zpool add MY_POOL log mirror sda sdb

Adding a striped log:
zpool add MY_POOL log sda sdb

Adding a striped, mirrored log:
zpool add MY_POOL log mirror sda sdb log mirror sdc sdd

Attaching a mirror to existing striped log device:
zpool attach MY_POOL sda sdb

 

[Mirfster]

Hmm, something seems off to me about the setup (like SLOG on CIFS/SMB with "sync=always"). But, maybe it is my mind deciding to take a vacation...

However, I have to leave soon and won't be back online for a while.

I'll ping a few other contributors to get their attention and input.

In the meantime, give this info a read over "Some insights into SLOG/ZIL with ZFS on FreeNAS"

@Ericloewe , @danb35 , @joeschmuck any chance you all have thoughts on this? ;)

 

[Sirius]

I was under the impression sync=always was a good idea with regards to data integrity? If sync=standard is fine then I'm happy to set that for my pools.

No worries, thank you for your advice so far.

Also, thank you Arwen for those SLOG commands!

 

[Zredwire]

Currently I use 2 x HGST 100gb SLC SSDs as SLOGs. They work well but they're not as fast as I'd like (at least with non-sequential writes) and I can't afford a PCIe NVMe SSD.

Have you tested your speeds? If so what speeds are you seeing?
Stux linked to his thread on testing (above), and it is an excellent guide. It is important to have some actual numbers to talk about so others can help.
Here is Stux's link from above: https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561/

 

[Sirius]

Hmm... the more I think about it the more I think that the SLOGs aren't actually the problem and I'm blaming them incorrectly... Anyway, here's the performance I get using them currently...

Is it normal for iSCSI to be *that* much faster vs SMB? Is there anything I can do to optimise SMB performance, specifically the 4KiB reads/writes?

Thanks for the advice, I'll read Stux's thread again and see if I missed stuff or misunderstood (entirely possible)

 

[c32767a]

In the meantime, give this info a read over "Some insights into SLOG/ZIL with ZFS on FreeNAS"

@Ericloewe , @danb35 , @joeschmuck any chance you all have thoughts on this? ;)

This.. :)

Somewhere there's a discussion about a poor man's slog using a (supported) RAID controller with a BBU, cache RAM and a small mirror of SSDs or spinning rust. The BBU in the raid controller allows the writes to complete to RAM instantly and doesn't have the wear issues of a SSD. If the ramdisk test proves the value of a SLOG in this use case, this could be a less expensive option.