Fragmentation - considering ZIL

[rvassar]

So I'm shuffling my storage pools around. Moving infrequently accessed long term storage / media / photos, etc... to a RAIDz pool, block storage and things with parallel access to a dedicated block storage mirror pool, and lastly trying to get my security camera's off of either, and on to their own pool. One of the things I've been watching is the pool fragmentation percentage on the pool with the cameras. I suspect I'm suffering from the ZFS "gang block" fragmentation problem, with my pool fragmentation running in the high 30% range. Besides making swiss cheese of my pool, the problem with gang blocks is they way the whole gang block get loaded back into memory whenever any component fragment of the gang is needed. This hits the IOPS budget by forcing extra data reads, and consumes extra memory in the ARC.

So I'm considering adding a ZIL / SLOG device to help ZFS perform fsync() and still aggregate the writes in unfragmented allocations on the storage pool. I have a couple small Intel S3500 series SSD's I can add, which have PFP capacitors. They won't be fast by any means, but they should suffice. Back in the day, adding a ZIL was a "mirror is the only best practice" kind of deal, as loss of the ZIL caused all kinds of problems. I wondering if I can add one ZIL to the camera pool, and one to the block storage pool and forego mirroring.

My questions:

1. I'm under the impression, possibly incorrectly, that running a ZIL without redundancy is less risky on a modern OpenZFS filesystem. Is this true?

2. Will there be any ARC benefit from a reduction in fragmentation? The camera pool with be almost write-only, and very rarely read from. The block storage pool will see multiple clients.

 

[rvassar]

49 views and no comments... Ok, I'm going to assume the ZIL mirroring was put to bed back in 2014 when I wasn't paying attention.

 

[jgreco]

49 views and no comments... Ok, I'm going to assume the ZIL mirroring was put to bed back in 2014 when I wasn't paying attention.

More or less.

So I'm considering adding a ZIL / SLOG device to help ZFS perform fsync() and still aggregate the writes in unfragmented allocations on the storage pool.

... say what, now?

A SLOG has zero impact on how data is written to the pool. The intent log is a parallel process to the transaction group process and the intent log is not any sort of magic defragmentation cache.

https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

 

[rvassar]

More or less.

That good to hear. Though I held off because I decided to go read the code, and see what I could figure out. The OpenZFS source is quite the gem in that the comments are actually quite useful / informative for us "rusty and never were that good at C anyhow", trying to read it.

Line 2384-ish...
https://github.com/openzfs/zfs/blob/master/module/zfs/zio.c

... say what, now?

A SLOG has zero impact on how data is written to the pool. The intent log is a parallel process to the transaction group process and the intent log is not any sort of magic defragmentation cache.

https://www.truenas.com/community/threads/some-insights-into-slog-zil-with-zfs-on-freenas.13633/

There is some misinformation out there, but this appears to be the case. Furthermore, I'm now not convinced I'm seeing gang-block fragmentation. My newly created camera pool isn't even 40% and it appears to be fragmenting away just like it did prior to the pool re-org. The send/receive did a nice job defragging it from 40% down to 1%, but it's adding 1% per day as the pool fills. Since the Zoneminder data is rather regular, I'm wondering if I can fiddle with the record size of the pool and find a better match to minimize the fragmentation. Basically match the record size to the resolution of the cameras. Or perhaps its just the effect of the daily churn of data added & deleted. I've never seen it go over 40%.

Any thoughts / suggestions?

 

[jgreco]

Any thoughts / suggestions?

Bear in mind that the fragmentation metric in ZFS isn't like other fragmentation values you may have run across in the computer world. It measures fragmentation of the available free space. It has no idea how fragmented your data is.

You could fill your brand new pool with small files, sequentially, no fragmentation, remove a bunch of randomly placed ones, and end up with a very high fragmentation percentage, despite having no actual data fragmentation.