Gifted a SM846 system which I am upgrading. Looking for feedback, suggestions, and guidance as a first time TrueNAS user and homelab builder

[Koop]

Happens to everyone at some point. Worst I've had was a Noctua NF-F12 iPPC at 2-3k RPM. Less linear velocity, but a very sharp leading edge.

Oh yeah, don't worry I've seen and probably done worse in my time roaming data centers. I've eyed my same Noctua fans that I have jammed in that USPS box with caution now haha. Those are 3x NF-A14 iPPC-3000s to help with additional drive cooling. The fans used by Supermicro are much smaller and obviously much faster. The housing is perfectly fine for protection against the fan itself unless you're a bonehead like me haha. It's all good though. Actually looking at the fan now it's kind of surprising how I managed to get my finger in there fast enough to do damage. https://store.supermicro.com/80mm-fan-0127l4.html

If the old one is still around, you can try re-pasting the heatsink on the expander, SAS2 expander backplanes are all probably 10+ years old by now. Also be sure to update the firmware on the backplane, if you can get it that far along - it's easy to miss, but high impact. Also, is this with a SAS2 or SAS3 HBA?

So I was actually curious on how to do this because the heatsink is directly attached to the controller it seems. Any ideas? I assume it's more of a thermal adhesive but I'm not too familiar with replacing. I would assume a heat gun. I wonder if that's really necessary though. I assume I'd have to replace the whole heatsink or some new interface (thermal tape)

 

[Ericloewe]

Hmm, depends on what they're using, but it would probably be a thermal adhesive pad. Typically those are of the twist-to-remove type, but that's not universal, so don't go blindly trying to pry it off.

 

[Koop]

Hmm, depends on what they're using, but it would probably be a thermal adhesive pad. Typically those are of the twist-to-remove type, but that's not universal, so don't go blindly trying to pry it off.

Yeah. I messed with a few HBAs and some were just thermal paste while others were definitely a glue. Then there's pads as well.

To be honest the backplane is pretty much shot as-is- it's unusable in it's current state. I almost want to keep it so I can experiment with it but I'll just send it back for a full refund. Thanks Ebay! Instead I've invested in some other interesting pickups. More news to follow, maybe will make a different topic about what I find. All hardware related but still relevant to TrueNAS.

 

[Koop]

Just an update that I've gotten everything up and running as expecting and moved a bunch of data onto my Scale system, hooray.

Moved a bunch of bulk data over from my desktop. started playing with some VM creation and app creation. Tried to play around a bit with some permissions and ACLs, all good fun- eventually called it a night.

For a day or so I noticed something interesting. Every five seconds I would see all my disks light up with activity. This is consistent and constant. Hmmm ok what's happening every five seconds... It must be sync writes being committed from my SLOG device? I don't say that with authority but from what I understood that happened- every five seconds eh? So that makes a lot of sense. I had put in a mirror pair of optane drives for SLOG on my spinnin' pool this just because they were super cheap, figured why not.

I found a very funny "Solution" on the forums- just make it wait longer!

Code:

echo 10 >/sys/module/zfs/parameters/zfs_txg_timeout

Obviously I found this kind of odd. Changing how this works seems like something you fundamentally shouldn't mess with. It did raise the question in my mind though- why is it every 5 seconds? Why not shorter or longer periods of time? Genuinely curious if this could be elaborated on.

I did play with this though just to see. Set it to 1 second to see all my hard drives click and flash every single second. Set it to 60 and watched them light up every sixty seconds. I put it back to the default. Obviously changing something so fundamental didn't seem "wise" or what I was really seeking to figure out.

Because It obviously still didn't answer the question. What was actually doing these writes? From more searching I found out about the System Dataset and how that was set to my large spinning disk pool. I changed it to use my boot pool instead but no change.

Finally it clicked- it was the apps. Well it clicked because it was literally the only other thing it could be haha. Then it made sense. Of course they're constantly writing something- I was playing with the netdata app. Constant monitoring how got to generate something right? So I turned off all my apps but it still did it. So I figured this must be something fundamental to how apps work on Scale. So I figured it means I should try to moves where the apps live off my spinning disk pool.

So my solution was to get an M.2 drive I had laying around, slap it into a M.2 NVME to PCIe adapter and move the apps over to that instead. Low and behold no more activity every five seconds.

Was there anything else I may have been missing in figuring this out? Think it's a good idea System Dataset Pool on my boot pool (2x SataDOMs) or perhaps move it to my NVMe? I think I'll expand my NVMe pool to a X16 card w/ bifurcation and use 4 NVMe drives in the future to accomidate more apps and a fast storage pool in general. I aint mirroring those things though. Stupid M.2 drives are expensive.

 

[HoneyBadger]

Obviously I found this kind of odd. Changing how this works seems like something you fundamentally shouldn't mess with. It did raise the question in my mind though- why is it every 5 seconds? Why not shorter or longer periods of time? Genuinely curious if this could be elaborated on.

It's not so much "don't mess with it" as "understand the function of the knob you're turning" - the txg timeout threshold is the duration after which ZFS will close off your pending transaction group and say "okay, no more updates here, start a new one." Like micro-snapshots, in a sense, driven by the Copy-On-Write behavior. Making the timeout window longer means you could potentially have more data at risk in a sudden shutdown scenario, making it shorter has the inverse effect.

It's not the only metric that determines when to close txg's though - there's the dirty_data tunable series as well, which is helpfully elaborated on here:

Some insights into SLOG/ZIL with ZFS on FreeNAS

they (all of a sudden) care more about the "speed" (exactly latency) than about the (small) possible data loss You can work with this. The lack of power loss prevention (PLP) on the drive is (or rather will be) directly related to the poor SLOG performance; most drives are very slow if you...

www.truenas.com

Of course, if you're gunning for "everything is safe" then you're down the sync-write rabbit hole, along with needing some speedy SLOGs to ensure you don't sacrifice performance.

 

[Koop]

It's not so much "don't mess with it" as "understand the function of the knob you're turning" - the txg timeout threshold is the duration after which ZFS will close off your pending transaction group and say "okay, no more updates here, start a new one." Like micro-snapshots, in a sense, driven by the Copy-On-Write behavior. Making the timeout window longer means you could potentially have more data at risk in a sudden shutdown scenario, making it shorter has the inverse effect.

It's not the only metric that determines when to close txg's though - there's the dirty_data tunable series as well, which is helpfully elaborated on here:

Some insights into SLOG/ZIL with ZFS on FreeNAS

they (all of a sudden) care more about the "speed" (exactly latency) than about the (small) possible data loss You can work with this. The lack of power loss prevention (PLP) on the drive is (or rather will be) directly related to the poor SLOG performance; most drives are very slow if you...

www.truenas.com

Of course, if you're gunning for "everything is safe" then you're down the sync-write rabbit hole, along with needing some speedy SLOGs to ensure you don't sacrifice performance.

Right right, makes sense regarding if you hold it in SLOG for too long and your system unexpectedly shuts down you're in a bad situation. Just strikes me as a not-so-good idea to turn it up higher just because you're annoyed your HDD lights show activity, haha. Just as you explain it puts more data at risk which makes sense.

Ahh this post you linked. I've read it before. That's where I specifically remembered reading the every 5 seconds and how I made my initial correlation. It also helps give better context to why you'd modify these values after the re-read, thank you.

For my system I'm using a mirror of 58GB Optane P1600Xs for SLOG. I legitimately did it beccause they were cheap and I figured why not haha. I also have my system hooked up to a UPS at 20% idle load. I planned to connect up my UPS via USB and look into the UPS/NUT Server stuff so I can make sure my system shuts down gracefully during power loss.

I wonder if there's any value for me to make any changes for when I introduce any type of sync-write (non SMB?) workload to my big pool. Something I'll need to ponder on later. Will bookmark your post to refrence and think about later when I want to revisit this rabbit hole haha.

 

[HoneyBadger]

Right right, makes sense regarding if you hold it in SLOG for too long and your system unexpectedly shuts down you're in a bad situation.

If something's in SLOG (or ZIL) it's safe, as that's a non-volatile storage area, either on the Separate LOG device, or internally in your pool as part of the regular ZFS Intent Log. It's the stuff that isn't in the SLOG/ZIL and is just being aggregated into RAM alone that goes poof, and might cause you to have to re-import the pool at the last stable transaction. That txg_timeout value is what helps determine if that might mean "five seconds ago" or "thirty seconds ago" depending on the value set. :)

 

[Koop]

If something's in SLOG (or ZIL) it's safe, as that's a non-volatile storage area, either on the Separate LOG device, or internally in your pool as part of the regular ZFS Intent Log. It's the stuff that isn't in the SLOG/ZIL and is just being aggregated into RAM alone that goes poof, and might cause you to have to re-import the pool at the last stable transaction. That txg_timeout value is what helps determine if that might mean "five seconds ago" or "thirty seconds ago" depending on the value set. :)

Ok duh that makes sense and what I said didn't. Time to re-watch Lawrence as punishment.

So only async writes are only in danger. Makes sense.

Curious side question then- how would Optane DIMMs work in this power loss scenario? Could they save async writes in memory? I'm sure this is actually pretty complicated but I'm going to google into this. Hm.

 

[HoneyBadger]

Optane DC PMEM - as in, "persistent memory" - are SSDs with extra hooks and UEFI support to let them pretend to be RAM under certain circumstances. They're power-failure safe by design, and in their default "storage mode" show up as /dev/pmem devices.

They've been discontinued though, and you'll want to ensure you've got a motherboard that has them explicitly listed on the HCL.

 

[Koop]

Optane DC PMEM - as in, "persistent memory" - are SSDs with extra hooks and UEFI support to let them pretend to be RAM under certain circumstances. They're power-failure safe by design, and in their default "storage mode" show up as /dev/pmem devices.

They've been discontinued though, and you'll want to ensure you've got a motherboard that has them explicitly listed on the HCL.

Yeah I was initially curious about it since you could cram so much memory into your system using them and get to 1TB+ easily. I confirmed with Supermicro that if I were to drop a gen 2 scalable CPU in my motherboard it should work.

But yeah, discontinued- which sucks because it seems pretty awesome. RIP Optane. Long Live Optane. Still they do float around ebay for what seems like cheap prices- especially compared to typical memory of the same capacity. Maybe worth experimenting with one day unless it all dries up.