First build with a few open questions

[cloudycelt]

Hello all, I've been reading up on FreeNAS for some time and I'm finally at the stage where I'm ready to commit to buying and building a system. I've read up on some threads regarding particular initial areas of interest so I hope my questions come out making sense.

Firstly, my use-case:
I currently have no NAS in my home but I want to start to self-host a number of things such as the usual photos, media, movies, music, but also run a number of services that will need some backing persistent storage such as some type of git, NextCloud, Bitwarden RS, Kubernetes clusters and more, so databases are certainly in the equation.

Some requirements I made for myself:
- Physical space usage: I live in an apartment, so I don't have a basement to stash my "noisy equipment" in, so I've iterated through a few designs and figured that buying a cabinet (9U) and stashing it in a particular spot is my best bet, as that will allow me to rackmount most things for this home-lab-in-progress and keep the space usage confined to that cabinet including UPS and so on
- Noise: For the same reason as above, I've been selecting certain things to help with reducing noise, such as quieter fans and a taller chassis (2U vs 1U)
- Growth: I want to have room to expand the available storage in future without having to replace all the disks I am soon to buy

This is what I have come up with so far, I'm providing Amazon links where possible as requested in Josh's sticky post as well as a comment for each component.

Component​	Part Name & Link​	Comment​
Cabinet	Tripp Lite 9U Wall Mount Rack Enclosure Server Cabinet	Chosen as it has a 20.5" mountable depth, so it will fit the server chassis and comes already assembled for convenience
Chassis	SilverStone Technology 2U Rackmount RM21-308	Chosen as it is a relatively short depth chassis but has the nice hot-swap bays and 3 80mm fans - many others I found only had 1 or used the squealer 40mm fans
Motherboard	Supermicro X11SCH-F Micro-ATX	Chosen as it is Micro-ATX fitting in the chassis, friends highly recommend Supermicro, has 8 SATA3 ports and two M.2 slots, IPMI and uses the more recent Intel C246 chipset
CPU	Intel Core i3-9100 Coffee Lake 4-Core 3.6GHz	Chosen partly due to the hardware recommendations guide plus this being the latest gen the board supports and the price seeming alright
RAM	Crucial 16GB DDR4-2666 ECC Unbuffered x2	Chosen as I wanted ECC UDIMMs and at least 32GB to start with, the board supports max 128GB in 4 slots but I couldn't find a 32GB DIMM of this sadly. Also the CPU actually only supports DDR4-2400 not 2666 but I think that just means the RAM will run at 2400, right? I didn't see any 2400 frequency for a similar price so went with this.
PSU	Seasonic FOCUS PX-650 650W Platinum Modular	Chosen to have plenty of available wattage for when the system expands, as well as its physical dimensions (less than 160mm) for the chassis. Additionally my understanding is it will run quieter since it will be further from any kind of high load scenario.
HDD Cable	Silverstone 36 Pin Minisas SFF-8087 to SATA cable	Chosen as it is apparently needed by this chassis to connect the backplane to the SATA ports on the motherboard
Rail	Silverstone 20"" Ball Bearing Sliding Rail Kit RMS04-20	Chosen as it should allow me to slide out the chassis from the cabinet rack if desired, which just seems like a nice-to-have.
Chassis fan replacements	Noctua NF-A8 PWM 4-Pin 80mm x3	Chosen as these came *highly* recommended by friends to reduce noise
CPU fan replacement	Noctua NH-L9i Low-profile CPU Cooler for LGA115x	Chosen for the same reason as above.
Data disks	Western Digital Red NAS 10TB 5400rpm x 4	Chosen from the WD site as the Amazon page is full of reviewers saying they purchased from Amazon and WD refused to honour the full warranty saying the drives had been sold to Amazon years ago, so it seems best to order new drives direct from the manufacturer, is that correct? I chose 10TB each but wondering how much louder it would be to use the Pro 7200RPM drives and get them at a smaller capacity due to price.
PDU	CyberPower CPS1215RM Basic PDU, 120V/15A	Chosen as it'll fit neatly into the cabinet rack and doesn't also have surge protection (which the UPS will provide)
UPS	CyberPower OR1500LCDRM1U Smart App LCD UPS	Chosen as it seems to be a nice rackmount design with plenty of sufficient power. I saw only the 3 major brands APC, CyberPower and Tripp Lite around for UPS function but many of the reviews on the Tripp Lite of the same size says it makes noise all the time even when just recharging and I certainly don't want that.

I have yet to properly choose a boot disk as I have some questions/observations around utilisation of the remaining slots.

- I would ideally like to keep the remaining 4 SATA ports free to add more data disks down the road.
- I've heard it recommended that an SSD is preferable as a boot volume rather than a USB device and have looked over https://www.servethehome.com/buyers...as-nas-servers/top-picks-freenas-boot-drives/
- I've read https://www.ixsystems.com/blog/zfs-zil-and-slog-demystified/ and it sounds like I should at the very least have an SSD for use as a SLOG to make things feel speedy, which I could provide via an M.2 form factor SSD, my understanding is on a 1GbE network I can only write 625MB every 5 seconds so my SLOG does not need to be larger than that unless I use link aggregation or go to 10GbE, I see from https://www.servethehome.com/buyers...as-servers/top-picks-freenas-zil-slog-drives/ that ideally I should have a SLOG device with Power Loss Protection but that feature appears to be reserved for datacenter versions of SSDs that cost $400+ which is a little prohibitive as well as them being much larger in size than I'd actually need which feels like a waste.
- I've read https://www.ixsystems.com/community/threads/how-large-l2arc-for-64gb.45021/ which indicates I shouldn't have an L2ARC unless I have at least 64GB of RAM, which I won't initially but could down the road, so it seems preferable to leave room to add one
- The motherboard's two M.2 slots are on the PCIe 3.0 x4 bus, there is no slot for that
- The motherboard has a PCIe x 8 and PCIe x 16 slot so I could conceivably add an expansion card such as https://www.amazon.com/Mailiya-M-2-PCIe-3-0-Adapter/dp/B01N78XZCH/ to give myself another M.2 slot

So the question from these observations comes down to - what arrangement should I go for, given that I think I want an SSD boot device, a SLOG plus room for a future L2ARC?

1. Add an expansion card in the x8 slot to get another M.2 slot then use an M.2 each for boot, L2ARC, SLOG? How would they be arranged given the two existing M.2 slots share the PCIe x4 bus
2. Add an expansion card for the boot and then use the two M.2 slots in a pair for both L2ARC and SLOG as in https://clinta.github.io/freenas-multipurpose-ssd/ ? Feels like I would gain some redundancy with the pair but not against power loss which the UPS could help with, plus would putting both on the same disk impact performance, or would it not matter since the two slots are on the same bus anyway?
3. Some other arrangement? It seems like there's quite a few ways to arrange this

Thoughts on the overall build as well as the specific questions welcome! I know it maybe seems like overkill for an initial build but as mentioned I'm looking to host a decent number of things as well as running this for quite a few years rather than outgrowing it quickly.

Thanks in advance!

 

[cloudycelt]

Bump!

 

[artlessknave]

note that your UPS link seems to go to the PDU instead.

SLOG power loss protection isn't really an option. SLOG with no power loss protection can risk the whole pool, and definitely risks whatever is still in transit. you mention a database, afaik databases are typically extremely sensitive to lost writes, as are VM's. SLOG also only applies to synchronous writes; unless you are doing sync writes the SLOG will do nothing since zfs will use RAM.

not all USB drives are created equal, most are crap but there are a few gems, mainly from sandisk

I'm not sure about your SAS cables, usually you want reverse breakout, but I do notice that with these sas the sff side is the target. I have never seen that before, it seems weird that it has a sideband cable, afaik that only applies to SAS controllers to begin with.

it looks like that chassis has a slim DVD slot, you could get a slimHDD adapter and possibly boot from that

also note that "bumping" you post may have the opposite effect, as some of us specifically look at the unanswered posts list, and that might count as an "Answer", thus reducing its visibility
good post though, clearly a ton of research went into this.

 

[cloudycelt]

Thanks for the reply!

Oh whoops...I don't see a button to edit the UPS link but it is here: https://www.amazon.com/CyberPower-OR700LCDRM1U-System-Outlets-Rackmount/dp/B0016P7HJA/

On the SLOG: Given that there would be a UPS offering at least some power loss protection plus configuring the device to shut down cleanly in the event of a prolonged interruption, I would have thought that would help some at least? I haven't come across any power loss protected SSD that wasn't ~$600 but happy to be linked to something. It sounds like you're recommending not to bother with the SLOG at all if it doesn't have PLP though.

For the USB - yeah I'm fine with not using a USB device for boot.

On the SAS cable - this was just the exact recommended cable for the chassis backplane to connect to SATA disks - it's listed at https://www.silverstonetek.com/product.php?pid=843&area=en for reference.

That's a fair point on the slim DVD slot, I'll check into that!

As for the bump - fair point, though I'd not seen a reply since I posted a week ago so figured it couldn't hurt to try! :)

 

[IQless]

On the SLOG: Given that there would be a UPS offering at least some power loss protection plus configuring the device to shut down cleanly in the event of a prolonged interruption, I would have thought that would help some at least? I haven't come across any power loss protected SSD that wasn't ~$600 but happy to be linked to something. It sounds like you're recommending not to bother with the SLOG at all if it doesn't have PLP though.

The UPS might not save you, let's say that your PSU dies, the UPS won't save you. Having a proper SLOG with PLP would still hold the transactions in transit. There is a lot of things that can go wrong, just something to take into account when planning out your risk acceptance :)

 

[artlessknave]

There is a lot of things that can go wrong,

exactly this.

On the SAS cable

ya, I've never seen it described that way, but it looks like it should be correct

posted a week ago s

I looked at the timestamps after than and realized it had been awhile, so it was within "bump" territory

For the USB - yeah I'm fine with not using a USB device for boot.

my point here wasnt that you shouldnt use USB, just that you have to put a bit of work into getting the correct USB's. the main problem with USB sticks is that most of them have zero wear leveling, and those die REALLY fast with zfs. the sandisck cruzer fits are known to not have this problem, while also being very small.

 

[cloudycelt]

The UPS might not save you, let's say that your PSU dies, the UPS won't save you. Having a proper SLOG with PLP would still hold the transactions in transit. There is a lot of things that can go wrong, just something to take into account when planning out your risk acceptance :)

That's a fair point, would it be better or worse then to skip having a SLOG device at all if you don't have one with PLP? Without one things are just held in RAM anyways so they'd still be lost in a power event it seems.

I looked at the timestamps after than and realized it had been awhile, so it was within "bump" territory

No worries!

my point here wasnt that you shouldn't use USB, just that you have to put a bit of work into getting the correct USB's. the main problem with USB sticks is that most of them have zero wear leveling, and those die REALLY fast with zfs. the sandisck cruzer fits are known to not have this problem, while also being very small.

Ah yeah fair, I guess I just figured given a lot of them suck it'd be easier to just not bother with USB at all if possible, plus I'd imagine an SSD might perform faster than a USB device in many cases?

 

[artlessknave]

That's a fair point, would it be better or worse then to skip having a SLOG device at all if you don't have one with PLP? Without one things are just held in RAM anyways so they'd still be lost in a power event it seems.

the difference is that SLOG GUARANTEES to the writing application that the data is, for sure, written to non volatile storage. if it is then lost, the application doing the writing can be very broken because it thinks the data was already reliably stored. if you are using a SLOG device that cannot guarantee that all written data is not lose, they you are basically tricking the application creating the data into thinking the data is being written with high reliability when it actually isnt. you would be better off just letting the data die the normal way than lying to your applications.

this is similar to how you don't want to run zfs on raid cards or VM disks or other abstractions, they lie to zfs and data gets lost.

zfs already handles this with the ZIL located within the existing pool. what a SLOG does is move the ZIL from the within the pool to dedicated storage. if that, now external, ZIL, known as SLOG, fails, and loses data, and that data was critical to the pool functioning at all, the pool may become impossible to import.

almost every SSD is more reliable than USB drives, some USB SSD's are on the same level, but if you can fit SSD's and have the ports to connect them, there is no reason to use USB over SSD. the primary USB advantages are that they are small, dont require large storage bays in a case, and dont use your SATA/SAS controllers.

 

[cloudycelt]

the difference is that SLOG GUARANTEES to the writing application that the data is, for sure, written to non volatile storage. if it is then lost, the application doing the writing can be very broken because it thinks the data was already reliably stored. if you are using a SLOG device that cannot guarantee that all written data is not lose, they you are basically tricking the application creating the data into thinking the data is being written with high reliability when it actually isnt. you would be better off just letting the data die the normal way than lying to your applications.

Totally fair point and thank you for the clarification - so it sounds like yes, if I can't get a PLP SLOG, I shouldn't bother with a SLOG at all as there's a risk of this kind of incomplete-but-not-known-to-be-incomplete write, whereas without a SLOG the write would at least never be acknowledged as having completed successfully if it had not actually finished.

this is similar to how you don't want to run zfs on raid cards or VM disks or other abstractions, they lie to zfs and data gets lost.

Ah yes, I'm quite familiar with VM level abstractions (VMware vSphere background) so I can think of various features that could cause issues there if you care about your data integrity.

zfs already handles this with the ZIL located within the existing pool. what a SLOG does is move the ZIL from the within the pool to dedicated storage. if that, now external, ZIL, known as SLOG, fails, and loses data, and that data was critical to the pool functioning at all, the pool may become impossible to import.

I appreciate the clarification! Some of these was in the ZIL/SLOG blogpost but it certainly helps to have it explained in other words too.

almost every SSD is more reliable than USB drives, some USB SSD's are on the same level, but if you can fit SSD's and have the ports to connect them, there is no reason to use USB over SSD. the primary USB advantages are that they are small, don't require large storage bays in a case, and don't use your SATA/SAS controllers.

Fair points. If I can I'll try to use an SSD then, assuming I can re-use the slim DVD lot maybe. As mentioned in my first post I'd prefer not to use up one of my 8 SATA ports when the chassis has 8 nice hot swap bays I could use :)

Thank you! Happy to take any other comments you have on the build but it sounds like I got a decent number of things right at least :D

 

[artlessknave]

I will, though, reiterate that ZIL/SLOG writes also do only apply to sync writes, async writes skip it entirely, using the RAM write queue anyway.
also, a next version of zfs is supposed to be adding hybrid storage, which should allow layering of different speeds intelligently, though it's exact implementation remains to be seen. my understanding is that would allow true write caching, where you could have an SSD that takes all new writes and then stuff gets archived to the spindles on new writes.

 

[cloudycelt]

Good to know, and very interesting on the upcoming possible improvements :)

 

[IQless]

I haven't come across any power loss protected SSD that wasn't ~$600 but happy to be linked to something

Have you considered some used gear eg. eBay? You might find some slightly used enterprise SSD's with PLP they can be a lot cheaper than 600$ :)
Slog Benchmark discussion, this can point you in the right direction :)

 

[cloudycelt]

Have you considered some used gear eg. eBay? You might find some slightly used enterprise SSD's with PLP they can be a lot cheaper than 600$ :)
Slog Benchmark discussion, this can point you in the right direction :)

That's a fair point - though I guess warranty is questionable on a used device.

I did find this Samsung 983 DCT Series SSD 960GB - M.2 NVMe Interface Internal Solid State Drive with V-NAND Technology for Business (MZ-1LB960NE) for $226.67 though, might be worth it.

I guess the annoying part is that it's way oversize since a 1GbE would only need 625MB and even over-speccing for 10GbE means I'd only need 6.25GB, but there doesn't seem to be a lot of smaller size disks around...unless I split the function of the disk but that would surely impact performance no?

 

[artlessknave]

oversize is fine since it means you get more space for the wear leveling, and thus a longer lasting SLOG. they wear out fast since ALL they do is write data

 

[cloudycelt]

oversize is fine since it means you get more space for the wear leveling, and thus a longer lasting SLOG. they wear out fast since ALL they do is write data

ahhh okay that makes sense, thank you! This Samsung might be a nice choice then.

I guess I should skip the L2ARC for now, but leave room to add one later if/when I bump the RAM to 64GB or more, maybe via a PCIe card as mentioned above.

So then I could use one M.2 for boot and one M.2 for SLOG. I think they share the same PCIe x4 bus, but I'd imagine the load on the boot volume wouldn't be terribly high so that should be alright?

 

[artlessknave]

honestly, i dont see l2arc as all that useful, its slower than arc, takes up ram that could be USED for arc, and adds complexity.
newer drives have pretty good read speeds, its the writes that are slow, and zfs is designed to intelligently cache and speed all of that up.
SLOG is kind of similar, most of the times to add these 2 functions is later when you have data telling you you need them

 

[cloudycelt]

honestly, i don't see l2arc as all that useful, its slower than arc, takes up ram that could be USED for arc, and adds complexity.
newer drives have pretty good read speeds, its the writes that are slow, and zfs is designed to intelligently cache and speed all of that up.
SLOG is kind of similar, most of the times to add these 2 functions is later when you have data telling you you need them

So you'd say the read speed from a volume backed by say 4x 5400RPM WD Reds would be sufficient rather than overcomplicating with an L2ARC? Interesting!

 

[artlessknave]

the read speeds for anything on the L2ARC will be faster than reading directly from the spindles, but will naturally only apply to data already on the L2ARC, that isnt already in the ARC, so L2ARC only improves speeds for data read at least more than once. if you have a huge array that only accesses the same data regularly, and that data ALL fits on your L2ARC, but doesn't fit in the ARC, then yes, you would see a huge benefit. however, chances are if you could fit the data on an ssd, you would just use an ssd. so, again, it depends on your application, and the best way to see if you would benefit is to see your performance without the L2ARC first.
it's called L2ARC for a reason, it's inferior to ARC in everything but mass storage space.

 

[cloudycelt]

the read speeds for anything on the L2ARC will be faster than reading directly from the spindles, but will naturally only apply to data already on the L2ARC, that isnt already in the ARC, so L2ARC only improves speeds for data read at least more than once. if you have a huge array that only accesses the same data regularly, and that data ALL fits on your L2ARC, but doesn't fit in the ARC, then yes, you would see a huge benefit. however, chances are if you could fit the data on an ssd, you would just use an ssd. so, again, it depends on your application, and the best way to see if you would benefit is to see your performance without the L2ARC first.
it's called L2ARC for a reason, it's inferior to ARC in everything but mass storage space.

Makes sense to me, thank you very much for the explanation! I think I'm almost set to get everything.

Do you happen to have any insight on the query from above? Regarding using the two M.2 slots for SLOG & boot respectively, though they both share the PCIe x4 bus? Not sure if/how much performance of either might be impacted by each other...

 

[artlessknave]

not offhand, no. I would have to rtfm. except I'm not buying it so I don't wanna make my eyes bleed to read it for you. supermicro is usually pretty good for putting a block diagram that shows the exact layout electrically, which should answer that question.
boot will only be used on....boot, and small changes to config (basically a bit of text), and will have functionally zero bandwidth usage.