Drive usage suggestions for 2x Optane, 4x SSD, 5x HDD

[jenesuispasbavard]

So, I'm building a home server mainly accessed over SMB - primarily weekly backups of the home computers, storing phone/camera pictures and videos, and assorted media. Looking to "upgrade" from a directly-attached Stablebit Drivepool to a network-attached TrueNAS.

I have the following drives and am wondering how to best make use of them (and my tentative use cases for each):

Drives	Use case
2x Intel Optane P1600X 118GB NVMe SSDs	ZFS Metadata Special Device?
4x Samsung 860 Pro 256GB 2.5" SSDs	one for boot, two in mirror for SLOG, and one for L2ARC?
5x WD Red Plus 8TB 3.5" HDDs	main data pool in raidz1

Does this sound like a plan? I have 32GB of RAM, enough ports for all 9 SATA drives thanks to an LSI HBA card (still need to stuff them into a Jonsbo N1 though lol), and a 2.5G home network.

Other hardware:

• Motherboard: ASRock Z690M-ITX/AX
• CPU: Intel Core i5-12400
• HBA: LSI 9211-8i 6G SAS HBA
• LAN: Realtek RTL8125BG 2.5G (built-in to MB)

(Also trying to decide between Core for raidz1 write performance and Scale for all the other cool things it can do; but still got more reading to do on that decision, and I know next-to-nothing about containers.)

 

[ChrisRJ]

With the information provided, I would suggest to skip the metadata device (risk, not really needed) and the SLOG (not needed at all).

RAIDZ1 is not suitable for a backup system IMHO and certainly not recommended for 8 TB drives

The RealTek NIC will likely cause issues, you should get an Intel one.

For additional information, please have a look at the "Recommended readings" (blue button) in my signature.

 

[jenesuispasbavard]

Thanks, then for 5x 8TB drives should it be raidz2?

Will sync writes be significantly slower without a SLOG or metadata special device, than with SLOG and/or special?

And I did some reading on the NIC and looks like it should work on both Core and Scale? Choices were a bit limited given the miniITX chassis (Jonsbo N1), and I'm using the single PCIe slot for an LSI HBA card since the motherboard only has 4 SATA ports.

Thanks for the recommended readings! Most of my knowledge so far has been gleaned from Wendell at Level1techs', and Lawrence Systems' Youtube channels.

 

[ChrisRJ]

Thanks, then for 5x 8TB drives should it be raidz2?

RAIDZ2 is what I always go for, unless I needed mirrors for performance. In the latter case I would seriously consider 3-way mirrors. But you should not simply follow me here. I think Tom Lawrence just released a new video on this very subject. Still, I recommend the respective document because it will probably contain additional information.

Will sync writes be significantly slower without a SLOG or metadata special device, than with SLOG and/or special?

From your use-case description sync writes will likely not be in the game. So while an SLOG would improve sync write speed, I see it as irrelevant here. Plus, a RAIDZ (whatever the exact level) would not be suitable for the scenarios where you "need" an SLOG (VMs, databases, etc.).

And I did some reading on the NIC and looks like it should work on both Core and Scale?

Not sure where you found that information, but it is not correct. Yes, some people have been lucky. But quite regularly we have folks here who complain that once they put some load on the system, things went south. There is also a link in my recommended readings on this.

Choices were a bit limited given the miniITX chassis (Jonsbo N1), and I'm using the single PCIe slot for an LSI HBA card since the motherboard only has 4 SATA ports.

Seriously, go for another board. Also, if at all possible avoid Mini-ITX since it limits the choice too much. I know that many find the form factor appealing, and so do I. But in practice it comes with too many limitations and compromises. The case you have chosen is pretty, no doubt.

With this board, what would you do in case of problems with the NIC? You would have to replace it.

Thanks for the recommended readings! Most of my knowledge so far has been gleaned from Wendell at Level1techs', and Lawrence Systems' Youtube channels.

Those are certainly some of the better sources. What I always tell people is to carefully look at what videos are about at their core. Most only cover the building of a "cool NAS" until it starts successfully. But that is not what you want. You want to operate the NAS in real life and put your valuable data onto it. The NAS should take care that those data are not lost, when a SATA cable has a contact problem etc. And that scenario is a lot more demanding on the hardware than the "happy path" most (not all) videos show.

In addition, I recommend you spend time here on the forum and simply read as much as possible. That way you will learn a great deal.

 

[DigitalMinimalist]

I'm using the single PCIe slot for an LSI HBA card since the motherboard only has 4 SATA ports

Use something like this m.2 to SATA adapter to gain more SATA ports

https://de.aliexpress.com/item/1005004358703385.html?pdp_npi=2%40dis%21EUR%21%E2%82%AC%2033%2C96%2122%2C26%20%E2%82%AC%21%21%21%21%21%402101d1b116726494723555085e0a3d%2112000028893920478%21btf&_t=pvid%3Ad8e92a79-29a2-49c8-ad5a-048f63d6d606&afTraceInfo=1005004358703385__pc__pcBridgePPC__xxxxxx__1672649472&spm=a2g0o.ppclist.product.mainProduct&gatewayAdapt=glo2deu

one 256GB SSD for boot, 2x Optane mirrored as ZFS Special Device or VM drive (alternative 2xSSD) and the remaining HDDs into RAIDZ2

Use the PCIe slot for an Intel NIC

 

[Etorix]

Use something like this m.2 to SATA adapter to gain more SATA ports

Are you sure that the SATA controller in that works properly with TrueNAS? A cheap and crappy SATA controller is a direct risk to data. (For all the bad mouth they get, Realtek NICs do not put pools at risk.)

Use the PCIe slot for an Intel NIC

That's a very inefficient use of a x16 slot.
I would rather use a M.2 slot for that and keep the x16 slot for a proper SAS HBA, e.g.

M.2 NVMe to PCIe x4 Extension Cable

M.2 NVMe to PCI express x4 extender adapter jumper for GPU graphics video cards.

www.adt.link

Does this sound like a plan? I have 32GB of RAM, enough ports for all 9 SATA drives thanks to an LSI HBA card (still need to stuff them into a Jonsbo N1 though lol), and a 2.5G home network.

Nice looking case… but still one drive less than a Fractal Design Node 304. 6-wide raidz2 is better than 5-wide.
For information only, a 860 Pro is NOT suitable as a SLOG, one would use an Optane P1600X for that—but you do NOT need a SLOG—, and L2ARC is not to be considered with only 32 GB RAM (make that 64 GB or more).
A pair of 860 Pro may be used for containers/VMs if you add these.
Mini-ITX has its nicety, but the key is to get absolutely the right board: Right NIC (no 2.5 GbE! not even Intel!), and if possible enough SATA ports on board. (Small and cheap NVMe for boot may help saving one SATA port in small home NAS.)

Do you already own the "other hardware"? It's really most suited for a desktop.
If not, and you want to stay with mini-ITX, look for a board with a 1 GbE Intel NIC and at least 6 SATA ports, quite possibly an earlier generation.
Mini-ITX X10SDV boards (soldered Xeon D-1500) are favourites in this application for a small storage NAS, the issue is finding one for a reasonable price.

SuperMicro X10SDV-4C-TLN2F, Mini-ITX,Xeon D-1521. with active fan & I/O shield | eBay

THIS BOARD MUST BE PUT IN 1U OR ITX CASE WITH PROPER COOLING! Form FactorMini ITX. SuperMicro X10SDV-4C-TLN2F, Mini-ITX,Xeon D-1521. with active fan & I/O shield. Other SlotsM.2 PCIe 3.0 x4, M Key 2242/2280.

www.ebay.com

(No endorsement of the seller, but this one comes with the I/O shield…)
6 SATA ports for the pool, server-grade 10 GbE, uses cheap second-hand RDIMM. The PCIe slot may not even be needed—if it's a rev.2 board, it can bifurcate up to x4x4x4x4 for an additional NVMe pool with an Asus Hyper M.2 or similar.

 

[jenesuispasbavard]

Are you sure that the SATA controller in that works properly with TrueNAS? A cheap and crappy SATA controller is a direct risk to data. (For all the bad mouth they get, Realtek NICs do not put pools at risk.)

That's a very inefficient use of a x16 slot.
I would rather use a M.2 slot for that and keep the x16 slot for a proper SAS HBA, e.g.

M.2 NVMe to PCIe x4 Extension Cable

M.2 NVMe to PCI express x4 extender adapter jumper for GPU graphics video cards.

www.adt.link

Thanks for the advice! I did read from some recommendations in this forum that a proper SAS HBA would be more reliable than an M.2 to SATA adapter, so went the HBA route. Maybe I can use the M.2 to PCIe riser you linked for 10GbE.

For information only, a 860 Pro is NOT suitable as a SLOG, one would use an Optane P1600X for that—but you do NOT need a SLOG—, and L2ARC is not to be considered with only 32 GB RAM (make that 64 GB or more).
A pair of 860 Pro may be used for containers/VMs if you add these.

RAIDZ2 is what I always go for, unless I needed mirrors for performance. In the latter case I would seriously consider 3-way mirrors. But you should not simply follow me here. I think Tom Lawrence just released a new video on this very subject. Still, I recommend the respective document because it will probably contain additional information.

From your use-case description sync writes will likely not be in the game. So while an SLOG would improve sync write speed, I see it as irrelevant here. Plus, a RAIDZ (whatever the exact level) would not be suitable for the scenarios where you "need" an SLOG (VMs, databases, etc.).

Thanks. I had considered some drives for SLOG only because I thought sync=always would be safer than default, but the vast majority of my writes to the pool will be file copies or backups which can simply be restarted if they fail. Maybe I don't need sync writes after all... and therefore not need for a SLOG.

Nice looking case… but still one drive less than a Fractal Design Node 304. 6-wide raidz2 is better than 5-wide.

Mini-ITX has its nicety, but the key is to get absolutely the right board: Right NIC (no 2.5 GbE! not even Intel!), and if possible enough SATA ports on board. (Small and cheap NVMe for boot may help saving one SATA port in small home NAS.)

Do you already own the "other hardware"? It's really most suited for a desktop.
If not, and you want to stay with mini-ITX, look for a board with a 1 GbE Intel NIC and at least 6 SATA ports, quite possibly an earlier generation.
Mini-ITX X10SDV boards (soldered Xeon D-1500) are favourites in this application for a small storage NAS, the issue is finding one for a reasonable price.

SuperMicro X10SDV-4C-TLN2F, Mini-ITX,Xeon D-1521. with active fan & I/O shield | eBay

THIS BOARD MUST BE PUT IN 1U OR ITX CASE WITH PROPER COOLING! Form FactorMini ITX. SuperMicro X10SDV-4C-TLN2F, Mini-ITX,Xeon D-1521. with active fan & I/O shield. Other SlotsM.2 PCIe 3.0 x4, M Key 2242/2280.

www.ebay.com

(No endorsement of the seller, but this one comes with the I/O shield…)
6 SATA ports for the pool, server-grade 10 GbE, uses cheap second-hand RDIMM. The PCIe slot may not even be needed—if it's a rev.2 board, it can bifurcate up to x4x4x4x4 for an additional NVMe pool with an Asus Hyper M.2 or similar.

I do already own the "other hardware" - I moved my main desktop to 13th-gen Intel + ATX + DDR5 (mainly to accommodate the comically large RTX 4090 GPU), so I have these parts from my previous ITX system. I chose the Jonsbo N1 case since I already had ITX parts, and the Fractal Node 304 is considerably larger. But maybe if I move to 6+ drives in the future I can move everything over to a larger case.

Not sure where you found that information, but it is not correct. Yes, some people have been lucky. But quite regularly we have folks here who complain that once they put some load on the system, things went south. There is also a link in my recommended readings on this.

The "Core" and "Scale" words in my previous message were links to these threads, sorry:

2.5Gb NIC not detecting in 12.0-U4 RTL8125

Hi all. New to the community and not very experienced with Truenas but I do have a working system and appreciate any help I can get on the following matter. So I bought myself one of these Trendnet 2.5g nics which supposedly uses the Realtek RTL8125 chipset according to servethehome.com NIC ->...

www.truenas.com

Realtek RTL8125 2.5Gbps Ethernet

The 8125 Realtek chipset is now officially. supported as of 12u4 but apparently needs two tunables set to use the newly-adopted Realtek drives. See here Just to bump this and confirm, the 2.5 8125 nic IS supporting in the latest version?

www.truenas.com

I'll test out sustained 2.5G reads/writes and if things go south I can always switch over to the Intel 1GbE on the same motherboard until I can figure out 10GbE via M.2 or PCIe.

Seriously, go for another board. Also, if at all possible avoid Mini-ITX since it limits the choice too much. I know that many find the form factor appealing, and so do I. But in practice it comes with too many limitations and compromises. The case you have chosen is pretty, no doubt.

With this board, what would you do in case of problems with the NIC? You would have to replace it.

Thanks, with only five 3.5" drives I had hoped mini-ITX would be enough, but as you said there are other compromises too - only one PCIe slot, only four SATA ports etc. Unfortunately I already have the hardware from a previous system, but if things really don't work out I'll probably switch to microATX.

Those are certainly some of the better sources. What I always tell people is to carefully look at what videos are about at their core. Most only cover the building of a "cool NAS" until it starts successfully. But that is not what you want. You want to operate the NAS in real life and put your valuable data onto it. The NAS should take care that those data are not lost, when a SATA cable has a contact problem etc. And that scenario is a lot more demanding on the hardware than the "happy path" most (not all) videos show.

Ha, yes. The videos are great information, assuming everything works as expected.

In addition, I recommend you spend time here on the forum and simply read as much as possible. That way you will learn a great deal.

 

[ChrisRJ]

I had considered some drives for SLOG only because I thought sync=always would be safer than default, but the vast majority of my writes to the pool will be file copies or backups which can simply be restarted if they fail. Maybe I don't need sync writes after all... and therefore not need for a SLOG.

In principle sync writes are indeed safer. But that comes with a performance penalty (without SLOG usually a considerable one, at least with HDDs). To overcome that, a proper(!) SLOG setup can help. But the emphasis is on proper here. No consumer-grade SSD is able to do that. So you would have to end up with specific enterprise-grade SSDs and need at least a pair for a mirror. Plus a UPS and possibly redundant power supplies, plus ...

You go down a rabbit hole, sort of. An SLOG that is not suitable will be worse than async writes, and even the best SLOG is by definition slower than async writes. So overall you will likely be better off with a standard setup.

 

[Etorix]

Thanks for the advice! I did read from some recommendations in this forum that a proper SAS HBA would be more reliable than an M.2 to SATA adapter, so went the HBA route. Maybe I can use the M.2 to PCIe riser you linked for 10GbE.

If going this route, study the ADT-Link lineup (R41, R42) and carefully check the geometry and cable length that fits your case (too short is bad, but too long is inconvenient, especially in a tight case, and the cable should not be folded or bent too sharply).

Thanks, with only five 3.5" drives I had hoped mini-ITX would be enough, but as you said there are other compromises too - only one PCIe slot, only four SATA ports etc. Unfortunately I already have the hardware from a previous system, but if things really don't work out I'll probably switch to microATX.

I feared you already had the hardware… Then you need to carefully review your options, what you most want to keep, and at which point you'd be better off selling the hardware to buy something more appropriate rather than piling up adapters to fit a square peg in a round hole.

 

[jenesuispasbavard]

If going this route, study the ADT-Link lineup (R41, R42) and carefully check the geometry and cable length that fits your case (too short is bad, but too long is inconvenient, especially in a tight case, and the cable should not be folded or bent too sharply).


I feared you already had the hardware… Then you need to carefully review your options, what you most want to keep, and at which point you'd be better off selling the hardware to buy something more appropriate rather than piling up adapters to fit a square peg in a round hole.

Thanks. Yeah at some point even the cost of various adapters / PCIe NIC would be more than just selling my CPU+MB+RAM and going with something like the SuperMicro X10SDV...

 

[Davvo]

Watch out to not pack too much in a small case since 5 HDDs + CPU + HBA + 10G NIC require proper cooling.

 

[jenesuispasbavard]

Do you already own the "other hardware"? It's really most suited for a desktop.
If not, and you want to stay with mini-ITX, look for a board with a 1 GbE Intel NIC and at least 6 SATA ports, quite possibly an earlier generation.
Mini-ITX X10SDV boards (soldered Xeon D-1500) are favourites in this application for a small storage NAS, the issue is finding one for a reasonable price.

SuperMicro X10SDV-4C-TLN2F, Mini-ITX,Xeon D-1521. with active fan & I/O shield | eBay

THIS BOARD MUST BE PUT IN 1U OR ITX CASE WITH PROPER COOLING! Form FactorMini ITX. SuperMicro X10SDV-4C-TLN2F, Mini-ITX,Xeon D-1521. with active fan & I/O shield. Other SlotsM.2 PCIe 3.0 x4, M Key 2242/2280.

www.ebay.com

(No endorsement of the seller, but this one comes with the I/O shield…)
6 SATA ports for the pool, server-grade 10 GbE, uses cheap second-hand RDIMM. The PCIe slot may not even be needed—if it's a rev.2 board, it can bifurcate up to x4x4x4x4 for an additional NVMe pool with an Asus Hyper M.2 or similar.

Ok, already running into the limits of this non-server miniITX setup...

2.5Gbps is a bit slower than I thought it would be for this setup, and I'm frequently running into the ~250MB/s speed limit.

However, switching to 10Gbps would require an additional NIC on this Z690 motherboard, and things are already pretty cramped with all the stuff I've jammed in there. I could conceivably replace one of the Intel Optane P1600x M.2 drives with an M.2 to PCIe cable and then an Intel X540 PCIe 10GbE NIC.

Or just switch to the SuperMicro X10SDV...

My only concern is the performance of the Xeon D-1521, compared to my current Core i5-12400. It's six years older, and the 12400 is some 2.5x as fast in single-threaded and 3x as fast in multi-threaded applications.

Will it be noticeably slower in writing to a raidz2 than the 12400, now that network speed won't be the hard limit? The 12400 already saturates a single core when writing large amounts of data to the pool.

 

[Etorix]

It will depend on the use case.
SMB is single threaded and will favour high clocks.
Xeon-D can take more RAM than Core, and DDR4-2400 RDIMM is cheap. ZFS loves RAM, and will give it back as speed.

Writing to a raidz2 should be limited by the pool in either case.

 

[jenesuispasbavard]

It will depend on the use case.
SMB is single threaded and will favour high clocks.
Xeon-D can take more RAM than Core, and DDR4-2400 RDIMM is cheap. ZFS loves RAM, and will give it back as speed.

Writing to a raidz2 should be limited by the pool in either case.

Thanks for the distinction between SMB and raidz2 CPU usage; in that case it was definitely the SMB that was causing the high single thread usage.

I'll give it some thought, thanks. If I can't also jam a 40-60mm fan to cool the SAS HBA and 10GB NIC into the existing case maybe I should just switch to the motherboard that was actually designed for this use case lol.