Pool Layout for SSDs and HDDs - Separated zpools

[Shigure]

Hey guys, I'm new but finially joined the TrueNAS family after a long time delay due to several different reasons.

Here's my hardware list:
Ryzen 7 Pro 5750G
ASRock B550 Phantom Gaming itx/ax
Micron MTA18ADF2G72AZ3G2R1 x2(16G x2 3200 unbuffered ECC)
LSI 9208-8i
Mellanox MCX311A-XCAT ConnectX-3 EN(connected with M.2 to PCIe x4 raiser)
Fractal Design Node 304
EVGA 750W SFX full modular PSU
TureNAS will be installed on a 256G SSD connected via USB

For the drives, I plan to use one 2TB M.2 NVMe SSD laying around as L2ARC, 4x 1TB SATA SSD as "RAID10" and 3x EXOS X18 16TB as RAIDZ1.
Not sure what's the best practise for my case but I guess I shouldn't put SSDs and HDDs into a same zpool? The plan is to create two different zpools, one for the SSDs and another one for the HDDs. For the SSD zpool the layout should be two mirror vdevs with 2 SSDs in each vdev, which should be the ZFS equivalent of RAID10? (a screenshot is attached)

But how about the wear across the 4 SSDs, or will the wear spread evenly among drives/vdevs? I ask this because 2 of the 4 drives are used with about 1TB write, and I will mix one used drive and one brand new into each mirror vdev.

For the HDD zpool, it should be easier for now as I only have 3 drives now, 1 RAIDZ1 vdev is enough and if necessary I can add another 3 drives as a second vdev to expand that zpool.

Is my understanding correct? Or anything I can improve?

 

[ChrisRJ]

L2ARC will likely not make much sense with your amount of memory. You will find a number of threads on this in the forum.

16 TB HDDs are pretty big for RAIDZ1, so I would recommend either a 3-way mirror or purchasing an additional drive for RAIDZ2

 

[Etorix]

Welcome!

TureNAS will be installed on a 256G SSD connected via USB. Just lug the SSD

With 8 ports from the HBA and 4 on the motherboard there's no use for a USB adapter: Just plug the SSD directly.

For the drives, I plan to use one 2TB M.2 NVMe SSD laying around as L2ARC, 4x 1TB SATA SSD as "RAID10"

Thanks for the quotes. The proper ZFS terminology is "stripe of mirrors".

and 3x EXOS X18 16TB as RAIDZ1.

Unsafe!

Not sure what's the best practise for my case but I guess I shouldn't put SSDs and HDDs into a same zpool?

To advise on this, we'd need to know the use case for the server.

But how about the wear across the 4 SSDs, or will the wear spread evenly among drives/vdevs?

It should. 1 TB is not much wear anyway.

For the HDD zpool, it should be easier for now as I only have 3 drives now, 1 RAIDZ1 vdev is enough and if necessary I can add another 3 drives as a second vdev to expand that zpool.

In theory yes. But, as noted above, raidz1 is not safe with large drives; you'd need at least raidz2, and thus at least 4-wide, to keep your data safe in the event of a single drive failure: Better go as wide as you'll ever need right from the start.
The Node 304 only has six 3.5" bays, so you wouldn't have fit 6 HDD and 4 SSD anyway.

 

[Shigure]

L2ARC will likely not make much sense with your amount of memory. You will find a number of threads on this in the forum.

16 TB HDDs are pretty big for RAIDZ1, so I would recommend either a 3-way mirror or purchasing an additional drive for RAIDZ2

I read it as my RAM was not enough to utilize L2ARC, is that correct?
It's actually the first time I know large drives are not safe for RAIDZ1. From what I can think of, with large drives there might be a higher chance for a second drive failure during rebuild and result in data lost?

 

[Shigure]

Welcome!


With 8 ports from the HBA and 4 on the motherboard there's no use for a USB adapter: Just plug the SSD directly.


Thanks for the quotes. The proper ZFS terminology is "stripe of mirrors".

Unsafe!


To advise on this, we'd need to know the use case for the server.


It should. 1 TB is not much wear anyway.


In theory yes. But, as noted above, raidz1 is not safe with large drives; you'd need at least raidz2, and thus at least 4-wide, to keep your data safe in the event of a single drive failure: Better go as wide as you'll ever need right from the start.
The Node 304 only has six 3.5" bays, so you wouldn't have fit 6 HDD and 4 SSD anyway.

I found I didn't include enough context/details in original post lol.
I choose to use a USB adapter given two reason: the boot SSD is actually NVMe(the MB has two M.2 slot, the main one is planned for L2ARC and the back one is used for the raiser card), and it's easier to swap it if it is connected via USB. I know usb adapter w/ SSD is not really recommended by the community but did see a lot of people are using it?
This NAS will mainly serve as a media/downloading server and some backup of my phone/main PC. Therefore the pure SSD zpool will be the stagining/application pool to have newly donwloaded/frequently accessed files and jails running on it. The HDD zpool will be the storage pool where most of the file archive/back goes to. Does it make sense?
Well I did some research and it seems large drives are not safe for RAIDZ1 due to another failure might occur during rebuild so hot spare will not help as well(just thinking since I saw RAID5 with 1 hot spare before). Based on the advices I should go for 4x 16TB RAIDZ2. As for mounting, I actually 3D printed stack mount for the 2.5 SSDs so in each drive bay of the Node 304 I can mount at least two of them, it will be perfect if I have 4x HDDs and the two left used by 4x SSDs then...

 

[jgreco]

I read it as my RAM was not enough to utilize L2ARC, is that correct?

Not really. L2ARC can work with very small amounts of RAM, even 16GB, but there are two basic problems. One, the amount of L2ARC a system can support is usually around 5x the RAM, so 64GB-96GB would be the practical max for a 16GB system, and two, the less RAM you have, the harder it is for the system to identify good candidates for eviction to the L2ARC, meaning that the L2ARC's effectiveness is hobbled when you don't have enough RAM. In practice, L2ARC with less than 64GB of ARC seems to have a tough time working well, and we've suggested 64GB minimum to support L2ARC for a number of years now.

 

[Etorix]

I choose to use a USB adapter given two reason: the boot SSD is actually NVMe(the MB has two M.2 slot, the main one is planned for L2ARC and the back one is used for the raiser card), and it's easier to swap it if it is connected via USB. I know usb adapter w/ SSD is not really recommended by the community but did see a lot of people are using it?

This is generally done to save a SATA port for data drives.
In your case, with apps and "only" 32 GB, it's probably best to forget about a L2ARC and just put the boot drive in the free M.2 slot.

This NAS will mainly serve as a media/downloading server and some backup of my phone/main PC. Therefore the pure SSD zpool will be the stagining/application pool to have newly donwloaded/frequently accessed files and jails running on it. The HDD zpool will be the storage pool where most of the file archive/back goes to. Does it make sense?

Full sense.

An alternative layout would be to use the 2 TB NVMe as (single drive, not redundant) application pool and keep all six bays for 3.5" HDDs. With frequent backups (replication) to the HDD pool it should be possible to restore to a new drive if the single drive vdev fails.

Well I did some research and it seems large drives are not safe for RAIDZ1 due to another failure might occur during rebuild so hot spare will not help as well(just thinking since I saw RAID5 with 1 hot spare before). Based on the advices I should go for 4x 16TB RAIDZ2. As for mounting, I actually 3D printed stack mount for the 2.5 SSDs so in each drive bay of the Node 304 I can mount at least two of them, it will be perfect if I have 4x HDDs and the two left used by 4x SSDs then...

I suspected something like this for 2.5" drives.
And you did a good search on raid5/raidz1 issues.

 

[NugentS]

The only thoughts I have that haven't been mentioned is heat. Cramming that many drive in (I am not familiar with the case) - just make sure its well ventilated as both the drives and LSI Card need cooling (and so I suspect does the mellanox)

 

[Shigure]

Not really. L2ARC can work with very small amounts of RAM, even 16GB, but there are two basic problems. One, the amount of L2ARC a system can support is usually around 5x the RAM, so 64GB-96GB would be the practical max for a 16GB system, and two, the less RAM you have, the harder it is for the system to identify good candidates for eviction to the L2ARC, meaning that the L2ARC's effectiveness is hobbled when you don't have enough RAM. In practice, L2ARC with less than 64GB of ARC seems to have a tough time working well, and we've suggested 64GB minimum to support L2ARC for a number of years now.

Thanks for the explanation. I have a better idea of how L2ARC works. While I didn't plan to buy new RAM sticks and will stick on 32GB for now, the L2ARC drive will be repurposed to be something like Etorix said, an application or scratch vdev.

 

[Shigure]

This is generally done to save a SATA port for data drives.
In your case, with apps and "only" 32 GB, it's probably best to forget about a L2ARC and just put the boot drive in the free M.2 slot.


Full sense.

An alternative layout would be to use the 2 TB NVMe as (single drive, not redundant) application pool and keep all six bays for 3.5" HDDs. With frequent backups (replication) to the HDD pool it should be possible to restore to a new drive if the single drive vdev fails.


I suspected something like this for 2.5" drives.
And you did a good search on raid5/raidz1 issues.

Now I'm thinking either use the 2TB drive as application/scratch pool, or just install the boot drive into that M.2 slot, but I already have the adapter for a internal USB 3.2 header maybe I will still use that to boot.
The 2.5 drive stack bracket looks kinda funky but does the job in my other projects. https://www.thingiverse.com/thing:582781 I made some modification based on this and tested stack 2 drives is pretty good with Node 304. Though I won't really recommend others to do the same way lel.
With more reading and realized I have 4 HDDs in total, it's time for RAIDZ2 and stripe mirror debating(jk). Even I have the 10GbE NIC I don't think my work load(media/backup server) will benefit too much from the stripe mirror, my VMs was on a separate machine. And I can always fully utilize the 10GbE bandwith with my stripe mirror SSD zpool(maybe the NVMe scratch zpool as well) therefore I decided to go for RAIDZ2.
And thanks for all the great advices!

 

[Shigure]

The only thoughts I have that haven't been mentioned is heat. Cramming that many drive in (I am not familiar with the case) - just make sure its well ventilated as both the drives and LSI Card need cooling (and so I suspect does the mellanox)

Yeah that's something I considered when choose the case.
Node 304 is a case with decent air flow and with all other advices, I will have 4x HDDs and 4x SSDs. With two 80mm intake and one 140mm exhaust Noctua fans it should be fine, I will test the fan speed and temp before closing the case to make sure it is cool and queit.
And for the HBA and NIC, I'm designing a duct to use a 40mm/60mm Noctua fan to cool those two components, based on my test a 40mm fan running about 35% speed, LSI 9208 heatsink temp dropped from 52C to 40C(though the fan was basically stick on the heatsink). I know the 10GbE NIC will be hot, and some tests will be necessary before I put the server into production phase.

 

[Scharbag]

16TB drives in RaidZ1 for a production pool is a bit dangerous. As others have said, get another drive and go RaidZ2 for production.

Also remember, redundancy never equals a backup!!!

 

[Shigure]

16TB drives in RaidZ1 for a production pool is a bit dangerous. As others have said, get another drive and go RaidZ2 for production.

Also remember, redundancy never equals a backup!!!

Yeah I will get 4 drives instead, really read a lot of materials these a few days.
Though I cannot really achieve the 321 of backup but I do have a cold copy and a cloud copy. While if I want to access some archived files or check backups, I either need to plug in some external drives or need wait for data retriving from cloud. Therefore I started this home server project to provide more availability for my data lol.

 

[joeschmuck]

Do you have any idea about how much storage capacity you need? What I'm asking is how much storage will you need for about 3 to 5 years (what a drive typically will last). Do you need about 29TB of storage? That is approximately what four 16TB drives would get you in a RAIDZ2 setup, minus some overhead. As others have said, it's unsafe to place such a large hard drive in a RAIDZ1 due to resilvering times.

There are people here who have small capacities like myself, 21.8TB and I use to run just over 7TB for years. The only reason I have a larger capacity is due to a major reconfiguration of my pool and sale of some hard drives at the time. I'd still be fine if I only had 7TB of storage. But that is me. There are folks here who have very huge capacities that I can't wrap my head around. I was reading about a person who had 16x18TB drives (8 drives in a single system) to make up that set of pools. There was a lot of redundancy too so it's not 288TB, one big stripe would be foolish. But that is a huge capacity.

So before you buy hard drives, figure out what you need for up to 5 years. And remember, hard drives will fail and will require replacement eventually. It is the one item you can plan on failing. So don't spend too much money on the ultra large capacity if you won't actually be using it.

 

[Shigure]

Do you have any idea about how much storage capacity you need? What I'm asking is how much storage will you need for about 3 to 5 years (what a drive typically will last). Do you need about 29TB of storage? That is approximately what four 16TB drives would get you in a RAIDZ2 setup, minus some overhead. As others have said, it's unsafe to place such a large hard drive in a RAIDZ1 due to resilvering times.

There are people here who have small capacities like myself, 21.8TB and I use to run just over 7TB for years. The only reason I have a larger capacity is due to a major reconfiguration of my pool and sale of some hard drives at the time. I'd still be fine if I only had 7TB of storage. But that is me. There are folks here who have very huge capacities that I can't wrap my head around. I was reading about a person who had 16x18TB drives (8 drives in a single system) to make up that set of pools. There was a lot of redundancy too so it's not 288TB, one big stripe would be foolish. But that is a huge capacity.

So before you buy hard drives, figure out what you need for up to 5 years. And remember, hard drives will fail and will require replacement eventually. It is the one item you can plan on failing. So don't spend too much money on the ultra large capacity if you won't actually be using it.

Good point. I didn't choose the X20 drives because they are too expensive.
With the 20% free space recommendation, there will be about 24TB available for use. Currently I have 4x 3TB cold archive and about 3TB data on my main PC need backup. If I made a 5 year plan it seems not that bad...? It's about 50% full once I dump my data to the zpool.

 

[Scharbag]

ZFS is also very unforgiving. Once you have made a choice, it is almost impossible to tweak it. That is where your crystal ball comes into play. Running RZ2 with fewer than 6 drives sucks for space efficiency. Running mirrors with super huge disks is dangerous. RZ1 is pretty much a waste of time with large volumes. Speed comes with stripes, stripes need more drives. The list goes on and on.

So, if you only have 4 bays, then you are pretty much stuck with RZ2. The only positive is that if you need more space in the future, you can replace the 4 RZ2 drives (one at a time) and the pool will auto expand. I have done this successfully a few times. My 2x6 drive RZ2 production array has met my needs for 7+ years now. But that takes up 12 bays... And it took me a few stabs at it back in the day (starting with Open Indiana back in 2011) to get to where I am.

Anywho, proper planning prevents a poor performance is so very true with ZFS.

Happy TrueNASing!!

 

[Shigure]

ZFS is also very unforgiving. Once you have made a choice, it is almost impossible to tweak it. That is where your crystal ball comes into play. Running RZ2 with fewer than 6 drives sucks for space efficiency. Running mirrors with super huge disks is dangerous. RZ1 is pretty much a waste of time with large volumes. Speed comes with stripes, stripes need more drives. The list goes on and on.

So, if you only have 4 bays, then you are pretty much stuck with RZ2. The only positive is that if you need more space in the future, you can replace the 4 RZ2 drives (one at a time) and the pool will auto expand. I have done this successfully a few times. My 2x6 drive RZ2 production array has met my needs for 7+ years now. But that takes up 12 bays... And it took me a few stabs at it back in the day (starting with Open Indiana back in 2011) to get to where I am.

Anywho, proper planning prevents a poor performance is so very true with ZFS.

Happy TrueNASing!!

I have read a lot about the scaling ability of ZFS. Either you add a new vdev with the same layout of drives to the pool, or replace the exist drives one by one like the old school way. I know It sucks but at least there is a choice lel. And for a 4 drives RAIDZ2, I can only have 50% of the raw capacity just like a stripped mirror vdevs so yes the cost efficiency is really bad, which is the cost of reliability.
7 years is a long way, I hope my setup will last that long though I didn't really have more bays in the Node 304. I have been keeping those external drives or bare drives and enclosure for too long and it's time for TrueNAS to shine XD

 

[DigitalMinimalist]

I‘m new to ZFS too and I will experiment with 4x 4TB HDDs and 2x 1TB SSD to determine the best vDev design.
Special vDev for Metadata + small files with SSDs and either RAIDZ2 or striped mirror for HDDs.
Very curious about speed differences

 

[ChrisRJ]

For a relatively high-capacity small form factor system, a 3-way mirror is also an option. Together with e.g. a 1 TB SSD that would make a nice all-in-one system for SMBs, once TrueNAS Scale is mature enough.

At least here in Europe, with surging prices for electricity, the number of drives is increasingly becoming a factor to consider.

 

[Etorix]

And for a 4 drives RAIDZ2, I can only have 50% of the raw capacity just like a stripped mirror vdevs so yes the cost efficiency is really bad, which is the cost of reliability.

50% is not THAT bad, tough I would concur that 6-8 wide is the sweet spot for raidz2.

The Node 304 has 6 bays.
With two bays adapted to hold 4*2.5" that leaves a 4-wide raidz2, which would make for small and secure setup.
To use all bays for 3.5", you may shift the app pool to a single NVMe drive (with backup to the HDD pool), or, if the board can bifurcate x8x4x4, use this adapter to run the HBA and a mirror of M.2 NVMe drives for the app pool. This would cost more to set up, and I have no idea where the financial sweet spot is: Four 16 TB or six slightly smaller drives.