Best configuration with 10 Drives

[SuperMiguel]

Hello, i currently have a server with 10 4TB drives and 2 SSD 1 for booting one that i can use for cashing, server has 64GB of ram (i can add more if needed) This server will be used mostly for storing files and to hold daily backups from my proxmox cluster (NFS share), maybe store some plex movies, experiment with some new True Scale capabilities. Server is for home use so nothing ultra critical... that being said, how should i set up the 10 drives? multiple vdevs? or single raid-z2? raid-z3? I think with raid-z3 i get about 25T which should be enough for now. As far as read/write speed im hoping that cache drive and ram can help a bit

 

[TinyWorkshop]

I think raid-Z2 is the right one for your setup.

 

[SuperMiguel]

I think raid-Z2 is the right one for your setup.

Single vdev? Or 2 of 5 drive each raid-z2?

 

[TinyWorkshop]

I think 2x5 is better for your use-case.

but wait for someone more expert than me

 

[TinyWorkshop]

Anyway you can try some simulations HERE

 

[ChrisRJ]

I would certainly favor a RAIDZ3 over 2*RAIDZ2 with the information available.

How does the rest of the hardware look like?

Are the drives CMR or SMR? The latter will not work properly with ZFS.

EDIT:

Please also check this:

Welcome to TrueNAS SCALE - Beginners Intro

Since mid 2021 we have had many new users of TrueNAS wanting to try SCALE. Here are some things that are helpful for new TrueNAS SCALE users to know; SCALE is a specialized & targeted OS based on a Linux distro. However, SCALE is NOT a Linux...

www.truenas.com

 

[WI_Hedgehog]

Hello, i currently have a server with 10 4TB drives and 2 SSD 1 for booting one that i can use for cashing, server has 64GB of ram (i can add more if needed) This server will be used mostly for storing files and to hold daily backups from my proxmox cluster (NFS share), maybe store some plex movies, experiment with some new True Scale capabilities. Server is for home use so nothing ultra critical... that being said, how should i set up the 10 drives? multiple vdevs? or single raid-z2? raid-z3? I think with raid-z3 i get about 25T which should be enough for now. As far as read/write speed im hoping that cache drive and ram can help a bit

The first thing I personally want to question is what hardware will TrueNAS be running on?

Second is: Will it be running in a virtual machine?

2a: Will you use TrueNAS to run VMs?

Third (that rolls through my brain, unasked): Why, for the love of God, did you buy hardware before you started asking questions???? (this may not apply to your situation)

 

[souporman]

Sounds like you won't be needing any crazy performance. No matter what you select, you'll be able to saturate 1Gbps links easily. It's important to keep the future in mind when planning a big NAS as well. Do you think you might want to grow the pool down the road? Consider that a RAIDz1/2/3 in TrueNAS can't be expanded 1 or 2 disks at a time, you'd need to add a second stripe of the same size (another full 10 disks).

That being said, I'm a big fan of striped mirrors on a home NAS for this reason. When I need more space in my pool, I just buy 2 more disks instead of 10. Striped mirrors would be a much more performant pool for random writes, but obviously you pay for that in capacity. You could use the suggestion above, 2x5 RAIDz2 VDEVs, but at that point you're losing 40% of your capacity already, so you might as well just go for 50% and enjoy a faster pool with easier expansion paths.

 

[ChrisRJ]

Consider that a RAIDz1/2/3 in TrueNAS can't be expanded 1 or 2 disks at a time, you'd need to add a second stripe of the same size (another full 10 disks).

Not necessarily.

You can extend the size of the ZFS pool by adding any type of VDEV. But you also need to keep in mind that the additional VDEV can reduce the overall resilience. If you decide to add a 2-way mirror to the RAIDZ3, that mirror obviously has a lower level of resilience (1 disk can die) than the RAIDZ3 (3 disks can die) before you loose data.

But you are correct that extending is rather limited compared to e.g. a Synology, unRAID, etc. Here the data center origin of ZFS can be a bit of a challenge. In the data center, you would never add 1 or 2 additional disks anyway but another disk shelve with e.g. 30-50 drivers.

 

[souporman]

You can extend the size of the ZFS pool by adding any type of VDEV. But you also need to keep in mind that the additional VDEV can reduce the overall resilience. If you decide to add a 2-way mirror to the RAIDZ3, that mirror obviously has a lower level of resilience (1 disk can die) than the RAIDZ3 (3 disks can die) before you loose data.

Can you think of a legitimate use case for doing that?

 

[ChrisRJ]

With a RAIDZ3 it is probably not so realistic. But let's assume you have an older RAIDZ1 with 5*4 TB (ca. 16 TB) net capacity and want to add another 16 TB. Today you would probably just add a mirror with 2*16 TB.

 

[SuperMiguel]

Hello All! and thanks for the many wonderful replies, as far as hardware i have 10 st4000nm0043 (Seagate 4TB SAS 7200 RPM) the 2 SSD are cheap 250GB Samsung EVO SSD, as far as hardware i have a Dell R720XD

Other questions: Will it be running in a virtual machine? No TrueNAS will be installed directly
Will you use TrueNAS to run VMs? No, but ill probably run a container here and there
Why, for the love of God, did you buy hardware before you started asking questions???? Already had the hardware, was used for unraid, but got tired of it always having software issues after every update

Also 90% of the time this server will be used for daily backups from my proxmox cluster (NFS share) over a 10G line

 

[joeschmuck]

2 SSD 1 for booting one that i can use for cashing,

For your use case the SSD used for caching (L2ARC) would be a waste and just add more needless hardware into the mix. Caching is great IF you are repeatedly accessing the same set of files, not to cache large files infrequently. For example: You are accessing a database every few seconds to minutes, but it's always the same file(s). The cache would be able to present the data faster. If you are looking at cahing video content to watch a movie for example, or to create or restore a large backup file, these are all going to slow down due to the cache.

Also you have 64GB of RAM (hopefully ECC RAM) and that is a huge cache automatically.

hardware i have 10 st4000nm0043 (Seagate 4TB SAS 7200 RPM)

That is a lot of power eating, heat producing drives just for backups and video streaming. And for storing backups and videos, I'd choose probably two separate RAIDZ2 or something, one VDEV for video and backup content, another VDEV for important data. that is just me. If you go with a 10 drive single VDEV, I'd go RAIDZ3 but that depends on how important the data is to you. If not important then RAIDZ1 or RAIDZ2.

 

[SuperMiguel]

For your use case the SSD used for caching (L2ARC) would be a waste and just add more needless hardware into the mix. Caching is great IF you are repeatedly accessing the same set of files, not to cache large files infrequently. For example: You are accessing a database every few seconds to minutes, but it's always the same file(s). The cache would be able to present the data faster. If you are looking at cahing video content to watch a movie for example, or to create or restore a large backup file, these are all going to slow down due to the cache.

Also you have 64GB of RAM (hopefully ECC RAM) and that is a huge cache automatically.


That is a lot of power eating, heat producing drives just for backups and video streaming. And for storing backups and videos, I'd choose probably two separate RAIDZ2 or something, one VDEV for video and backup content, another VDEV for important data. that is just me. If you go with a 10 drive single VDEV, I'd go RAIDZ3 but that depends on how important the data is to you. If not important then RAIDZ1 or RAIDZ2.

Is there anything better for the ssd to be used for at this moment? Using it for raid1 install would be nice but dont want to install again

 

[joeschmuck]

Some people would use it as a mirror for the boot drive. Personally I would just not install it. You have a spare or you could use it for some other project.

 

[WI_Hedgehog]

Some people would use it as a mirror for the boot drive. Personally I would just not install it. You have a spare or you could use it for some other project.

I'm one of those people. Cheap and easy. (The mirror, not me necessarily. The mirror is cheap and easy. Yeah, that's what I meant.)

 

[HoneyBadger]

Is there anything better for the ssd to be used for at this moment? Using it for raid1 install would be nice but dont want to install again

You don't have to reinstall TrueNAS to do this - you can attach the second SSD to your boot-pool using System Settings -> Boot -> Boot Pool Status - then select the three-dot menu by your existing boot device and choose to Attach the other SSD. (Obviously it would need to be removed from the pool if it's in use as a cache device.)

You could also use it as a non-redundant high-speed pool for test data only.

 

[SuperMiguel]

You don't have to reinstall TrueNAS to do this - you can attach the second SSD to your boot-pool using System Settings -> Boot -> Boot Pool Status - then select the three-dot menu by your existing boot device and choose to Attach the other SSD. (Obviously it would need to be removed from the pool if it's in use as a cache device.)

You could also use it as a non-redundant high-speed pool for test data only.

Ha! tried attaching the drive using boot pool status, and realized that boot drive is a 250gb, while other ssd is 240gb, so i get an error saying its too small to be attached :(

 

[HoneyBadger]

Ha! tried attaching the drive using boot pool status, and realized that boot drive is a 250gb, while other ssd is 240gb, so i get an error saying its too small to be attached :(

Ouch. Sorry to hear that - if it had been the other way around (installed on 240gb, empty is 250gb) it would've worked - but sadly there's no zfs shrink command.

It can still be used as the extra drive for apps/containers/playing around; and it could still be attached as a cache device to your main pool, maybe set to metadata-only (to save writes) in a hope to accelerate directory listings?

 

[WI_Hedgehog]

Ouch. Sorry to hear that - if it had been the other way around (installed on 240gb, empty is 250gb) it would've worked - but sadly there's no zfs shrink command.

It can still be used as the extra drive for apps/containers/playing around; and it could still be attached as a cache device to your main pool, maybe set to metadata-only (to save writes) in a hope to accelerate directory listings?

Uh, no. Back up the config files from the larger drive, shut down, replace the larger drive with the smaller drive, do a fresh OS install, copy the config file over, boot, the server OS is now on the smaller drive. Add the larger drive. Bam. That easy.

--
You can drink me later.
Uh, I mean you can thank me later. By buying me a drink. Later.
Because it's Friday and I'm already loaded.
It is Friday? Right?