[TrueNAS noob] Please validate my config

[iRomain]

Hello!

I currently have a DS415+ that has served me well for years, but it is now time to improve my backup strategy and move on!

I plan to move this DS415+ off-site as a backup NAS and replace it at home with TrueNAS Scale on a custom build (details in my signature).

I have 3 types of storage needs for my NAS:

• Backups: Personal computer backups, photos, etc.
  • Estimated size: 4-6Tb
  • Protection against 2-disk failures
  • Encrypted
  • Off-site backup
• Apps: App configuration files for Docker containers
  • Estimated size: 50Gb
  • Protection at least against 1-disk failure (or more if it makes sense for configuring the pools)
  • Encrypted
  • Off-site backup
• Data Hoarding: Downloaded files
  • Estimated size: Whatever space will be remaining (10Tb+)
  • Protection against 1-disk failure
  • Not encrypted
  • No off-site backup

I have the following drives:

• 4x HDD WD Red 4Tb
• 2x HDD IronWolf Pro 8Tb
• 1x HDD IronWolf 8Tb
• 1x HDD Exos X18 18Tb
• 1x NVMe WD Blue 1Tb
• 1x NVMe Samsung 970 EVO Plus 256Gb

My motherboard ships with 2x NVMe and 8x SATA III.

• NVMe 1: EVO 970 256Gb
• NVMe 2: WD Blue 1Tb
• Sata 1: Exos 18Tb
• Sata 2: IronWolf Pro 8Tb
• Sata 3: IronWolf Pro 8Tb
• Sata 4: IronWolf 8Tb
• Sata 5: WD Red 4Tb
• Sata 6: -
• Sata 7: -
• Sata 8: -

TrueNAS configuration I am considering:

• Boot: NVMe 1: EVO 970 256Gb
• BackupPool (Three-way mirror):
  • Sata 2: IronWolf Pro 8Tb
  • Sata 3: IronWolf Pro 8Tb
  • Sata 4: IronWolf 8Tb
• AppsPool: (Mirror)
  • NVMe 2: WD Blue 1Tb
  • Sata 5: WD Red 4Tb
• DataPool: (Mirror)
  • Sata 1: Exos 18Tb
  • Buy another Exos 18Tb

I am thinking that my DS415+ can run Raid-5 to protect against one disk failure and load it with 3x 4TB drives. That way, I will have 7.3Tb of usable space for backing up my primary NAS' "BackupPool" and "AppsPool". I guess I could load up the last 4Tb drive and run Raid-6, but is that really necessary since this is already a backup NAS?

Is this a good use of what I already have? Is this iefficient? Can you think of something better? Could I merge the AppsPool with the BackupPool but use the SSD in front? Is that what ZIL does?

 

[Arwen]

Pool configuration looks reasonable. A couple of comments, which you may know already;

• When using lopsided mirror pairs, (aka drives of different sizes), ZFS will only use the smaller size for that vDev. (In a single vDev pool, that also means the pool size.) So, a mirror of 1TB & 4TB, it is only a 1TB pool. More than okay from what you listed as desired for Apps.
• Western Digital Red drives need to be checked for SMR. The WD Red Plus and WD Red Pros, were fine last I checked. But, the plain WD Reds at the 4TB size might be SMR, which is really not suitable for ZFS.
• ZFS encryption is only at rest. Meaning if you return a disk for warranty or the NAS is stolen, the encryption stays in place. (But, for theft protection, you need to use a password, required at each boot. Not the passphrase, which automatically unlocks you pools.) Whence the encrypted pools are unlocked, any hacker that gets in, has full access to the data.

You might create a proposed hardware set, CPU, system board, etc... and list it here too for review. Their are common less than ideal choices, like gaming system boards, or ones that have RealTek Ethernet chips that people make, causing issues during usage.

 

[iRomain]

Pool configuration looks reasonable. A couple of comments, which you may know already;

• When using lopsided mirror pairs, (aka drives of different sizes), ZFS will only use the smaller size for that vDev. (In a single vDev pool, that also means the pool size.) So, a mirror of 1TB & 4TB, it is only a 1TB pool. More than okay from what you listed as desired for Apps.
• Western Digital Red drives need to be checked for SMR. The WD Red Plus and WD Red Pros, were fine last I checked. But, the plain WD Reds at the 4TB size might be SMR, which is really not suitable for ZFS.
• ZFS encryption is only at rest. Meaning if you return a disk for warranty or the NAS is stolen, the encryption stays in place. (But, for theft protection, you need to use a password, required at each boot. Not the passphrase, which automatically unlocks you pools.) Whence the encrypted pools are unlocked, any hacker that gets in, has full access to the data.

You might create a proposed hardware set, CPU, system board, etc... and list it here too for review. Their are common less than ideal choices, like gaming system boards, or ones that have RealTek Ethernet chips that people make, causing issues during usage.

Thanks for your reply. I didn't know about the SMR/ZFS. I guess that makes even more sense to leave the WD drives in the Synology NAS.

What about having the apps configuration files on the same pool as my personal files and use my NVMe WD Blue 1Tb as a ZIL? Would that be a better strategy?

Regarding my hardware specs, they're in my signature, the only thing I'm unsure about is the RAM:
16Gb is more than enough for the TrueNAS minimum recommended 8Gb + My Docker apps. But should I get more? Why?

 

[Etorix]

First, check the WD Red. If SMR forget them for TrueNAS; use in your desktop, or as external backup.
If they are not SMR, you have a possible alternative layout:
Backup = 3*8 TB + 18 TB as 4-wide raidz2 (and increase size by replacing 8 TB drives by larger ones as you go)
Data = 4*4 TB as 4-wide raidz1
Pool = WD 1 TB, and buy another 1 TB NVMe drive, held on a (cheap) adapter in a PCIe slot

If "4-6 TB" is the estimated size of Backup right now, a 3-way mirror would be 50-75% at start, and it is advised to stay below about 80%.

- CPU: Intel Core i3-9100
- Motherboard: New C246 Chipest i226 Dual 2x2.55GBE lans 8SATA 3PCIE MATX 9.6\*9.6inch NAS Computer Motherboard Support 8TH 9TH CPU
- Case: Fractal Design Node 804
- Power supply: Be Quiet! Straight Power 11 750W
- RAM: 2x Kingston KSM32ED8/16MR 16 Go DDR4 3200 MHz ECC

Fair setup, with two caveats:
If the PSU brand is an indication of a design goal, be warned that the Node 804 is NOT a quiet case. HDD noise goes straight out through the top mesh.
Congratulations on finding a new C246 motherboard for 145,- EUR! But the unknown manufacturer on AliExpress is a cause of concern with respect to reliability—or even whether ECC is properly implemented. Plus the lack of IPMI and the new-but-no-so-great 2.5 GbE NICs. Right now, one can find AsRock Rack E3C246D4U-2T for $250 (China) or $280 (US). No "basic" variant without 10 GbE on offer as of this writing, but much safer options even you do not need 10 GBase-T.

 

[Arwen]

..
What about having the apps configuration files on the same pool as my personal files and use my NVMe WD Blue 1Tb as a ZIL? Would that be a better strategy?
...

It is helpful to learn & use ZFS terminology;

Terminology and Abbreviations Primer

We realize that new users have a lot to learn when they come to FreeNAS. There's a certain amount of confusion added to discussions when users pick random/approximate terms to describe things. I've spent a lot of time quietly trying to...

www.truenas.com

All ZFS pools contain a ZIL, (ZFS Intent Log). What you probably meant was a SLOG, (Separate Intent Log). Further, s SLOG is not a write cache. ZIL or SLOG are used for synchronous writes, for iSCSI VM storage or NFS. But, you don't list that as something you want to use.

Yes, you can have your Apps configuration files on the same pool as your personal files. It's all about trade offs for YOU. If the speed is not good, you can always change it later.

Some NAS hardware / software vendors, (QNAP, Netgear, etc...), supply both hardware and a NAS OS. Then give you some hardware options, (size of disks, sometimes amount of memory, sometimes a PCIe slot for 10Gbps Ethernet). Plus, some OS options like Samba shares, some Apps and maybe some backup choices.

With non-iX built TrueNAS, all the choices are yours.

 

[iRomain]

First, check the WD Red. If SMR forget them for TrueNAS

I checked and they are SMR. I'll stick the four of them in my backup NAS.

Backup = 3*8 TB + 18 TB as 4-wide raidz2 (and increase size by replacing 8 TB drives by larger ones as you go)
Data = 4*4 TB as 4-wide raidz1
Pool = WD 1 TB, and buy another 1 TB NVMe drive, held on a (cheap) adapter in a PCIe slot

If "4-6 TB" is the estimated size of Backup right now, a 3-way mirror would be 50-75% at start, and it is advised to stay below about 80%.

The problem with 4-wide raidz2 is that the usable space will be 2x the smallest disk. So that means I won't get any benefit from larger drives until the last 8Tb is replaced, so basically until this becomes effectively 4x18Tb, am I right? I don't expect this to happen before a few years since I just bought recently two of these.

If the PSU brand is an indication of a design goal, be warned that the Node 804 is NOT a quiet case. HDD noise goes straight out through the top mesh.

Thanks for the warning, actually I don't mind some noise since it's not in my bedroom. The alternative was a Define 7 but that adds on the costs and seems to be even more bulky. I guess I'll see when I receive it how I feel about it.
My main design goal with this setup was actually power efficiency.

Congratulations on finding a new C246 motherboard for 145,- EUR! But the unknown manufacturer on AliExpress is a cause of concern with respect to reliability—or even whether ECC is properly implemented. Plus the lack of IPMI and the new-but-no-so-great 2.5 GbE NICs. Right now, one can find AsRock Rack E3C246D4U-2T for $250 (China) or $280 (US). No "basic" variant without 10 GbE on offer as of this writing, but much safer options even you do not need 10 GBase-T.

I am in Europe and I can't find an affordable motherboard that supports ECC. Note that for any import over 150€, I will have to pay additional customs tax + transporter's tax filling fees, so I don't to go down that PITA route.
So yes this was indeed really hard to find and I kind of gave up and purchased the chinese one because of the (two :rouler des yeux:) comments on that listing that are, to me, hints that it works. There's also a free 15 days return policy on that item from the seller.

All ZFS pools contain a ZIL, (ZFS Intent Log). What you probably meant was a SLOG, (Separate Intent Log). Further, s SLOG is not a write cache. ZIL or SLOG are used for synchronous writes, for iSCSI VM storage or NFS. But, you don't list that as something you want to use.

You're right that's what I meant (And I had just read the article ZFS ZIL and SLOG Demystified - shame on me!). Thanks for the answer, I got confused because I currently run my apps on a separate machine and use NFS a lot. But I don't think I will be using it that much with that new setup.

Yes, you can have your Apps configuration files on the same pool as your personal files. It's all about trade offs for YOU. If the speed is not good, you can always change it later.

I think that's the wisest route.

So to recap my configuration:

I will setup my backup NAS on Raid-5 with 4x4Tb WD Red drives.
Regarding my TrueNAS NAS, since everything on the DataPool will be on my backup NAS 20mn away (+ another backup in the cloud), I guess I can live with 1-disk protection and not be so close to full disk capacity as @Etorix warned.

• Boot: NVMe 1: EVO 970 256Gb
• DataPool (Raidz1):
  • Sata 2: IronWolf Pro 8Tb
  • Sata 3: IronWolf Pro 8Tb
  • Sata 4: IronWolf 8Tb
• HoardingPool: (Mirror)
  • Sata 1: Exos 18Tb
  • Buy another Exos 18Tb

I will setup a regular snapshot replication task from the "Apps" dataset (located in the DataPool) to the HoardingPool so I can recover those quicker in case of a 2 or more disk failure on my DataPool.

 

[Etorix]

The problem with 4-wide raidz2 is that the usable space will be 2x the smallest disk. So that means I won't get any benefit from larger drives until the last 8Tb is replaced, so basically until this becomes effectively 4x18Tb, am I right? I don't expect this to happen before a few years since I just bought recently two of these.

All correct. It was also my assumption that expansion would not be complete before a few years, but that would let you start with 16 TB of raw space, and 4-wide is a good fit with the Node 804 (or 5-wide with a full drive cage plus a 5th drive on the floor of the CPU chamber).

I am in Europe and I can't find an affordable motherboard that supports ECC. Note that for any import over 150€, I will have to pay additional customs tax + transporter's tax filling fees, so I don't to go down that PITA route.

Same here, but I'd take a $250 (plus ca. 25% duties+fee) AsRockRack board with IPMI and X550 over a 145 E (+25%, no minimum here) unknown board any day.
Please let us know how it goes with the "Cloud Star" board.

 

[iRomain]

Please let us know how it goes with the "Cloud Star" board.

I will definitely do and also post my feedback on the AliExpress listing like that other guy did.

 

[iRomain]

I was re-questioning my redundancy strategy vs my backup strategy because nothing on this NAS needs 100% uptime. So I thought of another layout...

Primary NAS:

• Boot: 238 GiB | 0-disk protection | replicated locally | backed up remotely
  • SSD Samsung EVO 970 256Gb
• AppsPool: (Stripe): 0.9TiB | 0-disk protection | replicated locally | backed up remotely
  • SSD WD Blue 1Tb
• DataPool (Raidz1): 14.6TiB | 1-disk protection | backed up remotely
  • IronWolf Pro 8Tb
  • IronWolf Pro 8Tb
  • IronWolf 8Tb
• HoardingPool: (Stripe): 16.4TiB | 0-disk protection | backed up remotely
  • Exos 18Tb

Backup NAS:

• DataVolume (SHR): 7.3TiB | 1-disk protection
  • WD Red 4TB
  • WD Red 4TB
  • WD Red 4TB
• HoardingVolume (SHR): 16.4TiB | 0-disk protection
  • Exos 18Tb

- If my Boot drive fails, I can put my NAS offline and totally wait until I buy another disk and restore the snapshot locally.
- If AppsPool drive fails, same, I'll buy another disk and restore the snapshot locally.
- If DataPool (my most critical data) fails, I can drive 20mn and copy my files from the Raid array to a new disk. If the array fails, I have my third backup in the cloud.
- If HoardingPool fails, I can drive 20mn and clone my backup disk to a new disk. If the backup disk fails, I'll be sad and have to re-download stuff but it won't be the end of the world.

Any remarks regarding this new layout?

 

[Etorix]

Boot: 238 GiB | 0-disk protection | replicated locally | backed up remotely

No need to replicate a boot pool. Just keep a copy of the configuration file outside of the NAS, and reinstall a fresh OS on a new drive.

DataPool(Raidz1): 14.6TiB | 1-disk protection | backed up remotely

Raidz1 is not advised with large drives, and you have no "upgrade path" to safer raidz2. Consider buying a fourth drive.