Advice:RaidZ2 or mirror with ability to expand.

[Simtech]

Good evening,

I have decided to move my data from storage solution to TrueNAS and initially decided on RaidZ2 but I'm concerned about re-silvering and availability. I currently have 4 x 14 TB drives and won't be in a position to buy any additional drives for a few months and it would be limited to 2 drives, unless one failed and I needed to rebuild the array. If I opted for Mirror could I later add another 2 drives to pool1 and increase the pool by another 14TB?

In the scenario below using mirrors how is data distributed in ZFS, if a single file was written to the array would it be stored on Drive 1 and a copy on drive 3. Once drive 1 was full it would then use drive 2 and mirror that across to drive 4? I suppose what I am asking is if a single drive was pulled could you read the data or would 2 drives be required as the data is split?
Pool 1
MIRROR
drive 1 p2
drive 2 p2
MIRROR
drive 3 p2
drive 4 p2

Thanks for your time
Simon

 

[Adrian]

Yes, you could add another pair of mirrored disks (vdev) later.
The data content of the mirrored disks (e.g. drive 1 and drive 2 or drive 3 and drive 4) is identical.
So data written to drive 1 would be written (mirrored) on drive 2.
If you were to lose or pull drive 1 all your data would still be available, but if you then lost drive 2 you would lose the entire pool.
So, it is a good idea to have a burnt in disk ready for use.
As I understand it, when new data is written the least full vdev is used, so usage is evened out.

My main system has 8 disks, as 4 mirrored pairs (vdevs).
If I had more drive slots I could add more pairs.

[root@freenasxl /mnt/tank/unix]# zpool status tank
pool: tank
state: ONLINE
scan: scrub repaired 0B in 09:39:45 with 0 errors on Tue Feb 9 18:39:55 2021
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/fb89e15d-fc36-11e6-8813-d05099c235eb ONLINE 0 0 0
gptid/ff9b7c27-be30-11e9-81d2-d05099c235eb ONLINE 0 0 0
mirror-1 ONLINE 0 0 0
gptid/5c19c300-fc37-11e6-8813-d05099c235eb ONLINE 0 0 0
gptid/5d54f579-fc37-11e6-8813-d05099c235eb ONLINE 0 0 0
mirror-2 ONLINE 0 0 0
gptid/f7257f49-fc37-11e6-8813-d05099c235eb ONLINE 0 0 0
gptid/f854d536-fc37-11e6-8813-d05099c235eb ONLINE 0 0 0
mirror-3 ONLINE 0 0 0
gptid/35c5529f-fc38-11e6-8813-d05099c235eb ONLINE 0 0 0
gptid/36f42a8e-fc38-11e6-8813-d05099c235eb ONLINE 0 0 0
logs
ada2 ONLINE 0 0 0
cache
ada3 ONLINE 0 0 0

Whereas my backup system (kept up to date with ZFS snapshot replication) has the same number and type of disk as a single 8 disk RAIDZ3 vdev.
If I had more drive slots the only way I could use it would be by adding another vdev. RAIDZ expansion is coming, but do not hold your breath for it!

This layout is "performant" but losing the wrong two disks would lose the entire pool, and resilvering a lost disk stresses the remaining one in the vdev.

[root@freenasxlp ~]# zpool status tank
pool: tank
state: ONLINE
scan: scrub repaired 0B in 07:58:22 with 0 errors on Tue Feb 9 16:58:24 2021
config:

NAME STATE READ WRITE CKSUM
tank ONLINE 0 0 0
raidz3-0 ONLINE 0 0 0
gptid/02ba94e8-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0
gptid/03a92de4-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0
gptid/04927b07-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0
gptid/0578ff90-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0
gptid/066befa2-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0
gptid/07549fec-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0
gptid/08495616-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0
gptid/0983ff78-c25f-11e9-8582-ac1f6b0adb00 ONLINE 0 0 0

This layout is (much) less performant but more resilient. I would have to lose 4 disks to lose the pool. Resilvering should be less stressful. Additionally, the pool has another disk's worth of space.

I strongly recommend reading Eric Loewe's Introduction to ZFS before making any decision!

 

[ChrisRJ]

@Simtech, what are your requirements? I mean how critical are the data, what performance do you expect, how good is your fully automated backup, etc.?

 

[Simtech]

@Adrian Thank you for the link, that document made things a lot easier to understand :)

@ChrisRJ, The requirements aren't that demanding, I'm the only user and want general storage for anything and everything I own. I'm running TrueNAS on Esxi which is allows me to run different virtual environments for university work. 1Gbe network speeds are fine as the server only has me to contend with. I run a windows VM once a week to update my Steam, Uplay, Epic and origin games. I'm fully aware I can download these at any time but I'm slightly OCD and like the idea of having any game I fancy playing at hand :). I sometimes stream movies on an evening but nothing 4k, my eyes arn't that good :)

Once a week I have an old 2700k system with a selection of random drives (2x2TB | 2x3TB | 2x4TB | 2x6TB) which only copies files that have been modified. No striping/mirror/parity/raid just drives formated with LUKS so if one dies I only lose the data it stores and the rest are fine. I have about 21TB of data to store - 17TB if I delete games.

Originally I planned on RaidZ2, but reading how long a resilver process can take on larger drives made me worry, I didn't want the data stored to potentially be unavailable or slow to access during this time, the machine is only a 4700k 4/8 with 32GB with an LSI SAS 9207-8.

On a lighter note, I came across my old 486dx-66 a few months back in my parent's loft, that beast has a 40MB hard drive :)

Thanks,
Simon

 

[ChrisRJ]

Since you want to run inside ESXi, please have a look here

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

 

[Simtech]

I'm currently running passthrough for the LSI card and another intel dual-port NIC, I initially went with Esxi7.0 but they list the LSI SAS 9207 as unsupported/deprecated now so went with 6.7u2. I could go back to running 2 separate systems but I only really need 2 VM's running and they don't go under any heavy loads. If I can go from 3 computers to 2 it would make life easier :)

 

[Etorix]

Definitively go for RAIDZ2 if you care about data safety. With 14 TB drives, 2-way mirrors are not safe, you'd need 3-way mirrors to have a fair chance of resilvering after losing one drive—the issue is the risk of Unrecoverable Read Error during resilver without redundancy.
Then indeed increasing capacity is less easy, but how much do you have to store?

 

[Simtech]

Definitively go for RAIDZ2 if you care about data safety. With 14 TB drives, 2-way mirrors are not safe, you'd need 3-way mirrors to have a fair chance of resilvering after losing one drive—the issue is the risk of Unrecoverable Read Error during resilver without redundancy.
Then indeed increasing capacity is less easy, but how much do you have to store?

I'm storing about 20TB, with raidZ2 I will have about 4TB free.

 

[Etorix]

Then your Z2 should rather be 5- or 6-wide. Do you have old HDDs which may allow creating the pool before you buy extra drives?

Safety check: The VMs are NOT hosted among these 20 TB in the main pool, are they?

 

[Simtech]

Then your Z2 should rather be 5- or 6-wide. Do you have old HDDs which may allow creating the pool before you buy extra drives?

Safety check: The VMs are NOT hosted among these 20 TB in the main pool, are they?

No, the Esxi VMs all run from a local M.2 drive. I might purchase and shuck a 14TB WD elements drive, do a full test and create a 5 drive RaidZ2 and not worry about upgrading storage for some time. I have spent most of the week playing with TrueNAS bare-metal but bhyve VMs don't work very well with various test labs I have running. I might test TrueNAS scale and see how that goes, I'm not too bothered about spending a day or so importing and configuring my storage pools again.

 

[Simtech]

I've decided to go down the router of RaidZ2, still running within an Esxi VM but swapping from i7 to Xeon workstation with ECC Memory. I can give Truenas VM more reserved memory/cores and passthrough my LSI card and dual-port Intel NIC.