Is it possible to fully use Mirror instead of RAIDZ?

[molay]

Hi everyone, thanks for the previous answers.

I have created 2 pools on the R730XD, and the pool1 is 6x16TB RAIDZ2, the pool2 is 2x 3x16TB 3-way Mirror.

System: DELL R730XD
CPU: Intel E5-2680V4 x2
Memory: Samsung 32G DDR4 2400MHz REG ECC x8
Network: Dual 10G SFP+
Boot Device: Samsung PM893 480G SSD SATA (rear 2.5 bay) (The R730XD cannot be booted from disks other than SATA/SAS unless it is booted from the Clover USB device)
SLOG Device: Intel Optane 900P 280G (I don't know how to configure, and now the device is only given to pool2)

No important data is stored yet, just testing (I focused on testing 3-way Mirror).

Now I have a few questions:
0. Is 3-way Mirror safe enough?
1. In the 3-way Mirror pool, I found that adding N 3-way Mirror vdevs will increase the available capacity of the pool by N times. Do these vdevs form stripes?
2. Is this a recommended practice?
3. Does doing so increase the security risk of data loss?
4. What if it is replaced by 2-way Mirror pool?
5. When 2-way Mirror/3-way Mirror vdev loses one hard disk, is the reconstruction time of adding a new hard disk much faster than RAIDZ?

Excluding the issue of space utilization, is it recommended to delete pool1 based on RAIDZ2 and instead use all 12 hard disks to form a 2-way Mirror/3-way Mirror pool (6x 2-way Mirror vdevs or 4x 3-way Mirror vdevs)? Or is it recommended to put block storage, which is frequently modified, in a separate pool?

 

[NugentS]

Whats mirror 3 - 3 way mirror perhaps?

0. Yes - it is quite safe (never 100%) as its has 2 additional copies of data - so you can lose 2 disks per vdev. 1/3 useable capacity though
1. Yes
2. Yes
3. No, not really. I suppose you have to take into account that losing a vdev loses the pool, so multiple vdevs gives more chances to lose a pool overall - a bit
4. Then each vdev will only have 1 spare copy of data, so can lose 1 disk per vdev. 50% useable capacity though
5 Yes, potentially much. ZFS only rebuilds actual data, so if the vdev is empty then rebuild time will be quick with mirrors or Z1/2/3. However when the vdevs have data then mirrors are potentially much quicker to rebuild than RAIDZ and less stressful to the whole system

Your last question is hard to answer, it depends on use case. However block devices should be on mirrors rather than RAIDZ (and preferably SSD as well)
Mirrors have high IOPS, RAIDZ has lower IOPS. Block storage needs IOPS. Block storage that is frequently modified needs high IOPS and low latency (as with good SSD's). Also consider a SLOG for safety (assuming sync writes)

See The path to success for block storage

 

[Davvo]

I strongly suggests you to read the following resource.

ZFS Storage Pool Layout

This amazing document, created by iXsystems in February 2022 as a "White Paper", cleanly explains how to qualify pool performance touching briefly on how ZFS stores data and presents the advantages, performance and disadvantages of each pool...

www.truenas.com

All vdevs in a pool are considered striped, being mirrors or RAIDZ: if you lose one vdev your pool goes kaputt.

 

[molay]

Thanks for all replies!

I tested the read and write operations of a large number of small files through SMB on a pool with 2x 3-way HDD Mirrors, and found that the performance was relatively poor.

I run a NodeJS scaffolding, which will generate hundreds of thousands of small JavaScript files in the "node_modules" folder according to the dependent library information.

Compared to my old MacBook Pro (Retina, 15-inch, Mid 2015) running the same scaffold on the built-in SSD, the progress on the 2x 3-way HDD Mirrors was much slower.

Is there any way for me to check if the hardware is running at its best? In other words, it can no longer be optimized faster under the current hardware environment.

PS, I am really excited to use TrueNAS for the first time. I want to do a lot of tests to judge which daily tasks are more suitable for 2x 3-way HDD Mirrors in the current hardware environment.

 

[Davvo]

Thanks for all replies!

I tested the read and write operations of a large number of small files through SMB on a pool with 2x 3-way HDD Mirrors, and found that the performance was relatively poor.

I run a NodeJS scaffolding, which will generate hundreds of thousands of small JavaScript files in the "node_modules" folder according to the dependent library information.

Compared to my old MacBook Pro (Retina, 15-inch, Mid 2015) running the same scaffold on the built-in SSD, the progress on the 2x 3-way HDD Mirrors was much slower.

Well, it's to be expected.

Show : As you have been told...

[...] block devices should be on mirrors rather than RAIDZ (and preferably SSD as well).
Mirrors have high IOPS, RAIDZ has lower IOPS. Block storage needs IOPS. Block storage that is frequently modified needs high IOPS and low latency (as with good SSD's). Also consider a SLOG for safety (assuming sync writes)

See The path to success for block storage

A standard HDD is hardly able to do 250 iops, an SSD easily reaches 10000 iops: the difference is ruthless, and small files lay it bare.

In other words, it can no longer be optimized faster under the current hardware environment.

PS, I am really excited to use TrueNAS for the first time. I want to do a lot of tests to judge which daily tasks are more suitable for 2x 3-way HDD Mirrors in the current hardware environment.

Well, there are a few things that can be done to get better performance.
Without adding any SSD, changing your pool configuration from 2 vdevs in 3-way mirror to 3 vdevs in 2-way mirror would reduce your resiliency a bit and help your IOPS; adding more RAM could improve your reads (depends on your use case).

If you want to achieve IOPS similar to that of a (single!) SSD without using one, you will likely need to double your drives.
There are a few other options (L2ARC, special vdev), but they aren't that convenient in your case. Imho if space is not of your concern slap 2 SSDs in a mirror and you will fly.

Also, do note that a SLOG only helps when you have syncwrties. It will not make your async writes faster.

Some insights into SLOG/ZIL with ZFS on FreeNAS

What is the ZIL? POSIX provides a facility for the system or an application to make sure that data requested to be written is actually committed to stable storage: a synchronous write request. Upon completion of a sync write request, the underlying filesystem is supposed to guarantee that a...

www.truenas.com

That being said if you want to measure your pool performance you can use either fio or irrc jgreco's solnet array.

Please read the resources that we link, they help a lot.

 

[molay]

Well, there are a few things that can be done to get better performance.
Without adding any SSD, changing your pool configuration from 2 vdevs in 3-way mirror to 3 vdevs in 2-way mirror would reduce your resiliency a bit and help your IOPS; adding more RAM could improve your reads (depends on your use case).

If you want to achieve IOPS similar to that of a (single!) SSD without using one, you will likely need to double your drives.
There are a few other options (L2ARC, special vdev), but they aren't that convenient in your case. Imho if space is not of your concern slap 2 SSDs in a mirror and you will fly.

Also, do note that a SLOG only helps when you have syncwrties. It will not make your async writes faster.

Thanks!
I already have 256GB (32GBx8) memory, and configured the entire Intel Optane 900P as a SLOG device for the pool of 2x 3-way Mirrors.
I checked the system status and found that the ARC Hit is basically above 96%, and the memory is only used less than 12GB.
Does this mean the hardware has reached its speed limit?
Is there any script or command that can list specific data for diagnosis?

About Mirror I have a new question: The 2x 3-way Mirrors are safer than the 3x 2-way Mirrors, right?
If I change the pool from 2x 3-way Mirror to 3x 2-way Mirror, will the pool loss risk increase significantly?

Due to the limited slot, I can only set up 1x 2-way SSD Mirror through PCIe riser card.
And I have to create separate pool to contain these two SSDs to avoid being slowed down by the HDDs.
Will the risk of losing this pool be greater?

 

[NugentS]

Do you have backups / a good backup regime? I know you have an offline backup and a cloud backup, but is the backup regular.
If the backup is good then I would suggest switching to 2 way mirrors
However for your use case a pool of SSD's (mirrored) would be a better idea - depending on size requirements - for the NodeJS stuff (whatever that means).

Can you go over your use case in more detail please?
We know about the NodeJS - but how much data is involved over time, are the files temporary or permanent. Whats your 3-5 year capacity requirement here?
What other use do you have, what capacity do you require, both now and in the future

The PCI Riser Cards (assuming a bifurcated slot) are very very simple and unlikley to fail, just a few traces on the board. The M.2 drives themselves, mirrored anyway.

 

[Davvo]

I already have 256GB (32GBx8) memory

Oopsie, missed the x8.

and configured the entire Intel Optane 900P as a SLOG device for the pool of 2x 3-way Mirrors.

So you do plan of usinc sync wrties, right? Just making sure you know what you are doing with that.

I checked the system status and found that the ARC Hit is basically above 96%, and the memory is only used less than 12GB.
Is there any script or command that can list specific data for diagnosis?

Regarding the arc use arc_summary, but if you don't have any data in the pool yet it's to be expected.

About Mirror I have a new question: The 2x 3-way Mirrors are safer than the 3x 2-way Mirrors, right?
If I change the pool from 2x 3-way Mirror to 3x 2-way Mirror, will the pool loss risk increase significantly?

First question: yes, 3-way mirrors can survive up to 2 disk deaths per vdev while 2-way only 1 per vdev.
Second question: you will go from withstanding up to 4 disk malfunctions (2 per vdev) to withstanding up to 3 disk malfunctions (but only a single disk per vdev); while I'm not good at probability, it does indeed increase the risk of losing your pool.

I will take a step back and ask you to explain a bit more what's your planned use.

 

[molay]

Can you go over your use case in more detail please?
We know about the NodeJS - but how much data is involved over time, are the files temporary or permanent. Whats your 3-5 year capacity requirement here?
What other use do you have, what capacity do you require, both now and in the future

I will take a step back and ask you to explain a bit more what's your planned use.

In my daily work, I need to use NodeJS (a JavaScript engine) for some front-end development, as well as Unity3D and Unreal for some rendering development.
They all produce a considerable number of small files, especially NodeJS (every medium-sized NodeJS project basically has hundreds of thousands or more small files).
And what's worse, if different projects depend on the same libraries, the files of the dependent libraries will be copied repeatedly under the folders of these projects.
These project files are estimated to only occupy up to 200GB of space per year, that is, 600-1000GB of data will be generated in 3-5 years, most of which are dependent library files.
However, when the project is no longer active, the files of dependent libraries can indeed be deleted (Storage usage will be reduced by 80-90%), but it will take time to re-download the dependent libraries when it is activated again.

The overall requirements for data storage are roughly as follows:
1. Access via SMB
1.1. Fixedly allocate 4TB to macOS TimeMachine for backup.
1.2. Up to 250GB of static media files (pictures, videos, audios, etc.) can be added each year.
1.3. Up to 200GB of project data containing a large number of small files can be added each year.
2. Access via block storage (iSCSI)
2.1. Fixedly allocate 2TB to the Proxmox VE server as data disks for virtual machines. (The system disks of vms use the local SSD on the Proxmox VE server, and the number of vms running at the same time does not exceed 5)
3. Not yet clear
3.1. Up to 50GB of database files can be added each year. (I'm still considering whether to run the database service with TrueNAS Scale Docker to access the local storage directly, or run the database service vm on the Proxmox VE server to access the block storage.)

Should I create two separate 2x 3-way Mirrors pools?
One for macOS TimeMachine backup and static media files, one for block storage and project data containing a large number of small files?
Or can I just use one 4x 3-way Mirror pool to store all the data?

Do you have backups / a good backup regime? I know you have an offline backup and a cloud backup, but is the backup regular.
If the backup is good then I would suggest switching to 2 way mirrors

I regularly connect the MacBookPro to the Drobo5D3 (it is a DAS device), and run a custom script to back up the important data from the TrueNAS to it.
I will also run custom script regularly on TrueNAS server to encrypt and synchronize important data to the cloud disk.
These are my naive understanding of the data backup 3-2-1 strategy.
In fact, I am not sure whether these regular backup methods are really safe.
If 2-way Mirror does have a relatively big risk, I would rather waste one more hard drive.

So you do plan of usinc sync wrties, right? Just making sure you know what you are doing with that.

I also don't know if my SMB and block storage are properly configured for synchronous writing :p, and I may need to read related documents.

 

[Davvo]

Can you quantify as much as you can the performance that you want achieve? Especially the small files.
The resource that @NugentS linked is vital for a succesfull completition of your plans, make sure to read it and its branches.

Honestly this is a bit beyond my comfort zone so take everything I say with a grain of salt:
you likely want 2 different pools: one dedicated to iSCSI (definitly mirrors) and one for the SMB share (this could be RAIDZ depending on your performance needs); if I understood the part about small files right you might want to take a look at deduplication (and ways to make it better so L2ARC and special vdevs).

I think you definitly need that SLOG.

 

[fcorbelli]

Hi everyone, thanks for the previous answers.

I have created 2 pools on the R730XD, and the pool1 is 6x16TB RAIDZ2, the pool2 is 2x 3x16TB 3-way Mirror.
(...)

Now I have a few questions:
0. Is 3-way Mirror safe enough?

"Too safe" (aka: waste-of-space, or better inefficient). Use the "third" for internal backups (and a kind of "hot spare", just in case)

1. In the 3-way Mirror pool, I found that adding N 3-way Mirror vdevs will increase the available capacity of the pool by N times. Do these vdevs form stripes?

If you have 12 x 16TB HDD drives, I will prefer 5 mirrors (2 drives each), and two single internal-backup

2. Is this a recommended practice?

They are points of view

3. Does doing so increase the security risk of data loss?

Personally I would use space for internal backups, even with higher snapshot depth

4. What if it is replaced by 2-way Mirror pool?

With TrueNAS it's hard to tell

5. When 2-way Mirror/3-way Mirror vdev loses one hard disk, is the reconstruction time of adding a new hard disk much faster than RAIDZ?

It depends on the version of zfs you use (assuming you're using TrueNAS it's now OpenZFS). The time essentially depends on how much data there is
If, absurdly, they are empty, it is almost immediate

Excluding the issue of space utilization, is it recommended to delete pool1 based on RAIDZ2 and instead use all 12 hard disks to form a 2-way Mirror/3-way Mirror pool (6x 2-way Mirror vdevs or 4x 3-way Mirror vdevs)? Or is it recommended to put block storage, which is frequently modified, in a separate pool?

If I were you I would logically separate the pairs (all mirrors, unrelated to each other, no RAIDz) by functions

I tested the read and write operations of a large number of small files through SMB on a pool with 2x 3-way HDD Mirrors, and found that the performance was relatively poor.

I run a NodeJS scaffolding, which will generate hundreds of thousands of small JavaScript files in the "node_modules" folder according to the dependent library information.

The reasons are varied. Obviously the most important is the use of spinning disks instead of solid state.
There are others, obvious, but I don't think it's appropriate to write about them

In fact, I am not sure whether these regular backup methods are really safe.

I could explain as advanced a backup strategy as you like, but then I'd have to kill you (it's a joke of course)

In fact, if you don't have the option of using solid state drives, I'd share a large ramdisk in your case
Yes, a ramdisk (of course assuming you're willing to take a risk regarding power shortage)
Otherwise, the performance you'd get from a $500 QNAP with a couple of consumer-grade SSDs will be significantly higher

I'd also do different things, but they're not very tolerated on the TrueNAS forum (i.e. I'd use the only "real" SSD, the system one, to store up to 400GB of small files, with an internal replica, two internal backup). Of course no redundancy, only backup every 15 minutes

 

[NugentS]

[For discussion[
I am seeing the following.

1. Access via SMB
1.1. Fixedly allocate 4TB to macOS TimeMachine for backup.
1.2. Up to 250GB of static media files (pictures, videos, audios, etc.) can be added each year.
1.3. Up to 200GB of project data containing a large number of small files can be added each year.
2. Access via block storage (iSCSI)
2.1. Fixedly allocate 2TB to the Proxmox VE server as data disks for virtual machines. (The system disks of vms use the local SSD on the Proxmox VE server, and the number of vms running at the same time does not exceed 5)
3. Not yet clear
3.1. Up to 50GB of database files can be added each year. (I'm still considering whether to run the database service with TrueNAS Scale Docker to access the local storage directly, or run the database service vm on the Proxmox VE server to access the block storage.)

1.1 & 1.2 - This fits a RaidZn pool. Bulk data
[This is the Bulk of the storage - wants to be about 10+TB]

1.3 - based on the description above this fits a mirrored SSD setup, say 2TB
[2*2TB SSD, Mirrored}

2, 2.1 - Probably fits HDD Mirrors with a SLOG (maybe). Unsure of what this data is, what it does and what you want from it

3, 3.1 - Again a mirror of SSD's as its database.
[Same mirror as 1.3 - so will need to be bigger, maybe vdev?]]

How many drives (and of what kind) can a/this specific DELL R730XD fit?

Thoughts?
Also note use of MAC as a workstation - I think a MAC uses sync writes by default (even for SMB) so either use a SLOG or set sync=disabled

 

[Davvo]

Thoughts?

I was reasoning along the same lines, maybe fitting the small files into the RAIDZ with a few tricks.

 

[NugentS]

A large special vdev on the RAIDZ? Would need testing as it can't be removed without trashing the pool.

BTW for resilver times on a mirror. I have a 45ish percent full pool of 12TB HDDs that took a few hours to resilver rather than the days I would expect a RAIDZ to take.

 

[Ericloewe]

"Too safe" (aka: waste-of-space, or better inefficient). Use the "third" for internal backups (and a kind of "hot spare", just in case)

That's a narrow interpretation.

Three-way mirrors are indeed unusual because their space efficiency is very low. They are, however, very reliable (three copies) and very fast (lots of vdevs) - though at the cost of extra disks.

Are they worth it? That's for everyone's individual risk analysis to decide. External factors also play into it: For instance, when running Gluster, you absolutely need three-way mirrors at the Gluster level (tl;dr - arbiters are a joke and split-brain situations are almost guaranteed without a third mirror) - at that point, for the storage backing Gluster, it's probably safe to say that two-way mirrors are good enough for most. Remote site with no trusted local hands? Three-way mirrors start to make great economic sense,

The "internal backups" thing is a bit dangerous. Many things likely to kill a pool outright won't stop at a single pool and easily wreck whatever else is on the server. Using the disks as a backup in a different server? Totally valid.

 

[fcorbelli]

Are they worth it? That's for everyone's individual risk analysis to decide. External factors also play into it: For instance, when running Gluster, you absolutely need three-way mirrors at the Gluster level (tl;dr - arbiters are a joke and split-brain situations are almost guaranteed without a third mirror)

It would be interesting to know the technical reasons for this statement.
If they exist

- at that point, it's probably safe to say that two-way mirrors are good enough for most. Remote site with no trusted local hands? Three-way mirrors start to make great economic sense,

It's an OT hypothesis, different from the request of the thread

The "internal backups" thing is a bit dangerous. Many things likely to kill a pool outright won't stop at a single pool and easily wreck whatever else is on the server. Using the disks as a backup in a different server? Totally valid.

Here too it would be interesting to know something more on these "things"

For a non-amateur, non-SOHO server, in short, with important data, having at least (the bare minimum) one internal backup's drive, preferably two, is essential for any non-amateur level of safety.

And if the volumes are essentially empty (aka: a lot of "useless/unused" drives, due to The overall requirements for data storage are roughly as follows:), it's wise to dedicate at least two (in deduplicated stripe) exclusively for restoring tests, concentrating (write) wear away from the main volumes, gaining a bit of bandwidth
It doesn't make much sense, in general and in this case, to have an "overall" space X almost all unused

This is the explanation of making a large ramdisk (on a 256GB machine), and using the system SSD (to store small files,sadly without redundancy) edit: and the Optane too, of course : get the most out of the existing setup.

 

[Ericloewe]

It would be interesting to know the technical reasons for this statement.

1. Arbiters only work with a single mirror pair. Want to scale to two mirrors? No arbiter for you.
2. The slightest interruption of networking has a very good chance of causing a split brain situation if you only have two copies.

Here too it would be interesting to know something more on these "things"

Non-exhaustive list of things that will likely kill n pools if they kill a single pool:

• Fire, floods, etc.
• Massive kernel or ZFS bugs
• Power surges
• Overheating disks
• Defective HBAs
• Theft, vandalism, etc.

Apart from contained, but extremely serious bugs in ZFS (and I don't think I've ever heard of one that would fit this scenario), there's nothing I can think of that would politely respect pool boundaries.

For a non-amateur, non-SOHO server, in short, with important data, having at least (the bare minimum) one internal backup's drive, preferably two, is essential for any non-amateur level of safety.

Fundamentally disagree. If they're (permanently) connected to the same machine, it's not a backup for the reasons listed above.

And if the volumes are essentially empty (aka: a lot of "useless/unused" drives, due to The overall requirements for data storage are roughly as follows:), it's wise to dedicate at least two (in deduplicated stripe) exclusively for restoring tests, concentrating (write) wear away from the main volumes, gaining a bit of bandwidth
It doesn't make much sense, in general and in this case, to have an "overall" space X almost all unused

I disagree with several parts of this statement.

• Free space is good for performance, as it makes it easier/faster for the allocator to do its job.
• Deduplicated stripe? For maximum unreliability combined with minimum performance? Dedup is, plainly, not a good feature in practice. It is far too slow and resource-hungry to work outside of very controlled workloads - which do exist, but not everywhere.
• A dedicated scratch pool might make sense in some cases, but "restoring tests" is a bizarre reason for such a pool, since a meaningful test would require at least as much space as the data to be restored. Reducing write wear and freeing up IOPS/bandwidth is a valid benefit, in some cases, but would come with the additional maintenance burden of a separate pool, meaning that it's not something to be recommended just because.

 

[NugentS]

[Cough} Thread Scope Creep

 

[jgreco]

[Cough} Thread Scope Creep

Thank you @NugentS .

[mod hat on] This thread is deemed to have turned uselessly argumentative, pointless, and off-topic to the intended purpose of this forum. If you wish to discuss non-ZFS topics or general computer science stuff such as "deduplicated stripes" or some of the other weird stuff that has come up, this needs to happen in the Off-Topic forum, where you are welcome to discuss everything from random CS topics to your favorite cookie recipes.

 

[molay]

[mod hat on] This thread is deemed uselessly argumentative, pointless, and off-topic to the intended purpose of this forum. If you wish to discuss non-ZFS topics or general computer science stuff such as "deduplicated stripes" or some of the other weird stuff that has come up, this needs to happen in the Off-Topic forum, where you are welcome to discuss everything from random CS topics to your favorite cookie recipes.

I'm sorry that my question may be meaningless debate, or offensive to the forum.
All the people who responded to me were enthusiastic and professional, which I appreciate.
I'm just trying to determine what form of pool configuration is best for my needs given my current hardware.
If you need me to modify the question (title and content), please also point out that I will modify it according to the rules of the forum.