Please, advice about configuring 9x4TB HDs for best speed & redundancy

[vicmarto]

Hello!

Please, need some advice about the best way to configure 9x4TB HDs for my needs. My use of the pool would be:

- I'm working directly with the files (shared with NFS), so really need the best possible random RW speed. The server and the (unique) client are connected by 10GBe
- My storage requirements are only of about 10TB



So, I'm thinking this:
# zpool create pool /dev/ada0 /dev/ada1 /dev/ada2 /dev/ada3
plus:
# zpool create pool-backup raidz1 /dev/ada4 /dev/ada5 /dev/ada6 /dev/ada7 /dev/ada8

The idea is to work directly with the files in 'pool', and configure 'pool-backup' in FreeNAS as a replication of 'pool', with a frequent snapshot policy (commented at the end).


Others (more simple) alternatives would be, always looking for the best random RW speed, maybe:
# zpool create pool mirror /dev/ada0 /dev/ada1 mirror /dev/ada2 /dev/ada3 mirror /dev/ada4 /dev/ada5 mirror /dev/ada6 /dev/ada7
or: # zpool create pool raidz1 /dev/ada0 /dev/ada1 /dev/ada2 raidz1 /dev/ada3 /dev/ada4 /dev/ada5 raidz1 /dev/ada6 /dev/ada7 /dev/ada8


For backups, I'm thinking of NOT to use an external backup system and rely entirely in snapshots. In reallity, in a very agressive periodic snapshot policy, very frequent. Something like:

- Periodic snapshot 'pool' every 5 minutes, with 2 day lifetime
- Periodic snapshot 'pool' every 1 day, with 15 day lifetime
- Periodic snapshot 'pool' every 1 week, with 2 month lifetime


What do you think? Is this reasonable? Please, comments, advices, THANKS!!

 

[kdragon75]

really need the best possible random RW speed

# zpool create pool mirror /dev/ada0 /dev/ada1 mirror /dev/ada2 /dev/ada3 mirror /dev/ada4 /dev/ada5 mirror /dev/ada6 /dev/ada7

Except you should not be making the pool using adaX identifiers, you should be using the gptid. Of better yet, USE THE UI. The UI will automaticky partition the disks, add the recommended 2GB swap, and build the pool with the gptid's and not confuse the UI about the state of the ZFS mount,

For backups, I'm thinking of NOT to use an external backup system and rely entirely in snapshots.

Then your not backing up PERIOD.

 

[Chris Moore]

best way to configure 9x4TB HDs for my needs

For the pool of mirrors, you need an even number of drives, so perhaps you can come up with a 10th? It would be good if you shared hardware details though, there is an example in this post: https://forums.freenas.org/index.php?threads/updated-forum-rules-8-3-16.45124/

Others (more simple) alternatives would be, always looking for the best random RW speed, maybe:

Are you making your suggestions in this format simply as a way to communicate your idea? You should not be using the command line to create your pool, you should use the GUI and a pool of mirrors is your best performance option.

What do you think? Is this reasonable? Please, comments, advices, THANKS!!

Making snapshots gives you the ability to recover from accidental changes such as deletions or to go back to an earlier state if you change your mind, but it does not protect you from a catastrophic failure, so you still need some form of backup, even if it is to a single drive or a cloud storage provider. Something, that will safeguard your data against the possibility of some catastrophe that we can't imagine, like a flood or earthquake, that might swallow your server whole. Safety first.

 

[vicmarto]

Yes, I'm using this command line examples to only show the idea. The pool would be created using the GUI, of course.

And... Please, confirm: Are you saying mirrors the disk are more quick than striping its?

 

[kdragon75]

Yes, I'm using this command line examples to only show the idea. The pool would be created using the GUI, of course.

And... Please, confirm: Are you saying mirrors the disk are more quick than striping its?

No, a stripe would be faster but you said you wanted redundancy. Id you went with your first idea, you would have NO redundancy on your main pool but your would have a *sort of* bakup with the replication to the other pool.

What is your workload *EXACTLY*? You say it's random IO but don't specify if its small IO or random large IO or if it's 50/50 read/write or 90.10 etc... We may make other recommendations that could have drastic impacts on performance depending on that information.

 

[Chris Moore]

And... Please, confirm: Are you saying mirrors the disk are more quick than striping its?

If you put multiple mirror sets into a pool, it is a stripe of mirrors, kind of like RIAD-10, but the terminology in ZFS is a pool of mirrored vdevs. You appear to need to review the guides with regard to terminology and best practices:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

If you want fast random IO, the pool of mirrored vdevs is the way to go.

 

[vicmarto]

Thanks all for your kindly help, really appreciate it.

No, a stripe would be faster but you said you wanted redundancy. Id you went with your first idea, you would have NO redundancy on your main pool but your would have a *sort of* bakup with the replication to the other pool.

Yes, this is the idea in the first case. To enjoy the best speed offered by the stripe, and get some sort of redundancy, replicating the pool to another raidz1 pool.

What is your workload *EXACTLY*? You say it's random IO but don't specify if its small IO or random large IO or if it's 50/50 read/write or 90.10 etc... We may make other recommendations that could have drastic impacts on performance depending on that information.

Well, that would be really difficult to know. I can't give you exact numbers (how can this be done?!). I only can say this is a home server with the typical files used. Not only big files like movies, but also directories with thousand of small files. Well, 50/50?

If you put multiple mirror sets into a pool, it is a stripe of mirrors, kind of like RIAD-10, but the terminology in ZFS is a pool of mirrored vdevs.

Thanks Chris, I was really little careful to use the correct terminology. Luckily you could understand me correctly :)

If you want fast random IO, the pool of mirrored vdevs is the way to go.

Your vote is then to the pool of mirrored 4 vdevs? (with only 8xHD, In this case I will use the 9th HD to another task). But, why do you prefer this to my first idea? Which disadvantages or problem do you see in it? THANKS!!

 

[Chris Moore]

But, why do you prefer this to my first idea? Which disadvantages or problem do you see in it? THANKS!!

The very first option you suggested, a stripe of single disks, has no redundancy. It should be similar in performance with four physical disks instead of four vdevs, (vdev = virtual device, similar but not equal to a disk) but in a single disk there is no redundancy. This does not simply cause you to loose the pool if a drive fails, this also costs you the ability of ZFS to error correct. If there was a data error, ZFS would not have a second copy of the data to recover from. ZFS would be able to compare the checksum and tell you that a data error had occurred, but the value that ZFS uses to correct such an error comes from the redundant disks. No redundancy is bad, simple as that. There are also some performance advantages on read from a mirror vdev as opposed to reading from a single disk. The information I am providing is available in the documentation. Why have you not reviewed it yet.

 

[vicmarto]

Yes Chris, thanks for your response. I know a stripe has no redundancy, and this is the reason to do a 5 minute replication to a raidz1 pool. I though would be a good compromise: to benefit from the high speed of a stripe of single disks, and get redundancy from a (slow) raidz1 pool.

If a drive fails in the stripe of single disks, there is no problem to replace it (after all this is a home file server, not a big server in a big company), and send/receive from the raidz1 pool to the (new) stripe, don't is?

Which other problem may I facing with this setup? (one big stripe of single disks, replicated (each 5 minutes) to a raidz1 pool)

About your suggestion of offside backup, your are absolutely right, I plan to setup a rclone (encrypted) sync to some cloud storage provider.

 

[Chris Moore]

About your suggestion of offside backup, your are absolutely right, I plan to setup a rclone (encrypted) sync to some cloud storage provider.

I did not suggest that. Not part of anything I have said.

Which other problem may I facing with this setup? (one big stripe of single disks, replicated (each 5 minutes) to a raidz1 pool)

I already answered that. Read what I wrote until you understand.

 

[kdragon75]

Just build it the way you want. The first time a disk fails, you will rebuild with a RAIDZ pool.

I don't think you want advise at this point. You just want someone to say your plan is good...

Your plan is good and will be the best way to setup your pool for lots of extra work and down time.

 

[Chris Moore]

to benefit from the high speed of a stripe of single disks, and get redundancy from a (slow) raidz1 pool.

I think it is a language (or terminology) problem. A second pool of disks could be a backup, but it is not redundancy. Redundancy works to prevent down time, a backup allows you to recover from a failure. In ZFS, the file system of FreeNAS, if you have redundancy inside a pool, ZFS can detect and correct errors in the data from something known as 'bit rot'. If your pool has no redundancy, ZFS can detect errors, but can not correct them. Having a second pool with a backup copy of the data is not redundancy, as I said, it is a form of backup.
You need to review the guides and improve your understanding. You are missing the point with regard to why we even use ZFS.

 

[Evertb1]

Having a second pool with a backup copy of the data is not redundancy, as I said, it is a form of backup.

I would say it's a very poor form of backup. A "backup" not only in te same room but even in the same chassis as the original data? Bad idea, giving a false sense of security. Still Chris, I admire your patience.