SOLVED HOME USER: advice needed. How to distribute a bunch of HDDs?

[zamana]

Ok!

So I reached a point where I have a bunch of HDDs and I'm wondering what would be the best way to arrange them, both physically (where/which size) and logically (zfs-1,2,3, mirror vdevs etc).

That's my situation:

So, my challenge is: how to distribute this bunch of HDDs among my systems, so that I maximize space with good redundancy/security, and keeping at least 2 backups.

The MAIN SYSTEM has 10 slots, the BACKUP 1 has 6 and the BACKUP 2 has 4. Of course that the first approach is to allocate the larger disks in the smallest systems and vice-versa, but how?

Currently my RAW data size is around 22TB.

Thanks for any advices.
Regards.

 

[Patrick M. Hausen]

Are you going to use transaction heavy workloads or is this mostly data at rest?

The main system has got the most slots and more spindles/vdevs gives more IOPs so I would fill it completely with the 4 T drives. Then it depends on your workload if you go

• 2x 5 disk RAIDZ2
• 1x 10 disk RAIDZ3

The first will perform a bit better. Mirrors seem to be out of the game because of your storage needs. The 24 T the first option gives you might even be too little depending on what you mean by "raw data size" ... ZFS raw or used by files?

Kind regards,
Patrick

 

[Yorick]

Gig Ethernet? Type of data, and type of use?

What I'm getting at: Do you need IOPS at all? I, for example, do not, because backups and video files, video and music streaming (Plex): Not IOPS intensive.

Your options are a little limited. Backup 2, assuming it can be okay to have a risk of dying because backup, can't go beyond 24TB (4x8TB raidz1).

Backup 1 could have 28TB but that seems not that useful, so maybe 24TB (4x8TB raidz1) as well.

Which leaves main with 10x4TB raidz3, 32 28TB, or 2x5x4TB raidz2, 24TB.

All of those numbers are way too close to your raw 22TB for performance. With the exception of the 32 28TB, which at least doesn't fill your pool immediately.

You may consider using the 8TB for main storage and rethinking how backup will be done, if you see yourself having additional data in the next few years.

Edit: Please clarify. Your current data is 22TB or 22TiB? As in, does Windows show you 22TB? Because that would be TiB and that changes things above as to "what would fit". 22TiB is just a hair above 24TB, something like 24.2.

Edit2: Fixed my horrible math, and option "no backup just now": 8x8TB raidz2 for main, the other two slots open for whatever (special allocation?), so just under 40TiB useable (https://wintelguy.com/zfs-calc.pl). And then think about what you'll do for backup. 12TB shuckables are coming on sale now and then, that'd get you there in backup 1 with 6 drives raidz2.

Edit3: "use my 4TB for backup now and think outside the box" option: Literally outside the box. External disk shelf via HBA, then 2 x 7-wide raidz1 with the 4TB, just enough to back up the main. A little dangerous for resilver but hey, it's a backup. Use the backup 2 box for that. Then still pursue the 6x12TB raidz2 for backup 1, if you like.

That's my best idea for space and resilience right now, assuming "typical home use, IOPS are not an issue" setup.

 

[Patrick M. Hausen]

Which leaves main with 10x4TB raidz3, 32TB

28 T with my calculator

 

[Yorick]

28 T with my calculator

Math is hard durr!

 

[zamana]

Thanks Patrick and Yorick for your replies.

I dare to say that my concerns with performance are NONE. Seriously. This is a home system, I'm the only user, and all it has is my Plex data (movies, music and TV shows), and some personal files, but nothing that I can't download again or that are not backed up in cloud providers (Google, Microsoft, Dropbox etc).

Sorry for the confusion: currently, my allocated data size, as displayed by FreeNAS, is 16.41TiB. The pool total size is 21TiB.

In my tables I'm using the rounded numbers to make the math easy. No problems.

I guess that the BACKUP 2 is the easiest: there's nothing better than Z1, unless I take the risk to use it as a JBOD:

Regarding the MAIN SYSTEM, it's more complicated, specially if I try to accommodate the BACKUP 1 also.

If I put all 10x4TB in the MAIN SYSTEM, I'll be left with only 4x4TB disks. So I'll need to put the other 8TB somewhere. The problem is that this "break" the arrangements, in such a way that I'll need to use mirror vdevs or work with 2 pools in one system or another, or both.

Here are some scenarios that I'm considering:

By other hand, I don't need to backup my whole MAIN SYSTEM in both BACKUP systems. I could live with the backup splitted between them. For example: I could backup my Plex data in both BACKUP 1 and 2, and the non-Plex data in just one of them.

My data grow is very irregular. I guess it grows, let's say, 1TiB a year, maybe.

What do you think?

 

[Patrick M. Hausen]

I would go Z3 with the main system and 4x 8 Z1 for both backups and not worry about the "wasted" slots.

 

[Yorick]

Thank you for clarifying.

So with 16TiB now, you can reasonably do 10-wide raidz2 with 4TB drives in main, giving you 26.7 TiB or so usable, or do raidz3 for more resilience in main, for 23.4 TiB usable. Given that you want to stay within 80%, that's 2-3 years of growth for raidz3 and 4-5 years of growth for raidz2.

Then do 4x8TB raidz1 plus 2x4TB mirror in backup 1, and 4x8TB raidz1 in backup 2. That's 20.4 TiB in backup 2 and 23.9 TiB in backup 1.

I still like "more room for growth" with all 8TB for main in raidz2, all 4TB for backup in an external shelf, and second backup TBD. And, you know your requirements and need for growth best.

 

[zamana]

Thanks Patrick and Yorick for your replies.

Both proposals are feasible. Now it's a matter of choice from my part.

Thank you very much!