Pick the right pool for the job

[Black Ninja]

My Freenas has 16 bays and 16 HGST 4TB coolspin HHDs. It will be used for storage of movies mostly and RaidZ2 will work best from what I understand. Now the question is this:

Should I make 2 separate Raidz2 pools - one with 10 Drives and one with 6 drives
or
Should I make 1 pool containing 2xRaidzs vdevs with 8 drives each.

I know in both cases I will have the same amount of space (12 hdd with data and 4 for parity) and I read that 2 devs in same pool will somewhat give me more IOPS, But..

What other consideration I should consider in order to take one over the other ?

Thanks in advance to anybody who can provide advice.

 

[Bidule0hm]

If you can balance the number of drives then do it (AFAIK it's better for performance to have identical vdevs inside a same pool) ;) and it's more elegant :P

 

[Black Ninja]

If you can balance the number of drives then do it (AFAIK it's better for performance to have identical vdevs inside a same pool) ;) and it's more elegant :p

In the first option I meant to have 2 separate pools (10xRaidz2 and 6xRaidz2) and I can put data , scrub, and etc, in separate smaller pools.

And second option was 1 single pool made out of 2 vdev ( each vdev is 8HDD in Raidz2)

 

[Bidule0hm]

Oh sorry, didn't pay enough attention... :p

So, why you want two pools? usually the space is better managed if you have one pool and multiple datasets instead of using multiple pools to sort the data. I'd say that if you don't have a good reason to use multiple pools then don't do that (a bit like L2ARC...) :)

 

[Black Ninja]

I am not sure which way is better that's why I am asking to discuss it. I don't know if 2 smaller pools will be easier on freenas to manage, scrub, fragmentation , etc.

It will be the hardware in my signature in both cases and it will be made with all 16 drives. I am trying to find the pros and cons of both configurations.
Free space obviously is identical but I don't know about performance and weight putting on the freenas system ?!

How would you do it , and why ?

 

[Bidule0hm]

I'd chose one pool because it'll be easier to manage, probably less fragmented, have better performance (IO/s), space will be used more efficiently (what happens if you want to put more data on one of the pools which is nearly full when at the same time there's space left on the other one? with one pool you'll not have this problem) ;)

But in the end it's your server and your needs, we can't chose for you :)

 

[Black Ninja]

Fragmentation I think will be less in one pool you can put data that is change often( working sets) and in the other could be the one you don't usually change often, but you point for the one side full other not - I see the point there.

1. From what I read Raidz2 should be made out of 4,6,10 disks in vdev, with 16 drives I have to make 2 Raidz2 vdevs with 8 disks in each. Isn't it 8 drives in raidz2 vdev "not optimal" configuration ?

2.I also wonder if the stress on the freenas system is the same with 2 smaller raidz2 is separate pools , or if they are both striped under one pool. I can't put more than 32GB ram and I want to configure it optimal for the resources I have , because it will have 64 TB data.

"But in the end" like you said, I can't chose unless I know how everything in back of freenas and freebsd works, so I'll have to take somebody who knows word for it.

 

[Bidule0hm]

1. It's a recommendation for perfs who doesn't apply if you use compression (and it's now the default so you probably use it) but it still apply for space overhead. However you don't lose much if you don't have the "optimal" number of drives so no worries here ;)

2. I'd say the ARC will probably be used more efficiently with one pool than two but it's pure intuition and logical deduction so wait for a more experienced member answer :)

 

[L]

Over and over, when doing performance tuning I get about 50MB/sec per vdev/raidset. If you have 2 you will typically get about 100MB/sec. More vdevs better performance.

ZFS is designed to use one big pool. The only time I will use 2 is if I have dramatically different disk types.. like 10k sas and 5600 sata or a full flash pool.

You can't calculate optimal raid sets. Compression and dynamic block sizes make it difficult. Compressed transactions are unknown values. Read this http://blog.delphix.com/matt/2014/06/ Matt is co-inventor of zfs.

 

[Black Ninja]

Over and over, when doing performance tuning I get about 50MB/sec per vdev/raidset. If you have 2 you will typically get about 100MB/sec. More vdevs better performance.

ZFS is designed to use one big pool. The only time I will use 2 is if I have dramatically different disk types.. like 10k sas and 5600 sata or a full flash pool.

You can't calculate optimal raid sets. Compression and dynamic block sizes make it difficult. Compressed transactions are unknown values. Read this http://blog.delphix.com/matt/2014/06/ Matt is co-inventor of zfs.

Thanks Linda. I'll read the link you send me to understand this better , but I am assuming you are trying to tell me that it shouldn't matter anymore if raidz2 vdev is 4,6, 8 or 10 disks since we have a compression on ?

For the drives: Right now I have 10 drives different size and speed from other 6 ( which were much older drives) and that was one on the reasons to set 2 pools I had in mind, but I will replace these 6 drives and I will have 16 identical 4TB HGST Coolspin drives which I am trying to configure optimal way based on my system resources.

I have 32GB ram and the pool will be 16x4TB=64TB. It's better for me have one pool with 2 vdevs(raidz2) inside like you said, but I wonder if that make it harder on my freenas to work twice more hard to process data in 2 vdevs(raidz2) at the same time when they are part of the same pool ? I hope I explained my concern correct, because I really want to know what you think about it ?

 

[Bidule0hm]

it shouldn't matter anymore if raidz2 vdev is 4,6, 8 or 10 disks since we have a compression on ?

Yep, exactly what I said too ;)

Right now I have 10 drives different size and speed from other 6 ( which were much older drives) and that was one on the reasons to set 2 pools I had in mind, but I will replace these 6 drives and I will have 16 identical 4TB HGST Coolspin drives which I am trying to configure optimal way based on my system resources.

If you plan on having the same drives then I'd say don't chose two pools just because you actually have different drives (unless the space isn't sufficient with the 4 TB drives capped at 2 TB by the others 2 TB drives).

32 GB for 64 TB is a bit low but we know that the 1 GB for 1 TB rule is less critical the bigger the size.

 

[joeschmuck]

This is a tricky one, sort of. Because of your intended use you don't need any super fast drive system. Because you have two different hard drive sizes gives you a different issue to deal with. If you have plans to replace your 2TB drives with 4TB drives in the future and you are fine with a total pool size of 25TB until you expand your pool, then a single pool made of all the 16 drives would be fine, however because you do have so many drives, maybe a RAIDZ3 setup would be better (from a repair perspective) and you would lose 2TB for the time being.

Just a thought and I've never put 16 drives into a single pool before.

 

[L]

Thanks Linda. I'll read the link you send me to understand this better , but I am assuming you are trying to tell me that it shouldn't matter anymore if raidz2 vdev is 4,6, 8 or 10 disks since we have a compression on ?

For the drives: Right now I have 10 drives different size and speed from other 6 ( which were much older drives) and that was one on the reasons to set 2 pools I had in mind, but I will replace these 6 drives and I will have 16 identical 4TB HGST Coolspin drives which I am trying to configure optimal way based on my system resources.

I have 32GB ram and the pool will be 16x4TB=64TB. It's better for me have one pool with 2 vdevs(raidz2) inside like you said, but I wonder if that make it harder on my freenas to work twice more hard to process data in 2 vdevs(raidz2) at the same time when they are part of the same pool ? I hope I explained my concern correct, because I really want to know what you think about it ?

So yes, there is no magic number of disks in a vdev.

You can run with different sizes and speeds. The only reason I will do it is if there are business/workload reason.

Without going into a ton of internals, more vdevs actually make it easier on zfs, more smaller metaslabs, only until near full, then it might get a little harder to find free space.. You want your system to work hard, that's what it's there for :)

 

[L]

This is a tricky one, sort of. Because of your intended use you don't need any super fast drive system. Because you have two different hard drive sizes gives you a different issue to deal with. If you have plans to replace your 2TB drives with 4TB drives in the future and you are fine with a total pool size of 25TB until you expand your pool, then a single pool made of all the 16 drives would be fine, however because you do have so many drives, maybe a RAIDZ3 setup would be better (from a repair perspective) and you would lose 2TB for the time being.

Just a thought and I've never put 16 drives into a single pool before.

This also reminds me that the bigger the vdev(more drives) the harder and longer it takes to resilver at drive failure. IHAC recently with 7+1 raidz 4TB vdevs where resilver took 83 hours(on zol). It is non-trivial. The system will remain up and operational during resilver, and freenas/freebsd is much better at managing resilver times, but still.

 

[Black Ninja]

This is a tricky one, sort of. Because of your intended use you don't need any super fast drive system. Because you have two different hard drive sizes gives you a different issue to deal with. If you have plans to replace your 2TB drives with 4TB drives in the future and you are fine with a total pool size of 25TB until you expand your pool, then a single pool made of all the 16 drives would be fine, however because you do have so many drives, maybe a RAIDZ3 setup would be better (from a repair perspective) and you would lose 2TB for the time being.

Just a thought and I've never put 16 drives into a single pool before.

Most 3u chassis have 16 bays and I want to use them all.
Let's forget about different sizes so it won't get complicated, I will have very soon 16 identical 4TB HGST Coolspin drives. The question is how to set them best way for movies storage.

 

[Black Ninja]

So yes, there is no magic number of disks in a vdev.

You can run with different sizes and speeds. The only reason I will do it is if there are business/workload reason.

Without going into a ton of internals, more vdevs actually make it easier on zfs, more smaller metaslabs, only until near full, then it might get a little harder to find free space.. You want your system to work hard, that's what it's there for :)

That is very good to know.
I understood(on theory) how good is to have more vdevs in pool for speed and IOPS , but I wasn't sure if this applies for vdevs like raidz2 or raidz3 because of the overhead that raidz1,2,3 parity brings.
So I just assume(I hope I am wrong) that is better to have few wider spread raidz2 vdevs in a pool than to many smaller raidz2 vdevs that can overload the sytem by doing to much parity for each vdev ?

 

[Black Ninja]

This also reminds me that the bigger the vdev(more drives) the harder and longer it takes to resilver at drive failure. IHAC recently with 7+1 raidz 4TB vdevs where resilver took 83 hours(on zol). It is non-trivial. The system will remain up and operational during resilver, and freenas/freebsd is much better at managing resilver times, but still.

Pardon my ignorance but I assume IHAC is Institute for Human-Animal Connection. That is a very longtime to resilver. My current pool is 10x4TB HGST 5900RPM (RAIDZ2) is made out of 1 vdev if I remove a drive it will not take more that 9 hours to resilver. ( pool is 56% full).
I like so much how freenas resilver compared to raid controller( using max speed of drive and only resilver data) I almost want a drive to fail sometimes ))

 

[Black Ninja]

Yep, exactly what I said too ;)



If you plan on having the same drives then I'd say don't chose two pools just because you actually have different drives (unless the space isn't sufficient with the 4 TB drives capped at 2 TB by the others 2 TB drives).

32 GB for 64 TB is a bit low but we know that the 1 GB for 1 TB rule is less critical the bigger the size.

You said it but you didn't say if 512 and 4k sector drives matters for drive number selection ?:)

"32 GB for 64 TB is a bit low" to make this little better 4TB is actually 3.61TB usable space , so is 32GB for 57.56TB raw usable space.:p

 

[L]

Freenas defaults to 4k sectors... There are a bunch of reasons why this is good.. You do lose a little space for 4k sectors.

If i had 16 drives. I would put into 4 3+1 raid sets, for movies, maybe even 3 4+1 sets and a spare. It really depends. What annoys you the most?? I have my servers downstairs in a closet. I really don't like to have to go too often to replace disks. I like that i can use a spare now. Some people want as much storage as possible, then you might go 15+1. There isn't a best way. All ways have trade offs. Bigger sets, longer rebuild times lower perf, but more usable storage. Smaller sets more performance, shorter rebuild times, but less usable storage. You have to decide what is important to you.

 

[jgreco]

4 3+1 raid sets, for movies, maybe even 3 4+1 sets and a spare.

wtf is a "3+1 raid set"? Do you mean a 4 disk RAIDZ1 vdev? We do have technical terms that are good for avoiding ambiguity.

We no longer recommend the use of RAIDZ1, certainly not with 4TB drives, and absolutely not at

15+1

16 drives wide, if that's what Linda meant.