Pick the right pool for the job

[Bidule0hm]

You said it but you didn't say if 512 and 4k sector drives matters for drive number selection ?:)

Doesn't matter. Plus all actual drives are 4k and FreeNAS is 4k too anyway :)

"32 GB for 64 TB is a bit low" to make this little better 4TB is actually 3.61TB usable space , so is 32GB for 57.56TB raw usable space.:p

Nop, 4 TB is 3.61 TiB (not 3.61 TB). But the RAM works the same for GB and GiB, so in the end the ratio [pool size]:[RAM size] is the same whatever you chose.

@Linda Kateley Are you suggesting 4 or 3 RAID-Z1 vdevs? because with 4 TB drives I'd never chose RAID-Z1. I think with 16 drives the best you can come up with is 2x RAID-Z2 vdevs (or RAID-Z3 if you're paranoid).

 

[Black Ninja]

Freenas defaults to 4k sectors... There are a bunch of reasons why this is good.. You do lose a little space for 4k sectors.

I don't mind 4k sectors anymore. It is good that freenas defaults to 4k, but a lot of HDD's are 512 bytes and I don't know how bad is that for freenas , but we don't have a control on hdd side ?

If i had 16 drives. I would put into 4 3+1 raid sets, for movies, maybe even 3 4+1 sets and a spare. It really depends. What annoys you the most?? I have my servers downstairs in a closet. I really don't like to have to go too often to replace disks. I like that i can use a spare now. Some people want as much storage as possible, then you might go 15+1. There isn't a best way. All ways have trade offs. Bigger sets, longer rebuild times lower perf, but more usable storage. Smaller sets more performance, shorter rebuild times, but less usable storage. You have to decide what is important to you.

Thank you very much Linda for sharing how you would do it. When you said "4 3+1 raid sets", I assume you mean 4 vdevs(raidz1 with 3 drives) and hot spare one ,all that in one pool, right ?

In my case I think I narrow it down to 2 choices which will have the same amount of usable space and the same level of redundancy , I just need to pick one that is better for freenas base on my system resources, but I can't because I don't understand what happening underneath the surface. So what would you chose from:

1. One pool with 2 vdevs (each vdev is 8 HDD's in raidz2)

2. Two smaller pools: 1 pool with 1 vdev (10 HDD's in raidz2) and 1 pool with 1 vdev (6 HDD's in raidz2)

Will it make a difference for freenas ?

 

[Black Ninja]

Doesn't matter. Plus all actual drives are 4k and FreeNAS is 4k too anyway :)

I think you are wrong here. Example my 2TB are 512bytes, my 4Tb I am not sure but I think like many other hdd they are navite 4k, but controller emulates 512 for compatibility reasons so they end up as 512 for freenas , and some are 4k native , but not most.

Nop, 4 TB is 3.61 TiB (not 3.61 TB). But the RAM works the same for GB and GiB, so in the end the ratio [pool size]:[RAM size] is the same whatever you chose.

I incline to disparage , but for different reasons: In 4GB ram to 4TB ratio , you still get much less space on HDD side , while in ram you get pretty much all. You see what i mean: 4TB hdd is advertised space not actual , like 80GB Intel ssd is actual 74.x GB usable. With ram you get all , it's not a matter of conversion byte to KB to MB to TB is about the fact that some space of the HDD is not available for the consumer and that shrinks the actual space.

@Linda Kateley Are you suggesting 4 or 3 RAID-Z1 vdevs? because with 4 TB drives I'd never chose RAID-Z1. I think with 16 drives the best you can come up with is 2x RAID-Z2 vdevs (or RAID-Z3 if you're paranoid).

This is what I was incline to do - 2x RAID-Z2 vdevs in one pool, I am kind on paranoid for reliability , but I think raid z3 is more for the people who use non HGST drives , which I don't so it should be safe enough.:)

The question is which of the 2 option I am suggesting will have less stress on freenas , and will give me better performance naturally.

 

[Bidule0hm]

Even if your drives are 512B it's not an issue, 4k pool on 512B drives isn't a problem (but the reverse would be a problem).

the fact that some space of the HDD is not available for the consumer and that shrinks the actual space.

Mmh... nop. The difference comes from the TB/TiB difference. For example the datasheet of the Seagate ST3000VN000 (chosen just because I already have it) say that the guaranteed number of sectors is 5860533168. So 5860533168 * 512 = 3000592982016 bytes or 3000.6 GB. The drive is announced at 3000 GB so in fact you get just a bit more than that. Then if you do the conversion TB --> TiB you go from 3 TB to 2.73 TiB which is exactly the size you'll see in an OS ;)

 

[Black Ninja]

Even if your drives are 512B it's not an issue, 4k pool on 512B drives isn't a problem (but the reverse would be a problem).

I hope you are right, I just remember reading that number of drives in vdev matter if you don't use compression ( which I use) or if your drives are 512 byte sector ( which I have no control over it is decided by manufacturer). This is actually in ZFS guide you are referring to in your signature.[/QUOTE]

Mmh... nop. The difference comes from the TB/TiB difference. For example the datasheet of the Seagate ST3000VN000 (chosen just because I already have it) say that the guaranteed number of sectors is 5860533168. So 5860533168 * 512 = 3000592982016 bytes or 3000.6 GB. The drive is announced at 3000 GB so in fact you get just a bit more than that. Then if you do the conversion TB --> TiB you go from 3 TB to 2.73 TiB which is exactly the size you'll see in an OS ;)

This I won't comment on , just because I don't want to steer the discussion in different direction. Is just more important to find out how to set the pool so I can start working on while I still have a space to temporary move the data. I am just holding on to find the right answer because it wont be easy to change it after filling the pool.

 

[Bidule0hm]

Is just more important to find out how to set the pool

Well, I'd say just test both solutions for a few days and keep the one you prefer ;)

 

[Black Ninja]

Well, I'd say just test both solutions for a few days and keep the one you prefer ;)

Few days will take just to fill the pool with gigabit connection. Plus I don't have the extra 6 matching drives. If I am not going to use 1 single pool I can buy them at some later time , but if it's better to have One pool with 2 vdevs (each vdev is 8 HDD's in raidz2) then I have to buy them now , before I build the pool.

That's why I am waiting the hear from the specialists which way is better. I can also spend years of learning free bsd, zfs and code is made of but it just doesn't make sense for me to do that especially when I don't even work in IT field, to acquire all that knowledge just so I can setup freenas pool effectively.

 

[joeschmuck]

I think the talk about 512 byte vs 4k got bit off topic and it is of no consequence regardless, honest. Explaining what happens is very easy but it takes a bit of time and I've explained it a few times several years ago when 4k became the standard for hard drives. Simply put, even if the drive uses 512 byte emulation (most still do) the hard drive still records in 4k chunks on the platter. When 4k is enabled in FreeNAS (default setting now) it aligns the formatting of the drive so when a record is written it will always write on a 4k boundary.

It shouldn't take 83 hours to resilver a 4TB drive unless you have a very slow CPU and slow IOPs. The system you have listed is pretty speedy so you should not notice any speed difference in your final setup, the HBAs really help you out. I'm sure the use of multiple vdevs might be a bit speedier but will you even notice it. What is better is a bit subjective because you don't have high availability needs or stuff like that.

Few days will take just to fill the pool with gigabit connection.

That is true but I'd just create several large files to fill it up using the 'dd' command, should take several hours and you could type the commands to create all these files at the same time, and then copy over several usable files you want to test with. If you have no idea how to use 'dd' then I suggest you Google it.

You know, I'm a bit confused, you are running FreeNAS now with your system, you don't know how well it is running right now under your current configuration? Also you list a RAID10 setup, really?

 

[Black Ninja]

You know, I'm a bit confused, you are running FreeNAS now with your system, you don't know how well it is running right now under your current configuration? Also you list a RAID10 setup, really?

Everything in my signature is correct, nothing confusing about. What is wrong with the second pool being raid 10, what what I heard is the best raid of all , if you want speed, iops, and redundancy with loosing half of the space.

I know how my current setup works, I don't know who will it work if I buy 6 more 4TB HGST 5900RPM and remove second pool and put everythign in one pool. Simple way to put it: you have 16x4TB identical drives and my system , what is the best way to configure for movies storage.

 

[joeschmuck]

Simple way to put it: you have 16x4TB identical drives and my system , what is the best way to configure for movies storage.

In my opinion, a single pool RAIDZ3 only because you are using so many drives in a single pool. This gives you ~ 46TB of usable storage, damn that is a lot. It also gives you three drives of failure which I feel is a fair tradeoff for a massive amount of data storage.

Now if your storage needs are half that and you are still wanting to use 16 drives, you could go with 2 mirrored vdevs of 8 drives in a RAIDZ2 configuration which would give you about 21TB of usable space. What this gets you is a bit more safety in drive failures but more importantly more IOPS, but you really do not need high IOPS for the system you are looking to build, you are building a simple media server which has modest requirements.

I don't think you are going to get anyone to tell you how you should exactly configure your device, it's got to come from you. What I mean by that is if you value your data so highly well you could create 16 mirrors of a 4TB drive and have only 3.5TB of usable space however have fantastic data safety from a drive failure perspective. Some folks do not deem streaming media as high value and might tell you to use a RAIDZ1 setup which in my opinion would be plain stupid but that is just me.

So at this point I'm not sure what else anyone could tell you. You have all the facts you need to pull the trigger.

Last comment: If it were my system and I had all that hardware available to me and I wanted to use all the drives and I wanted to save a lot of media for streaming. I'd create two vdevs.
1) Media Storage - RAIDZ2 using twelve drives (36.4 TB usable storage)
2) Backup Storage - RAIDZ2 using four drives (7.5 TB usable storage)

This configuration gives me separation of my data from my media and if I need access to my data because my system fails, I only need to take my four hard drives to any computer with at least 4 SATA ports (damn near every computer has 4+ SATA ports these days), boot up the system with a FreeNAS USB stick and I have access to all my data. I don't have to find a computer that has 16 SATA ports to make it work. And there is another factor for you to think about. If you are going to use this system of yours for maybe backups of you home computers or to place important data such as family photos or financial data on, consider what it might take to retrieve that data if there was a serious failure of your hardware like one of the HBA boards becoming toast and then it takes out your MB and power supplies, not the drives. Shit happens, it really does.

I'd be curious to know what you end up doing.

 

[jgreco]

In my opinion, a single pool RAIDZ3 only because you are using so many drives in a single pool.

Cyberjock has said that he's seen problems going that wide, IIRC. 11 wide hasn't been problematic with RAIDZ3 though.

 

[mjws00]

@joeshmuck, clarify your final solution as two pools. Media and Backup. VDEVS can't be split out of a pool for recovery.

Ninja, this is excellent and practical advice.

I'll tl;dr it.
16 wide z3 is slow, cumbersome, and unsuitable for much at all. Meh.
2 vdevs of 8 drives is practical, with OK IOPS. Simple and balanced.
Joe's advice is clever and gives an extra recovery out.

 

[joeschmuck]

@joeshmuck, clarify your final solution as two pools. Media and Backup. VDEVS can't be split out of a pool for recovery.

Two separate pools, sorry, it was obvious to me but I definitely was not clear in my posting.

 

[Black Ninja]

Two separate pools, sorry, it was obvious to me but I definitely was not clear in my posting.

I figure you meant 2 pools not 2 vdevs. This is what I have now, you can see my 2 pools in my signature. The bigger pool is for media and backups, the second pool you notice it's raid10 I use it for data that is moved very often back and forward from there.
For me it's good to have 2 pools with different raid levels for different purposes. Example: one pool for storage one for speed and you put you data accordingly,

BUT

What I am trying to understand is what is easier for freenas to deal with:

1. One pool with 2 vdevs (each vdev is 8 HDD's in raidz2)

2. Two smaller pools: 1 pool with 1 vdev (10 HDD's in raidz2) and 1 pool with 1 vdev (6 HDD's in raidz2 or Raid10)

Al HDD will be same size and type.


Do you see what I mean ?

 

[Ericloewe]

I figure you meant 2 pools not 2 vdevs. This is what I have now, you can see my 2 pools in my signature. The bigger pool is for media and backups, the second pool you notice it's raid10 I use it for data that is moved very often back and forward from there.
For me it's good to have 2 pools with different raid levels for different purposes. Example: one pool for storage one for speed and you put you data accordingly,

BUT

What I am trying to understand is what is easier for freenas to deal with:

1. One pool with 2 vdevs (each vdev is 8 HDD's in raidz2)

2. Two smaller pools: 1 pool with 1 vdev (10 HDD's in raidz2) and 1 pool with 1 vdev (6 HDD's in raidz2 or Raid10)

Al HDD will be same size and type.


Do you see what I mean ?

I don't expect a measurable difference.

 

[Black Ninja]

I don't expect a measurable difference.

Why not ?
I am expecting an answer even if it turn to be that freenas don't care if you have 3 different pools or you have one pool with 2 vdevs , freenas will feel the same load becasue ... this what matters... or that what it matters...

Why sounds so hard , if I wrote frenas I will know in an instance what's better , because I would know how it works in a background ?!

 

[Ericloewe]

Why not ?
I am expecting an answer even if it turn to be that freenas don't care if you have 3 different pools or you have one pool with 2 vdevs , freenas will feel the same load becasue ... this what matters... or that what it matters...

Why sounds so hard , if I wrote frenas I will know in an instance what's better , because I would know how it works in a background ?!

The pragmatic answer is "because nobody has really complained either way". :p

Having two vdevs adds a trivial amount of complexity, easily masked by performance improvements.
Having two pools isn't really an issue either, since ZFS already handles multiple file systems at once (datasets).

Besides, nobody noticed odd slowdowns after 9.3 switched to a ZFS boot volume, despite the fact that most people have one (data) pool per server.

 

[Black Ninja]

The pragmatic answer is "because nobody has really complained either way". :p

Having two vdevs adds a trivial amount of complexity, easily masked by performance improvements.
Having two pools isn't really an issue either, since ZFS already handles multiple file systems at once (datasets).

Besides, nobody noticed odd slowdowns after 9.3 switched to a ZFS boot volume, despite the fact that most people have one (data) pool per server.

I like your setup with 6 drives in raidz2, then tell me how would you set it up if you had 16 identical HDDs ?

 

[Ericloewe]

I like your setup with 6 drives in raidz2, then tell me how would you set it up if you had 16 identical HDDs ?

Given compression, probably 2 vdevs with 8 drives.

 

[Black Ninja]

Given compression, probably 2 vdevs with 8 drives.

Let me ask you this: If you copy a single large file with 10Gb connection which will be faster:

a) from pool with 1 vdev( 10 hdd in raidz2)
or
b) from pool with 2 vdevs (8 hdd each in raidz2)