Drive layout question

[dTardis]

I am new to FreeNas so be gentle please. I have tried to read a bunch of the documentation, but I also wanted to ask here.

I have a SuperMicro x8 box with 16 x 2Tb drives connected up to a Adaptec 51645 and 8gb of Ram. I know the Adaptec controler is not optimal and I will be looking to add Ram soon, but I could not beat the price.

What I want to use this box for is just holding files for myself and as a media store/media server using Plex. Nothing really heavy duty.

When I first setup the system and went through the wizard it setup all 16 drives as a span and created a volume. That seemed not quite right to me, so I made this change. This is what I would like some input about.

I created 1 volume. In that volume I have 2 x ZRaid1 with 8 drives each. Does this seem like a good logical layout? Is there a better way to setup these 16 drives?

 

[Robert Trevellyan]

I have 2 x ZRaid1 with 8 drives each. Does this seem like a good logical layout?

Two vdevs makes sense, but RAIDZ1 is risky for drives larger than 1TB. Two 8-drive RAIDZ2 vdevs would be better.

 

[dTardis]

OK, can you explain why it is risky?

 

[Mirfster]

Depends actually on what your "use case" is....

There is always a decision needing to be made while thinking about Speed, Capacity and Redundancy...

For the record, I mostly lean towards RaidZ2... So I would agree with what Robert stated.

 

[dTardis]

ok, so I could be very wrong here, but in that configuration it would allow for up to 4 drives to fail? 2 from each vdev. Is that correct?

 

[Mirfster]

OK, can you explain why it is risky?

Take a little time to read @cyberjock 's "Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!"; it will help shed some light.

ok, so I could be very wrong here, but in that configuration it would allow for up to 4 drives to fail? 2 from each vdev. Is that correct?

Yes, you are correct in that assumption (if we are talking about the 2 RaidZ2 vdevs..)

 

[dTardis]

Take a little time to read @cyberjock 's "Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!"; it will help shed some light.


Yes, you are correct in that assumption (if we are talking about the 2 RaidZ2 vdevs..)

I will open that link now.

 

[Robert Trevellyan]

can you explain why it is risky?

It has to do with the expected rate of uncorrectable read errors (UREs) vs the size of the drive(s). With drives above 1TB, the probability of encountering a URE during resilvering (after replacing a failed drive) are high enough to be a significant risk. This is less of an issue with ZFS than traditional RAID5, because ZFS only traverses the in-use data where RAID5 blindly rebuilds the entire capacity, but the risk is still real. That's why RAIDZ2 exists (and why RAID6 was invented).

 

[dTardis]

It has to do with the expected rate of uncorrectable read errors (UREs) vs the size of the drive(s). With drives above 1TB, the probability of encountering a URE during resilvering (after replacing a failed drive) are high enough to be a significant risk. This is less of an issue with ZFS than traditional RAID5, because ZFS only traverses the in-use data where RAID5 blindly rebuilds the entire capacity, but the risk is still real. That's why RAIDZ2 exists (and why RAID6 was invented).

Thank you. That was a great answer. Also the slideshow was very interesting.

 

[Montel Bahn]

Another option, could be

2pools/

• poolSmall/4disks in raid 10 # nominaly 4000 GB for precious photos, maybe music, 64 k record size
• poolBig/2 vdevs of 6 disks in RaidZ2 # nominaly 8000 GB for less precious videos, 2048 k record size

So.....0 zvols, 2 Volumes, 4 vdevs, and 14 disk devices.

I know, it's so Inside baseball, but it's ok, they'll have a wonderful wizard in the next release.

Seriously, all good options. But you NEED more ram!

​

 

[Robert Trevellyan]

poolSmall/4disks in raid 10

Do you mean "striped mirrors"?

 

[dTardis]

Another option, could be

2pools/

• poolSmall/4disks in raid 10 # nominaly 4000 GB for precious photos, maybe music, 64 k record size
• poolBig/2 vdevs of 6 disks in RaidZ2 # nominaly 8000 GB for less precious videos, 2048 k record size

So.....0 zvols, 2 Volumes, 4 vdevs, and 14 disk devices.

I know, it's so Inside baseball, but it's ok, they'll have a wonderful wizard in the next release.

Seriously, all good options. But you NEED more ram!

​

So I acquired/found some more RAM. My FreeNas now has 32Gig of ram. So that should be a little better.
just out of curiosity. What if I broke it into 3 vdevs? Is that even doable?

 

[Robert Trevellyan]

My FreeNas now has 32Gig of ram.

What if I broke it into 3 vdevs?

You can break it down however you want. Just don't confuse vdevs with pools.
pool = 1 or more vdevs = Volume in FreeNAS GUI Volume creation screen
vdev = 1 or more disks = row in FreeNAS GUI Volume creation screen

 

[Montel Bahn]

Do you mean "striped mirrors"?

Yes, very sorry for sloppy terminology, given I was trying to make a point of being accurate with my words in that post.

 

[joeschmuck]

Since you are only looking to use your NAS as a storage and media streaming device, might I suggest a RAIDZ3 as an option as this would give you a little more storage capacity than the two vdevs of 8 drives in a RAIDZ2. It's really a toss up. I believe there is some rule of thumb about how many drives maximum a vdev should be made of but rules of thumb are not always absolute.

 

[dTardis]

Since you are only looking to use your NAS as a storage and media streaming device, might I suggest a RAIDZ3 as an option as this would give you a little more storage capacity than the two vdevs of 8 drives in a RAIDZ2. It's really a toss up. I believe there is some rule of thumb about how many drives maximum a vdev should be made of but rules of thumb are not always absolute.

That's an interesting idea. So one Pool of all 16 drives setup to RAIDZ3. It would have up to a 3 drive failure. Correct?

 

[Mirfster]

So one Pool of all 16 drives setup to RAIDZ3

Correct me if I am wrong, but I thought that it was highly suggested to not have any vdev consist of more than 11/12 drives?

 

[joeschmuck]

Correct me if I am wrong, but I thought that it was highly suggested to not have any vdev consist of more than 11/12 drives?

That is what I meant by rule of thumb, but a home system isn't a business system.

That's an interesting idea. So one Pool of all 16 drives setup to RAIDZ3. It would have up to a 3 drive failure. Correct?

Yes.

 

[Mirfster]

That is what I meant by rule of thumb, but a home system isn't a business system.

You rebel... :D

 

[joeschmuck]

You rebel... :D

I guess I am, however I still feel everyone should make up their own minds on how they want to run their pool. I just want to ensure they know what the options are, not what I personally prefer. Personally I can't imagine having 16 2TB drives for a home system, it seems a bit excessive and I'd opt for fewer larger drives. 16 drives is a lot to maintain as when the drive start to fail, replacing them will be expensive. 23TB of storage for a home is a lot of data and if the data ever needs to be moved off, well like I said, that is a lot of storage.

@dTardis do you really need that large of a pool? What are your real storage needs? I'm asking because of my comments above, that is a lot of drives to manage. In my situation I have six 2TB drives in a RAIDZ2 configuration. Even with all my movies and backups (I have a few backups for certain) I'm only using almost half my capacity. Now lets say I need to expand my capacity well I can do it one of a few ways... My personal favorite way is the most economical and that is with good planning and I forecast my data growth and I replace my 2TB drives with say 4TB drives to double my capacity, however I replace them as they either fail or as I can afford them. Of course you will not see an increase in your pool size until all the drives have been upgraded but it's better for your wallet. Other ways to grow your pool size it to add vdevs which means more hard drives to manage and is good for a large office, not the home office.

So if you can survive using a smaller handful of the 2TB drives for now, then replace them with larger drives in the future, that would be what I'd recommend you do. But it's up to you to figure out your storage requirements. You could always use 8 drives for an off-site backup or some other project, or keep a few for replacements.

Sorry, I get long winded.