16 Bay Supermicro Chassis

Status
Not open for further replies.

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
A vdev will only expand once all the drives in the vdev are replaced.

The other approach is to add another vdev.
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
But that's what I tried to do. Problem was that one bay doesn't work. soI only had 3 bays free. And you guys said a Raidz1 is no good.
What do I do now?
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
My layout is like this. I have a 16 bay system. 15 out of the 16 works.

I have 2 Vdev. Raidz2-0 has 6 (6TB) WD red. Raidz2-1 has 6 (TB). So I have 12 hard drives in total.

I bought 4 (8TB) WD reds to make a new Vdev. But now that I found out that 1 bay doesn't work I can only use 3 hard drives. I thought of using 2 for data and 1 for parity (Raidz1), but I was advised not to go that route, so I didn't. So what I did was replaced 2 (6TB) drives from Raidz2-0 and 2 (6TB) drives from Raidz2-1 with all 4 8TB drives. Resilvering went without a hitch.

Now I'm looking at my empty space and it's still the same. I just spent over $1,400 for these 4 (8TB) drives and I'm still in the same place. FreeNAS is still warning me with a yellow light that I'm low on space.

What can I do in my situation to get more space with the hardware I have now?
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
If you don't want to purchase 2 more 8TB drives, then you might as well restore the now spare 6TB drives.

You could add another vdev. Rather than stuffing up your pool permanently you could add it as another pool.
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
1. Do I have to resilver the 6TB drives back in or can I replace them back in the same bay as they were before?
2. (add another vdev. Rather than stuffing up your pool permanently you could add it as another pool.) Don't understand?
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
You need to resilver. Sorry.

But at least you can resilver 3 at once :)

You can have multiple pools. Each with multiple vdevs each with multiple drives.

You can not remove or reshape a vdev once created and added to a pool.
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
lol, now I'm really confused.

Are you saying I can shut down the server, take out the new 4 (8TB) drives, replace them back with the old 6TB drives, and it should work like normal without resilvering?

Now if the above is true, what do I do with the 4 (8TB) drives I have now with only 3 empty bay on my server?
 

ChriZ

Patron
Joined
Mar 9, 2015
Messages
271
... So what I did was replaced 2 (6TB) drives from Raidz2-0 and 2 (6TB) drives from Raidz2-1 with all 4 8TB drives...
If you want to expand the size of your pool, without adding another vdev, you must first replace all of the drives in an existing vdev with larger ones. You have only replaced 2 drives from each 6drive vdev.
If you want your available space to expand, you must choose which vdev you want to have new drives (the one with the older ones is a good idea)
Replace back the two 8TB drives from the other vdev with the previously installed 6TB drives. Then use the 8TB drives to replace another 2 drives in the chosen vdev.
Once this is done, the chosen vdev will have four 8TB drives and two 6TB.
Then you must purchase another two 8TB drives. Replace the remaining two 6TB drives.
After this, the chosen vdev will have all of its 6TB disks replaces with 8TB disks.
Then it can expand
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
If you want to expand the size of your pool, without adding another vdev, you must first replace all of the drives in an existing vdev with larger ones. You have only replaced 2 drives from each 6drive vdev.
If you want your available space to expand, you must choose which vdev you want to have new drives (the one with the older ones is a good idea)
Replace back the two 8TB drives from the other vdev with the previously installed 6TB drives. Then use the 8TB drives to replace another 2 drives in the chosen vdev.
Once this is done, the chosen vdev will have four 8TB drives and two 6TB.
Then you must purchase another two 8TB drives. Replace the remaining two 6TB drives.
After this, the chosen vdev will have all of its 6TB disks replaces with 8TB disks.
Then it can expand
Yep. I understand that part as I finished resilvering the 4th drive this morning when I read this post.
Well I built this server 2 years ago with all new 6TB drives. So they are all the same age.

So to understand this Raidz correctly, can I just swap back the same 6TB drives I had before in the same slots? They weren't destroyed as it looks like it just copied all the data to it as I used the empty 3 slot bays to resilver.

Also, with 3 bays empty, what can I do? Right now buying 2 more 8TB drives is out of my budget. For the $650-700 I can just buy a used JBOD on eBay and start a new vdev there with the extra 4 drives I'll have with the old 6TB drives. Then I'll have more space for future expansion. But the budget doesn't allow for that right now. I just shelled almost $1,400 which I didn't have to be in the same place. I just want to know how can I expand free space with what I have now (15 bay chassis, 2 vdev, 12 (6TB), 4 (8TB)).

Thanks
 

Scharbag

Guru
Joined
Feb 1, 2012
Messages
620
So to understand this Raidz correctly, can I just swap back the same 6TB drives I had before in the same slots? They weren't destroyed as it looks like it just copied all the data to it as I used the empty 3 slot bays to resilver.
NO!

You must resilver as those 6TB drives if you replaced them with 8TB drives. The pool will not understand and data loss can occur.

There are some good guides available that explain pools and vDevs. VDevs are sorta the thinger that once you make it, you are stuck with it. Pools are made up of 1 or more vDevs. You can never change a vDev or remove a vDev from a pool. You can always add another vDev to a pool. ZFS allows you to mix vDev types in a pool but best practice is to not do this. There are also some best practices when using Z1, Z2 and Z3. With todays drive sizes, Z1 is not really recommended due to the long resilver time.

Bottom line, with your setup, you should buy 2 more 8TB drives and then resilver them into a single vDev to increase the total pool capacity by ~8TB. You really have no other good option unless you can get a case that holds at least 18 drives. OR, if you have backups, you could destroy your entire pool and make a 2 vDev Z2 with 7 or 8 drives each (assuming you can fix the bad bay). This will treat all drives like 6TB drives until you have enough 8TB drives for at least one entire vDev. Soooo many options...

ZFS can be fun at times. I have been through this a couple times now and man, it really sucks trying to maximize capacity, reliability and performance- all while trying to figure out what is best for the next 5 years. For some reason, my system evolved to use 6 drive Z2 vDevs too. They work really well in 36 bay enclosure:) Due to Z1 sucking, I did change my backup pool to a 12 drive vDev Z2 pool because I could not afford 4 parity drives. But, my plan is to replace the 4TB production drives with 6TB or 8TB drives as needed and then use the 4TB drives to replace the 3TB backup pool drives. So, should still work out over time. Plus I can add another 6 drive vDev to my production pool with the spare bays in the existing case. Or, I can add a JBOD enclosure... Man, what a process...

Happy ZFSing.
 

AltecBX

Patron
Joined
Nov 3, 2014
Messages
285
WOW, thanks for the knowledge. I did the research and you're right. I'm sticking with the 2 vdev 6 drive setup. Since 1 slot doesn't work I can't add another vdev unless going z1 which I won't do. As of now, I have 4 free 6TB drives so my best bet, which you'll probably agree with, is to start a new vdev in another enclosure. It's no fun to spend $1,400 and be at the same disk space.

So knowing my chassis, can you suggest a JBOD cheap enough for me to jump the plunge in the next month. I don't care if is used, as I bought my chassis on eBay hence the 1 port I didn't know was bad. I'm also considering doing backups like you have in yours. I know I'll have to increase my memory as well to support the extra TB's.
 

Scharbag

Guru
Joined
Feb 1, 2012
Messages
620
It's no fun to spend $1,400 and be at the same disk space.
Lol. Do the math on what it will cost to upgrade 12 4TB drives to bigger ones... Fun times.

You should buy 2 more HDDs and update a complete vDev. That way you get space now. Once you buy a new enclosure, then you will have 6 drives ready to add another vDev.

Stick with supermicro IMO as they are good quality. Buy a 16 or 24 bay JBOD with SAS2 backplanes. Other than that, build up a Norco 4224 or 4220 but their quality is not close to supermicro.

Cheers,
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,358
Probably would work out cheaper to upgrade to a 24 bay enclosure and sell your 15 bay.

Or keep it so you can build a backup server.

Cheapest most immediate solution with best outcome is probably to buy 2 more 8 TB drives :(
 
Status
Not open for further replies.
Top