Is it safe for me to expand my pool?

[ROT Tom]

We have an iXsystems FreeNAS Certifiied server. It has 24 bays, of which currently 12 are filled with 6 TB disks.

These comprise 2 RaidZ2 sets of 6 disks each.

They're in one storage pool, primarily sharing one large dataset.

We want to expand the pool size. When we bought the FreeNAS, this was the plan, iXsystems said this would be fine. Now our managed services provider is telling us that it is not safe to expand a pool. They say we can add disks, but the added disks will need to be a separate pool and we'll need to segment our data across the two pools.

Is this true that there is no safe way to populate our 12 free bays and use the disks to expand our current pool?

We were planning to add 12 more 6 TB disks, once again divided into two RaidZ2 sets.

Also, I did Google and read things for quite a while, but did not find good resources. Can anyone point me to a guide to expanding zpools? I couldn't find anything explaining when and how you can expand a pool, or any guides using the FreeNAS GUI tools to expand a pool... can it be done via GUI, or does it have to be done via command line?

Thanks in advance for any help.

 

[garm]

Adding vdevs to a pool is perfectly fine, it would be interesting to hear their argument.
In a corporate environment there might be risk factors preventing tampering with the server

 

[ROT Tom]

Thanks. Their solution still involves adding the same disks to the same server. So I don't think it's server tampering that's the problem. They're saying it's a dangerous operation to expand a pool, like if we run a command to expand the pool, instead of expanding it, we might just lose all the data on it, or the system might crash, or something like that.

 

[jgreco]

Thanks. Their solution still involves adding the same disks to the same server. So I don't think it's server tampering that's the problem. They're saying it's a dangerous operation to expand a pool, like if we run a command to expand the pool, instead of expanding it, we might just lose all the data on it, or the system might crash, or something like that.

Sure. If you run "rm -fr /" on a UNIX box it's toast too. Therefore it's dangerous to use UNIX. And you can hose Windows systems too. We should all stop using these computer things because of the horribly high risk of catastrophic failure on a daily basis. (twirls finger dramatically).

Read the instructions. FIRST. They're in the manual.

Maybe do a dry run (since you have a bunch of spare disks to set up a second "experimental" pool with) and see how it works. Any danger aspect comes from something going horribly sideways. Maybe if a bunch of your new disks are bad (have you burned them in?) or if there's an error made in the vdev addition process. You fix that first one by burning them in. You fix the second with practice.

 

[ROT Tom]

I did wonder if the underlying issue might be that they're worried they might make a mistake in the process, but we have an external consultant iXSystems recommended to us to who we're offering to hire to remote in and perform the configuration. So that should reduce the probability of "rm -fr /" sorts of errors.

I was mainly just posting here to see if anyone was going to say "expanding zpools is buggy... sometimes the whole pool's lost... expect a high possibility of problems when you do that."

I'm getting the impression that this isn't the case. That if done correctly, this should be a safe operation to perform. That what we're looking to do is a pretty standard-practice, "should work reliably" thing to do.

 

[jgreco]

Yeah, it is reliable. It's just that it's something infrequently done, and because people are sometimes blundering through the UI, that SSD they're trying to add as a SLOG or an L2ARC ends up being made as a new data vdev (you can find dozens or more examples of this in these forums), which is fatally catastrophic. Well your pool doesn't melt like the Nazis in the Raiders of the Lost Ark or anything like that, but you have ruined your pool and then you need to migrate all the data off, blow the pool, and migrate it all back on. No data loss but a HUGE mistake with a HUGE amount of work to remediate, and if you have a massive server you might not have anywhere to do the migration to. This is why I would HIGHLY recommend that you insert the drives and do create a second pool, experimenting carefully to familiarize yourself with the process (make a 3-disk RAIDZ1 and then add another, and another, etc) until you are confident with pool operations. Definitely be sure to avoid torching your existing pool too. Once you get your head around how the GUI presents, it is really very straightforward.

 

[ROT Tom]

Can anyone point me to directions for doing this anywhere?

I think we'll hire the consultant, just to be safe, but I'd like to look it over myself, and I might do what you're suggesting even with the consultant - physical add the disks, make a 3-disk z1, expand by 3, test, expand by 3, test, etc. Just so I can follow along and look for mistakes myself when we're doing it for real.

Thanks for the feedback.

 

[ROT Tom]

Also, we do have the entire thing cloned to another FreeNAS in the same rack, and also backed up to the cloud. So while it would be a serious inconvenience if we did something that lost the data on this (we need to configure the user accounts on the backup and point everything at it), we can switch to running off the Backup FreeNAS while we delete and fix our pools on the primary, and then migrate back... once our 50 TB of data is done copying back.

 

[SweetAndLow]

Are you joking? It's super easy to extend a pool. Takes like 20 second. In the gui go to your storage pool, extend your pool, make sure the layout is correct and click ok. Everything is done in the gui not the cli. Using the cli is wrong.

 

[Fredda]

we do have the entire thing cloned to another FreeNAS in the same rack

Just one quick remark. From a backup perspective it is not a good idea to have the server and its backup in the same rack.
If you have the option, put them as far apart as possible. If you have a fire, water, power, AC problem there is a good chance both servers will be affected.

 

[danb35]

I couldn't find anything explaining when and how you can expand a pool

Start with the manual:

10. Storage — FreeNAS®11.3-RELEASE User Guide Table of Contents

www.ixsystems.com

 

[ROT Tom]

Thanks. We are aware it's non optimal to have them in the same rack and have talked about going as far as getting it into a different room. But we still have a cloud backup too if worst comes to worst.
We're about to be rearranging how all this works anyway and make the cloud the primary and have that backup to the FreeNAS, at which point the highly redundant cloud copy will always be canonical/current.

It seems to me that there's some differing opinions above between jgreco and SweetAndLow about the probability of doing something wrong by mistake.

But the important info I needed here was just confirmation that, if performed correctly, there is a low probability of this being risky from a "this is buggy" perspective.

Thanks.

 

[SweetAndLow]

Thanks. We are aware it's non optimal to have them in the same rack and have talked about going as far as getting it into a different room. But we still have a cloud backup too if worst comes to worst.
We're about to be rearranging how all this works anyway and make the cloud the primary and have that backup to the FreeNAS, at which point the highly redundant cloud copy will always be canonical/current.

It seems to me that there's some differing opinions above between jgreco and SweetAndLow about the probability of doing something wrong by mistake.

But the important info I needed here was just confirmation that, if performed correctly, there is a low probability of this being risky from a "this is buggy" perspective.

Thanks.

Also are your trying to add more drives or replace existing success with larger ones? Usually the word expanded means replace with larger drives and extend means to add more drives in a vdev.

 

[ROT Tom]

Extend then. Sorry, I didn't know that terminology difference.

 

[SweetAndLow]

Extend then. Sorry, I didn't know that terminology difference.

It's minor but clarification is good. Yeah I would just create a freenas vm and play around with extending the pool and adding new vdevs. With a VM you can even create the exact disk layout you currently have. Whole process should be super simple. If you do add the disks wrong there is no undo button so you will have to live with it or rebuild your pool.

 

[ROT Tom]

Thanks. Using a VM for practice is a good idea.

I did read the manual, but the entry on extending a pool is pretty short and lacking in detail, and some of it seems to not apply in a way I can make sense of. Maybe I'm on an older version of the UI. I'm on 11.1-U6.

"To increase the capacity of an existing pool, click the pool name,  (Settings), then Extend."

I assume when they say "Click the pool name" they mean to choose the "Storage" tab at the top of the UI, then click the pool name... I do that, but there's no "gear" icon, no "settings" option.

I found the ability to extend a volume in "volume manager" under "Storage" in the left side bar, but I'm not remotely confident in what's going to happen based on what I do in the UI.

"When adding disks to increase the capacity of a pool, ZFS supports the addition of virtual devices, or vdevs, to an existing ZFS pool."

OK, so do I have to create VDEV's first and then add them to the pool with "extend," or does the creation of VDEV's happen at once when I choose to extend? I searched the manual for "vdev" and didn't see any section on creating vdev's...

I'm not looking for someone to answer all these questions here and walk me through the process, I'm just saying that as a novice, I don't find the manual entry on this very helpful or informative. I was hoping that somewhere online there might be more of a walk-through with UI screenshots and some discussion about what's going on and why.

Anyway, I'll probably do a VM and play around with it to gain some confidence, but either way, it's not that much money to just hire a consultant for this so we have someone who knows exactly what they're doing.

 

[SweetAndLow]

Thanks. Using a VM for practice is a good idea.

I did read the manual, but the entry on extending a pool is pretty short and lacking in detail, and some of it seems to not apply in a way I can make sense of. Maybe I'm on an older version of the UI. I'm on 11.1-U6.

"To increase the capacity of an existing pool, click the pool name,  (Settings), then Extend."

I assume when they say "Click the pool name" they mean to choose the "Storage" tab at the top of the UI, then click the pool name... I do that, but there's no "gear" icon, no "settings" option.

I found the ability to extend a volume in "volume manager" under "Storage" in the left side bar, but I'm not remotely confident in what's going to happen based on what I do in the UI.

"When adding disks to increase the capacity of a pool, ZFS supports the addition of virtual devices, or vdevs, to an existing ZFS pool."

OK, so do I have to create VDEV's first and then add them to the pool with "extend," or does the creation of VDEV's happen at once when I choose to extend? I searched the manual for "vdev" and didn't see any section on creating vdev's...

I'm not looking for someone to answer all these questions here and walk me through the process, I'm just saying that as a novice, I don't find the manual entry on this very helpful or informative. I was hoping that somewhere online there might be more of a walk-through with UI screenshots and some discussion about what's going on and why.

Anyway, I'll probably do a VM and play around with it to gain some confidence, but either way, it's not that much money to just hire a consultant for this so we have someone who knows exactly what they're doing.

Get the documentations for your version of freenas. Don't use different version of the documentation

FreeNAS® 11.1-U7 User Guide — FreeNAS®11.1-U7 User Guide Table of Contents

www.ixsystems.com

Vdevs get created and added to your pool all in one step.

 

[ROT Tom]

Thanks, I did go look at the manual entry for the appropriate version, which clears up the UI question, and I think actually contains some better information than the docs for the current version, but is still sort of "sparse" on the information for such an important operation.

One question I was wondering about that didn't see answered in the manual - do we need to go with the same 6 TB disks? I assume since it said this will be "striped" that that's the case - that if we were to do, say, two raidz2 arrays of 6 disks each using 12 TB disks, that adding those to the existing zpool, it would only see half the space we'd added, so we might as well use 6TB disks? But I didn't see any direct confirmation of how this works.

I don't want to get this thread too off-topic here, but I'm wondering if for the long-term, we'd be better off first growing (expanding?) the pool, as per here:

8. Storage — FreeNAS®11.1-U7 User Guide Table of Contents

www.ixsystems.com

And then next time we need to upgrade we'll extend to the additional drive slots.

There are two reasons I'm thinking this would be better:

1. We have free drive slots now, so we can use the stated method using "replace" to upgrade all the disks without having to mess with iSCSI or degrading the array. Once we fill all our slots with used drives, we'd either have to use iSCSI or degrade the array to put in larger disks.

2. We'd save money in drives. Say our end state for this hardware in a few years is to have 24 each 12 TB disks. If we first upgrade the existing disks to 12's and then add another set of 12s, we'd save ourselves from buying 12 each 6 TB drives. If we extend now so the whole box is full of 24 each 6 TB drives, then we're replacing all 24 of those when we grow the zpool to 12 TB drives, instead of just adding 12 each 12 TB drives.

 

[blanchet]

It is a very good idea, to replace each existing 6TB disks by 12TB disks. It will double your available space, and you can decide later how to use the last free disk slots.

Nevertheless, if you really run out of space, you can add new vdevs with different disks sizes: it will work and you will waste space only if all the disks inside a particular vdev are not the same. Obviously it is not an optimal practice because the IO will not be well balanced among the vdevs (the larger vdevs will have to process more IO than the others).

Do not worry if you run out of free disk slots: with RAIDz2, it is not dangerous to degrade the array for few hours to put larger disks.

 

[kappclark]

Followup question on this topic -- I have a pool (4 x 1TB) comprised of 2 mirrors - seagate barracuda 3.5 spinning .... Would like to add a third 2x 1TB mirror to the pool, -- do the drives need to be the same make and model, or could I use a pair smaller 2.5" disks ? I know zfs can be picky about hw, but seems reasonable that I cld use 2.5" disks, as long as the mirror is setup the same ...