Drives added to server - extend existing Volume, or create a new one?

[Strohminator]

Hi gents.

Long time - hope everyone's well.

I have an existing Volume of 8x 8TB drives running in RAID-Z2 (40.6TiB) - it's sitting on 84% used capacity (I know, naughty naughty) --- I've taken the plunge and installed another 8x 8TB drives, with the intention of creating a new RAID-Z2 volume, and then manually moving across half the contents off the original one to balance everything out.

However, I'm now pondering the option of extending the volume, though I'm unsure whether I should.

I realize that it'd be a striped RAID-Z2 + 0 scenario (similar to RAID 60), which would see higher performance / throughput - but seeing as the original volume is already pretty much full, extending it probably won't offer any performance gain on the existing data (if any), and would only benefit data written post extending the volume --- unless the system will automatically do some kind of load balancing of the existing data between the vdevs once the Volume is extended?

Add to that the increased risk of losing the entire Volume dataset in the event of either vdev losing 3 drives during a resilver, and I'm not seeing any benefits, barring the convenience of not having to move some of my existing data over and tweak a few jails storage destinations...

What say you, oh wise FreeNAS community? Is there an added benefit that I'm missing here?

Thanks in advance!

 

[kdragon75]

Im sure other will have pros and cons but I will say you are right about no performance gain. ZFS does not rebuild or rebalance when adding vdevs to a pool.

 

[DrKK]

It is true that if you just striped in another 8x8TB into the pool as a second vdev, there would be no magic rebalancing across the thing.

However, I'm not sure that's a big deal. 84% isn't an emergency, especially on a pool that size. You won't have a performance *GAIN*, but why is that a big deal?

Your feasible options are to extend the volume, or to create a second pool. Here's what I'd say:

Option #1: You have a single pool, with two vdevs in it, each is 8x8TB, and one of which is the legacy (full'ish) VDEV

This is what most people would do, and what most people would mean when they added capacity to the pool, and certainly since you are adding a vdev of exactly the same structure as the existing one, it is also the natural thing to do (pools work best when the vdevs that comprise them are commensurate). All of your storage will be under a single pool. While the pool will not rebalance, new writing will be on the new vdev, and natural/organic motion of files will rebalance over time. There would be no performance gain. Each of the vdev's is in RAID-Z2, and in this case that means there is 25% redundancy. If you are running an honest FreeNAS, doing what we tell you, not virtualizing everything under the sun, and so on, I think your risk of losing a vdev (and hence the pool) is vanishingly small. In your spot, I "expand" the existing pool, 100% of the time.

Option #2: You now have two pools, each with one 8x8TB vdev in it.

You now have the hassle of two pools, two mount points, two roots to share to get at your data. The only thing you've gotten in exchange, really, is isolation of one vdev from the other, and the only benefit to that is (as you say) if you drop vdev during a million-to-one-meteor-strike-resilver-from-hell, your other vdev is safe. But my view is that a properly maintained FreeNAS system with 8-drive RAID-Z2 vdevs has negligible chance for dropping the vdev. Unless you're an idiot.

 

[Jailer]

I agree with @DrKK wholeheartedly and he stated it much more eloquently than I could have.

 

[DrKK]

he stated it much more eloquently than I could have.

You have an odd standard for "eloquent", sir.

 

[Jailer]

I'll take that as a compliment. :D

 

[DrKK]

I'll take that as a compliment. :D

I am just saying, there are certainly times when I am being eloquent. And that wasn't one of the times---I was simply just uploading what I thought about the situation.

 

[Jailer]

Well you still said it much better than I could have. I go back through my posts sometimes and think a lot of them should have been written in crayon.......

 

[Strohminator]

It is true that if you just striped in another 8x8TB into the pool as a second vdev, there would be no magic rebalancing across the thing.

However, I'm not sure that's a big deal. 84% isn't an emergency, especially on a pool that size. You won't have a performance *GAIN*, but why is that a big deal?

Your feasible options are to extend the volume, or to create a second pool. Here's what I'd say:

Option #1: You have a single pool, with two vdevs in it, each is 8x8TB, and one of which is the legacy (full'ish) VDEV

This is what most people would do, and what most people would mean when they added capacity to the pool, and certainly since you are adding a vdev of exactly the same structure as the existing one, it is also the natural thing to do (pools work best when the vdevs that comprise them are commensurate). All of your storage will be under a single pool. While the pool will not rebalance, new writing will be on the new vdev, and natural/organic motion of files will rebalance over time. There would be no performance gain. Each of the vdev's is in RAID-Z2, and in this case that means there is 25% redundancy. If you are running an honest FreeNAS, doing what we tell you, not virtualizing everything under the sun, and so on, I think your risk of losing a vdev (and hence the pool) is vanishingly small. In your spot, I "expand" the existing pool, 100% of the time.

Option #2: You now have two pools, each with one 8x8TB vdev in it.

You now have the hassle of two pools, two mount points, two roots to share to get at your data. The only thing you've gotten in exchange, really, is isolation of one vdev from the other, and the only benefit to that is (as you say) if you drop vdev during a million-to-one-meteor-strike-resilver-from-hell, your other vdev is safe. But my view is that a properly maintained FreeNAS system with 8-drive RAID-Z2 vdevs has negligible chance for dropping the vdev. Unless you're an idiot.

Thanks for clearing that up. You're right, I'm saving myself a fair whack of admin by not moving folders across, re-pointing jail storage locations, and then having to re-index + scrape those folders in Plex.

My concern with the 84% utilization was the fact that it's primarily static media storage, which won't be shifted around to eventually balance the load between the VDEV's - but if you say that doesn't pose a problem, I'm happy to continue as-is.

I've created a second Volume, and am copying some files over to it in order to put the new drives through their paces a bit before running a long SMART test to verify that all of them are healthy.... once done, I'll destroy the volume and re-create a second VDEV to expand the original volume.

Admittedly, I'm an idiot, but at least a functional one - I reckon I've got the basics covered - scheduled actions are a short SMART test each 3 days, long SMART test every 2 weeks, Scrub every 2 weeks - server terminal display is on permanently and in plain sight - so I simply keep an 8TB drive on hand to replace & resilver in the event of something going tits-up.

I'm running no virtualization or encryption - just a straightforward Media storage setup, Windows CIFS shares, with a Plex Plugin + Jail configured.

Am I missing anything in terms of maintenance or best practices?

 

[DrKK]

You know, if this is really bothering you, you can always (once you expand the vdev) take data from the original pool, copy it to some new location (with just 'cp'), delete the old files, and then 'mv' the files back to where you want them...and as you can do it directory by directory as needed, you control how much rebalancing will make you feel better, etc...

This little Irish dance will mix up the data on to the new vdev, presumably.