Are we any closer to vdev expansion?

koberulz

Dabbler
Joined
Aug 4, 2022
Messages
45
The last estimate I saw was "some time in 2023".

I've just hit 99% capacity on my setup, which was supposed to last me years longer than this. Parkinson's Law of Data and all that.

So I'm...kind of screwed, at least for now, it would seem. Trying to figure out what options I have, if any...
 

koberulz

Dabbler
Joined
Aug 4, 2022
Messages
45

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
So a raidz2 with 4 drives? Ok, was just curious. So right now you'd need 4 more to add another vdev. I can see why you'd want this functionality to land soon :)
 

koberulz

Dabbler
Joined
Aug 4, 2022
Messages
45
I don't know if the smile is supposed to be taunting, sympathetic, or condescending.

I'm not sure what adding another vdev even does, TBH, but right now 80TB of storage is not within my budget. But I'm at 99% capacity, so something needs to happen quicksmart...
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
You shouldn't have gone beyond 85-90%, now you are in trouble. Nothing will happen "quicksmart", either you pay to expand your storage (there are a few options) or revert to an older snapshot with less data.

Regarding RAIDZ espansion, please look into the following thread.

Last openzfs leadership call had a brief segment on this: ixSystems are sponsoring work to bring raidz expansion over the finish line. The person working on it has rebased the code, and is working on fixing issues in the test suite.

By 2024 it ships? This may actually happen.
 
Last edited:

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
I don't know if the smile is supposed to be taunting, sympathetic, or condescending.

I'm not sure what adding another vdev even does, TBH, but right now 80TB of storage is not within my budget. But I'm at 99% capacity, so something needs to happen quicksmart...
Sympathetic, since expansion would require 4 more 20T drives to add a new vdev. I totally get being in that boat, thats a lot of new drive $ to add all at once.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
You could mitigate the issue by creating a new pool with 2 disks in a mirror, resync some files over there and then delete them from the original pool, getting back some performance.
It will require a reorganization of how/where you store your data though.
 

koberulz

Dabbler
Joined
Aug 4, 2022
Messages
45
You could mitigate the issue by creating a new pool with 2 disks in a mirror, resync some files over there and then delete them from the original pool, getting back some performance.
It will require a reorganization of how/where you store your data though.
Most of it is a media server, but there's a chunk that isn't. 200GB of it was a temp file (lossless AVI out of Premiere ready for an x264 encode) so I got rid of that. Tons of movies I've recorded off TV which are now a priority. But there is some stuff I can move relatively painlessly, just don't know the best way to go about doing that...plus I have a huge stack of Blu-rays here that need to go on the media server. Ugh.

I have no idea where the last 10% went, TBH. I had some space, then I had nothing, it happened so fast.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Could be snapshots. Best way to move things is surely resync or zfs send | receive.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
I don't understand what you just said.
The 10% space you don't know why is full could be snapshots; it's an educated guess.

Resync and zfs send | receive are ways to move data, a search in the forum will give you plenty of results.
 

koberulz

Dabbler
Joined
Aug 4, 2022
Messages
45
The 10% space you don't know why is full could be snapshots; it's an educated guess.
I'm sure it's just stuff I've put on there, it just happened so quickly.

Resync and zfs send | receive are ways to move data, a search in the forum will give you plenty of results.
I don't presently have anywhere to move it to.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
I don't presently have anywhere to move it to.
[...] But there is some stuff I can move relatively painlessly, just don't know the best way to go about doing that... [...]
Also:
You could mitigate the issue by creating a new pool with 2 disks in a mirror, resync some files over there and then delete them from the original pool, getting back some performance.
It will require a reorganization of how/where you store your data though.

You can't move out from this situation withouth removing thata from that pool or spending money.
 

koberulz

Dabbler
Joined
Aug 4, 2022
Messages
45
"How to go about doing that" was more in reference to the logistics of creating somewhere to move the data to than actually how to move it, because Windows Explorer drag-and-drop is a thing.

You can't move out from this situation withouth removing thata from that pool or spending money.
Assuming we omit "permanently deleting data" from the list of possible solutions, it all involves spending money as I have zero spare drives at present. Which makes it all the more important to have some sort of plan here.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
"How to go about doing that" was more in reference to the logistics of creating somewhere to move the data to than actually how to move it, because Windows Explorer drag-and-drop is a thing.
That means setting up a SMB share and leaving a machine on for hours while you transfer files from a dataset or a pool to another. Not ideal, but possible.

Assuming we omit "permanently deleting data" from the list of possible solutions, it all involves spending money as I have zero spare drives at present.
From the informations you gave us I don't see any other options than:
  • deleting some files from the pool
  • expanding the pool by adding another vdev
  • creating a new pool and moving some files there
The first option means either deleting data (I'd start from the snapshots) or moving some files to other places such as another machine, the cloud or an external drive; this should cost you little or no money depending on what you have.
The second option means doubling your space and, as Kris already noted, costs a considerable sum of money.
The third option is to buy two big enough drives to create a new pool and move some of your data there; it could cost less than adding another vdev to your main pool, but it comes with less flexibility and the need for a reogranization of your storage and sharing strategy.

Which makes it all the more important to have some sort of plan here.
Nope, you should have had a plan before reaching this point or rather you shouldn't have let it reach this point: you should either have monitored better your available space (TN tries to tell you this in quite a few ways) or choose a different pool layout when you assembled your system (like mirrors), maybe both. Now there isn't much we can do without spending money.

I'd also like to point out that the tone I'm picking up from of a few of your replies isn't exactly going to make people address your issue; I understand your frustration, but please be aware of that since it's going to hurt you in the long run if it escalates.
 

koberulz

Dabbler
Joined
Aug 4, 2022
Messages
45
That means setting up a SMB share and leaving a machine on for hours while you transfer files from a dataset or a pool to another. Not ideal, but possible.
Wait, there's a way to move files to another drive that wouldn't take hours? I was just assuming that as a given, because either way you're physically moving data to another disk, right?

Nope, you should have had a plan before reaching this point or rather you shouldn't have let it reach this point: you should either have monitored better your available space (TN tries to tell you this in quite a few ways) or choose a different pool layout when you assembled your system (like mirrors), maybe both. Now there isn't much we can do without spending money.
I did have a plan before reaching this point: vdev expansion had an ETA of 2023, so I created a vdev I believed was large enough to last me through at least 2024. I just...got it very badly wrong. My Synology is 70TB and took six years to fill, my TrueNAS is 35TB and filled in about a year.

I'd also like to point out that the tone I'm picking up from of a few of your replies isn't exactly going to make people address your issue; I understand your frustration, but please be aware of that since it's going to hurt you in the long run if it escalates.
I don't know what tone you're picking up, but it doesn't seem to be the same one I'm typing with.
 

Davvo

MVP
Joined
Jul 12, 2022
Messages
3,222
Wait, there's a way to move files to another drive that wouldn't take hours? I was just assuming that as a given, because either way you're physically moving data to another disk, right?
You can do so without the need of a Windows machine staying on alongside your NAS and being capped by your network speed. You can also move data internally to the NAS or to the Cloud without having to rely on SMB, you just run the command and leave the NAS on.

I did have a plan before reaching this point: vdev expansion had an ETA of 2023, so I created a vdev I believed was large enough to last me through at least 2024. I just...got it very badly wrong.
It happens, suggested approach here is to double your storage expectations. Using mirrors would have made expansion more flexible and easier on the wallet. Also, relying on an uncertain feature is not a reliable plan as you unfortunately discovered.

The main issue now is the 99%: you have to bring that under 90 as your top priority.

I don't know what tone you're picking up, but it doesn't seem to be the same one I'm typing with.
Glad to hear read it.
 
Last edited:
Top