How to resize volume with existing data?

[tciweb]

Hello,

I am a zfs/freenas noob I have received the following error in my log email:

Traceback (most recent call last):
File "/usr/local/bin/graph.py", line 480, in <module>
GenerateInterfaceGraph()
File "/usr/local/bin/graph.py", line 135, in GenerateInterfaceGraph
'GPRINT:global_tot_tx:ca. %5.1lf%s Total\l'
rrdtool.error: Could not save png to '/tmp/graph/if-usbus0-1y.png'​

And also this:

+++ /tmp/security.j3xKXtNE2015-05-13 03:01:00.000000000 -0400
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3106 on /var: filesystem full
+pid 1999 (syslogd), uid 0 inumber 3113 on /var: filesystem full​

So I have attached a screenshot of my volumes, it seems that /mnt/MV1 is too small and that is why I am getting these errors. /dev/zvol/MV1/VOL1 has existing production data, so how do I increase the /mnt/MV1? Any help is greatly appreciated.

 

[SweetAndLow]

Please provide the information required by the forum rules.

 

[cyberjock]

There's a really good chance your data in vol1 is corrupted. You shouldn't have made the size of the zvol large enough to fill the zpool to 100%.

 

[tciweb]

There's a really good chance your data in vol1 is corrupted. You shouldn't have made the size of the zvol large enough to fill the zpool to 100%.

So what would be the best course of action from here?

 

[cyberjock]

Honestly, you have to free some space, somehow.

With zero bytes free you can't even add a vdev.

You can try deleting stuff from the .system dataset, but I'm not sure exactly how you'll get out of this corner. Pools shouldn't be filled >80%, and one of those reasons is to give you time to expand before you hit 100%. But you are at 100%. :(

My first attempt would be to add another vdev of disks. Even if you delete stuff from .system dataset, there will literally be "almost" no free space, so even more than the "5 second login to the iscsi device" is to add more free space. More than likely once you mount the iscsi device you'll run out of space within a few seconds just due to basic reads and writes from standard background stuff unless you add more disk space vice making more disk space. There's also no guarantee your data will be intact.

Edit: Just to clarify, what you will probably have to do is delete some stuff from the .system dataset, then go add a vdev before you start the iSCSI service again so that it doesn't have a heart attack when it has no free space.

 

[Bidule0hm]

Maybe use the swap space? but that would be a long procedure and walking on eggs I guess... <-- Edit: in fact if you lower the swap space in the GUI and then wipe and resilver the drives one by one it should be good, and without using the CLI :) any confirmation this would work?

If you have some jails you can see if you can delete some of them too.

 

[cyberjock]

You won't even be able to wipe and resilver since you won't have the free space for the transaction that says "adding a disk". So you'd still need to delete some .system dataset files first. :p

Basically the OP is in untested waters. It's gonna be tricky to get out of the situation, and there's no guarantee there is an exit strategy that isn't extremely time consuming (resilvering every disk is going to potentially take days and days) and/or won't require spending money.

I do wish you luck though!

If you manage to recover, I'm sure someone else in the future would like to know how you got out of this problem.

Edit: I will say one thing.
I did work with one person that had this happen to them. It went something like this:

1. They bought a new chassis full of drives.
2. They delayed installation for months.
3. zpool hit 100% full and the iscsi device dismounted very uncleanly.
4. I stopped all server services and destroyed the .system dataset
5. Immediately added the vdev to the zpool while there was freespace to add the vdev.
6. Rebooted so the .system dataset would be recreated properly.
7. Started the iscsi service.

At this point the user mounted the iscsi disk on their windows server, and a chkdsk was performed. It took something like 3-4 days to run because the consistency of the volume was so broken. They did lose a bunch of files and they realized they shouldn't have done #2. They didn't have a good backup, so the lost data was permanently lost.

 

[Bidule0hm]

Now I wonder what do you do next if you can recover some space?

Basically the OP is in untested waters. It's gonna be tricky to get out of the situation [...]

That's why this is interesting :D