HELP!!! I typo'd a command and I think I am screwed

[freebsdsa]

I know, I know... Use the GUI... Lets get past that part and on the helping.

Here is the gist of my problem...

Working on a FreeNAS 11.1 server with three pools on it. I just setup the third pool and was getting some zvol settings dialed in via command line to match what we do on the other two pools. I use the command line as I want to setup the volsize greater than 80% of storage size and the GUI does not let me... I had just set the volsize and wanted to look at and compare all the other settings. I hit the up arrow to do a "zfs get all" and I adjusted the command line to point to one of my production datasets. I was not paying attention and I guess I hit the down arrow on my keyboard, I change to command to point to one of the current prod zvols and hit return. It was then I realized I set/shrank the volsize from 200T to 25T on the prod zvol. The command has not returned yet and things are starting to look bad. I can not cancel the command "cntrl-c". So how screwed am I? Will I lose data?

 

[Chris Moore]

Sorry. Let us know how it works out.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[fracai]

I wouldn't think you could set the volsize to lower capacity than it's already using. I think I would give it more time to complete (though at this point I'd be done waiting) and then be tempted to restart the system and see if I can rollback the whole pool if necessary once it comes back up. There's the risk that the pool doesn't import, but if the command still hasn't returned and you can't kill it you might be in a deadlok anyway. I can't claim to have any real insight here though, good luck

 

[kdragon75]

Do you have snapshots that you can revert to? If you do have data loss, you would need to set the zvol back to 200T and revert to the snapshot.

 

[freebsdsa]

Update... So we let the command go till we just could not wait for any longer (an hour after my post) as we were seeing VMs start having more and more issues. I rebooted the server and it came back up without any issues. ZFS did not complain at all. I checked and the zVol was now 25T and about 50% of the VMs hosted on that storage no longer functional. I did change the zvol back to 200T for volsize, but we did not have any snapshots of it. "We" prefer to use veeam backups and veeam replicas instead. It took about 5 hours but we got everything back online. BUT I am looking forward to "zpool checkpoints" in 11.2. With that, we would do a checkpoint before doing any work on ZFS. Then if something bad happens, roll it back and stay safe.

 

[diskdiddler]

So - just to clarify did you just manage to shrink a volume to less space than it even had available? If so, I'd have thought a bug could be filed for that?

 

[Chris Moore]

It might be a nice feature if the command returned an error if you try to shrink the container to a size that is smaller than the data in the container. After all, it is discarding data if it runs the command. However, it is one of those Unix / Linux ROOT privileges, if you say do, it does, even if it would destroy the system. The expectation is that you know what you are doing and have given due consideration to it before issuing the command.

 

[freebsdsa]

I am taking a two-pronged approach here.

So - just to clarify did you just manage to shrink a volume to less space than it even had available? If so, I'd have thought a bug could be filed for that?

Yes I am going to file a bug report with FreeBSD itself. Asking if they could "fix" the bug. See my suggested fix below.

It might be a nice feature if the command returned an error if you try to shrink the container to a size that is smaller than the data in the container. After all, it is discarding data if it runs the command. However, it is one of those Unix / Linux ROOT privileges, if you say do, it does, even if it would destroy the system. The expectation is that you know what you are doing and have given due consideration to it before issuing the command.

I have already posted a feature request with the OpenZFS guys to see if someone would like to adjust the behavior to not allow you to set the volsize smaller than the data contained in the volume. Like comparing the set volsize value to the "referenced" value and if less than exit with error. In the spirit of "I am root and I know what I am doing" type needs, maybe a -f to force the action could keep the behavior the same as is now for those that might want it that way.

 

[fracai]

What I don't get about the command actually reducing the volsize is "how does it decide what data is destroyed?" @freebsdsa says a few VMs were non-functional. Did they become functional after increasing the size again? Did they have to be restored or recreated? I'd expect that decreasing the volsize would maybe just truncate some block list and with such a dramatic decrease in size I'd think that would lead to almost everything losing data, not just half your VMs. Maybe you got lucky? Or the lost data was in more recent VMs?

 

[Chris Moore]

In the spirit of "I am root and I know what I am doing" type needs, maybe a -f to force the action could keep the behavior the same as is now for those that might want it that way.

I imagine that this is an example of something that may have been overlooked, because there are other options in the ZFS command set that do work that way. As you say, it can be given the -f option to force the action, but errors out if a condition exists.

 

[freebsdsa]

What I don't get about the command actually reducing the volsize is "how does it decide what data is destroyed?" @freebsdsa says a few VMs were non-functional. Did they become functional after increasing the size again? Did they have to be restored or recreated? I'd expect that decreasing the volsize would maybe just truncate some block list and with such a dramatic decrease in size I'd think that would lead to almost everything losing data, not just half your VMs. Maybe you got lucky? Or the lost data was in more recent VMs?

Ok so this was the fun part. First background. The volume is being shared to VMware via iSCSI. So this was a 200T dataset for VMWare storage. Ok so with that knowledge... I have not figured out how it decided to delete data, but it deleted the data for sure. When I set the volsize back to 200T ZFS still only reported 25T of data in the volume. What was deleted is kind of random looking. Some new machine worked, some old machines worked. The very first server we put on the storage was DOA and some of the most recent machines are working still. I am sure this is related to CoW... I can also tell you that VMFS6 never reported a change in size. All the VM folders were still there and the VMDKs still reported there pre-change size. We copied off a few VMDKs of failed servers. VMFS reported like 30G, but when we copied it to another server via SFTP we ended up with an 8k file instead. The servers we booted and are running we VMotioned off. We decided to reformat the VMFS to make sure it had a stable filesystem on disk and remigrated all our working VMs back on to it. For all the other machines that did not work, we turned on the veeam replica we had for each. Those are currently being storage VMotion back onto the storage and new replicas will be created once done.

 

[freebsdsa]

I imagine that this is an example of something that may have been overlooked, because there are other options in the ZFS command set that do work that way. As you say, it can be given the -f option to force the action, but errors out if a condition exists.

Totally...

 

[fracai]

So strange. Please follow up with anything you hear back from the FreeBSD folks. It wouldn't hurt to file a bug with FreeNAS as iX is, I think, a fairly significant contributor to FreeBSD code in general.

 

[freebsdsa]

So strange. Please follow up with anything you hear back from the FreeBSD folks. It wouldn't hurt to file a bug with FreeNAS as iX is, I think, a fairly significant contributor to FreeBSD code in general.

For sure I will follow up on anything and yes it might be good to get the bug in the FreeNAS tracker as well.