Replace a disk

[Arun Gupta]

I am running FreeNAS-9.2.0-RELEASE-x64 (ab098f4). I have two disks in a ZFS volume. There is no RAID. I want to replace both these disks with higher capacity disks. Here is what I did.
a) Did a volume scrub. No errors.
b) Tried to offline one disk. Got an error:

cannot offline gptid/1709a8b2-0398-11e2-9e36-000c29035f66: no valid replicas

I checked on this forum/Wiki and everyone seems to be running in some RAID configuration which I do not have. Question is how do I replace these disks?


Thanks,
Arun

 

[cyberjock]

You can't replace them if you have insufficient replicas in your vdev. That's why you are getting the warning. You are probably running 2 disks in a striped configuration. This means if either disk fails you lose all of your data.

 

[Arun Gupta]

You can't replace them if you have insufficient replicas in your vdev. That's why you are getting the warning. You are probably running 2 disks in a striped configuration. This means if either disk fails you lose all of your data.

Yes, it does show striped configuration. So if my understanding is correct, there is no option but to lose all data just to replace disks?

Thanks...

 

[cyberjock]

You could go from striped to mirrored vdevs, then remove the smaller mirrors. But that is all going to be done from the CLI. To be honest, I'd just make your new pool in the GUI and do a local copy command from one pool to the other or do zfs replication from the old pool to the new one.

I'd highly highly recommend you ditch the striped configuration and go with something that's got redundancy. Multi-disk stripes are just stupid and dangerous for all but the most unimportant of data.

 

[Arun Gupta]

This is a non-critical system which I use at home for testing. Data is not critical at all, space is. I currently use only iSCSI feature and except for iSCSI file extents, there is no data. This is the reason why I chose striped configuration.

Anyways, here is how I replaced the drives.
a) Enabled FTP on FreeNAS.
b) FTP'ed all the iSCSI extents to my desktop.
c) Detached the volume.
e) Shutdown FreeNAS. Removed the old drives, installed new ones.
f) Started FreeNAS. Created a volumes/datasets of same name as before.
g) FTP'ed all extents back into the directories where they were before.

Everything works normally except that I ran into a what seems to be a problem with iSCSI. I will post separate question for that. Thanks for your reply. It gave me some direction to proceed.

 

[cyberjock]

This is a non-critical system which I use at home for testing. Data is not critical at all, space is. I currently use only iSCSI feature and except for iSCSI file extents, there is no data. This is the reason why I chose striped configuration.

If the data is so disposable why did you even create this thread? Simply delete your current pool and make a new one. Put your data where your mouth is. :P

 

[Arun Gupta]

Maybe it was my fault for not making situation clear.

I routinely install and configure and test Oracle RAC/ASM and databases on VMs using iSCSI disks, been doing it since before FreeNAS 0.7 was released. When I create an iSCSI extent, it serves as a raw device in Oracle VM. Not sure if you are familiar with RAC/ASM etc. but if an iSCSI extent is deleted, it takes me lot of time to re-configure the Oracle setup as it amounts to database losing a disk. Hence I try to save data if possible. This is the reason why I asked this question. In all the years that I have been using FreeNAS, I have never tried to change the disks. However, recently, I started getting warning that disks may fail, so I decided to find out if there is a way to save data and still be able to replace disks. It would save me several hours of re-configuring the Oracle VMs.

Worst case scenario, I lose data. The important thing is that data loss will not cause any monetary loss or expose me to any liability. When I say data is disposable, this is what I mean: No financial or any other liability.

Need to save data: It would save me hours of re-configuring time of Oracle VMs. So, if possible, try to save data.
Data is disposable: Worst case scenario, if data is lost, it does not expose me to any liability of any kind. I would only spend hours re-configuring Oracle VMs.

Hope you understand my situation. If you still feel that the question was unreasonable, please let me know.

 

[cyberjock]

I never thought your question was unreasonable. I just wanted you to realize you are telling me your data isn't important to use redundancy, but important enough to not lose from creating a new pool. Well, those two are exclusive of each other. Honestly, if you'd have to spend some time reconfiguring the VMs, then I'd consider redundancy a win-win situation. Disks fail. It's not a matter of if, its a matter of when. For you, the first time a disk starts having errors you might as well destroy the pool. Because any corruption in ZFS that isn't fixable with redundancy(which you have none) will result in a failure of the entire pool. More than likely you'll go to bed and all will be fine. You'll wake up and your data will be irretrievably lost.

 

[Arun Gupta]

I am a database admin by profession so I very well realize the importance of disk redundancy, backups, off-site backup storage, DR etc. I do these on a daily basis. However, in a home setup scenario, I do not have enough financial resources to invest in RAID. The intent of my question was simple. Is it possible to save data and still be able replace one working disk with another in striped configuration? If yes, I would have appreciated the answer. If no, I would have delete the volume and created a new one.

Through trial and error, I have learned that yes, I can save the extents by ftp'ing them to my desktop and delete and re-create the volume. This has saved me lot of time and aggravation. Yes, one day, the disk will crash. That day, I will not ask about replacing the disk. I know that I do not have redundancy. I will bite the bullet and re-create the volume. Spend hours re-configuring the VMs. But today was not that day. Today, I was just replacing one working disk with another. Before destroying the volume, I just thought that forum has more experienced users so why not ask.

Unfortunately, for some unknown reason, I was called a liar and accused of making up excuses, also my setup was labeled as "stupid". If striped configuration is so bad, maybe FreeNAS should not offer it as an option.
If I caused any offense, I apologize. This is my last visit to this forum. Thank you.

 

[cyberjock]

We've got plenty of people that use 5 fast small disks in a striped array as a temp storage space or working space. It has completely valid uses for some people that need extremely high speed temporary storage and a loss of the pool suddenly means no data loss at all.

The whole reason why FreeBSD and Linux are so superior to Windows is because they let you, the admin, make the decisions for your server. For better or for worse. It's not the OSes job to keep you from doing anything that causes data loss.

 

[tzd]

if an iSCSI extent is deleted, it takes me lot of time to re-configure the Oracle setup as it amounts to database losing a disk. Hence I try to save data if possible.

Need to save data: It would save me hours of re-configuring time of Oracle VMs. So, if possible, try to save data.

And hence you really would have benefited from having at least one redundant disk in your setup.


Sent from my iPad using Tapatalk

 

[Arun Gupta]

And hence you really would have benefited from having at least one redundant disk in your setup.


Sent from my iPad using Tapatalk

Actually you are supplying solution to a problem which does not exist.

My question was simple: how to you replace a working disk with another? From my perspective, the solution was simple. I have two disks in a striped configuration. FreeNAS should have been able to move all data to one disk (space permitting) and allowed me to remove the second disk and replace it. Every Enterprise level storage solution that I work with, allows me to do so. So I was kind of surprised that FreeNAS has this serious limitation.

The question about whether RAID is needed or not does not even arise. My question was twisted way out of context by suggestions that if I do not have RAID, I am doing something wrong. I never asked about RAID. FreeNAS has a serious limitation if it does not allow removal of a working disk from a multi-disk striped configuration. It does not matter what the disks are used for. The issue is FreeNAS features. Instead of admitting this limitation, I was criticized for not having RAID. It does not make any sense.

I miss the FreeNAS 0.7 forums where the intent of contributors was to resolve a problem instead of criticizing someone's setup and calling posters liars.

 

[Dusan]

FreeNAS should have been able to move all data to one disk (space permitting) and allowed me to remove the second disk and replace it. Every Enterprise level storage solution that I work with, allows me to do so. So I was kind of surprised that FreeNAS has this serious limitation.

This is a ZFS limitation. The only way to "replace" a disk in a ZFS stripe is to connect a new drive, attach it as a mirror to the disk you want to remove, wait for the mirror to resilver and detach the old drive. However, there is no GUI support for this in FreeNAS, you would have to do in from the command line.

 

[Arun Gupta]

This is a ZFS limitation. The only way to "replace" a disk in a ZFS stripe is to connect a new drive, attach it as a mirror to the disk you want to remove, wait for the mirror to resilver and detach the old drive. However, there is no GUI support for this in FreeNAS, you would have to do in from the command line.

Many Thanks!!! This is exactly the kind of information I was looking for. Can you please point me to some documentation on the command line steps to do this? I am very familiar with Linux (RedHat) and have setup SSH access to FreeNAS. Since my setup is not critical, I can experiment.

Off topic: A person of your name (Dusan) used to be very active on Oracle forums. Are you the same person by any chance?

Thanks...

 

[Dusan]

Can you please point me to some documentation on the command line steps to do this?

The commands you are looking for are zpool attach and zpool detach. Find the detailed documentation here: http://www.freebsd.org/cgi/man.cgi?query=zpool
You will also notice that FreeNAS creates a swap partition on every data drive and uses GPTIDs instead of device names. The commands to partition disks are gpart create and gpart add (http://www.freebsd.org/cgi/man.cgi?query=gpart). You will find lot of relevant information (both on partitioning and zpool attach) in this thread: Create zfs mirror by adding a drive?
The zpool detach part is fairly easy. Just wait for the new drive to resilver (run zpool status to check on the progress) and zpool detach the old one.

Off topic: A person of your name (Dusan) used to be very active on Oracle forums. Are you the same person by any chance?

Nope, the definitely wasn't me :).

 

[Arun Gupta]

The commands you are looking for are zpool attach and zpool detach. Find the detailed documentation here: http://www.freebsd.org/cgi/man.cgi?query=zpool
You will also notice that FreeNAS creates a swap partition on every data drive and uses GPTIDs instead of device names. The commands to partition disks are gpart create and gpart add (http://www.freebsd.org/cgi/man.cgi?query=gpart). You will find lot of relevant information (both on partitioning and zpool attach) in this thread: Create zfs mirror by adding a drive?
The zpool detach part is fairly easy. Just wait for the new drive to resilver (run zpool status to check on the progress) and zpool detach the old one.

Nope, the definitely wasn't me :).

Thank you so much. Really appreciate the help.

Best regards,
Arun