Replacing Drive with Itself

[c0re]

Hello everyone!

I'm a relative n00b to FreeNAS and am also quite new to FreeBSD. I just built a dedicated 6-bay FreeNAS box that I hope to be able to use in production for some of my clients. Currently, I'm doing everything I can to break RAIDz arrays and discovering the best possible procedures for recovering them so that I can document everything. I'm only a couple of days into playing with ZFS, so please bear with me if what I'm doing is totally wrong.

One such scenario that I have been testing is a very basic one - whereby, for example, a SATA cable is accidentally unplugged from a drive and this leads to a disk falling out of the array. Obviously, in a case like this the hard drive itself is okay and can safely be re-added to the array. Problem is, if I try to replace a drive in-place in the GUI on the "Volume Status" screen, I get the following error:

Error: Disk replacement failed: "invalid vdev specification, use '-f' to override the following errors:, /dev/gptid/f5c99782-74ac-11e2-9834-50465d51d5dd is part of active pool 'pool', "

Okay, I figure the disk is marked as offline so I'm not sure why this happens. Jumping into the shell and trying a basic replace command yields the same results:

zpool replace pool ada0

Alrighty then, lets try destroying all the GEOM labels. Surely then it could not possibly be seen as being part of the array, so I tried running:

gpart destroy -F ada0

Now I go back to the GUI and try to run a replace, but guess what? Exact same error as above. I find this quite strange so I jump back into the shell and try running the replace again, this time manually:

zpool replace pool 1234567890123456789 ada0

Lo and behold, this works! Great, but there is one last problem - in the GUI the drive no longer shows up as ada0p2 but rather ada0. This is of course because FreeNAS did not get a chance to format the drive on its own and I allowed the zpool command to do this for me.

I have found a workaround to fix this as well, but it isn't so pretty. I then have to gpart destroy ada0 yet again. Then at that point, I can replace it through the GUI and everything runs smoothly. I don't like this very much though because in a real situation, this actually means that to get FreeNAS to partition the drive on its own, I need to resilver the whole array twice. Very not cool.

Questions:

A) Surely, there is a way to avoid this. I don't mind doing everything through the shell, provided that the manual procedure is doing everything "right"
B) Could it possibly be due to the fact that FreeNAS is not handling 4k disks properly? All disks in this array are 3TB 4k advanced format drives

Thanks so much for reading, and hopefully somebody who knows way more than I about dealing with FreeBSD/ZFS/RAIDz can chime in!

 

[c0re]

bumpity bump? :o

 

[cyberjock]

Disclaimer: Ok, just thought I'd say this before anything else.. (I think you know this but I'll say it anyway...) do not unplug drives from a zpool if you value the data on the zpool. :)

With that out of the way, sector size has no bearing on your issue at all. Also, if you are really concerned with sata cables being unplugged from a running system, you are in serious trouble. While your test is good for failing a disk so you can experiment with recovery if you consider an accidental unplugged cable as a possible failure mode you are doing something terribly wrong. As a general rule a running system should not be opened at all except for non-invasive visual inspections. This means no touching cables. I know.. alot of us break that rule.

The issue(at least my understanding of it) has to do with the hot plugging of drives and their availability to ZFS. When you plug the drive it will be out of sync with the zpool. The proper situation for an unplugged drive would be to unmount and remount the zpool(if your system supports hot-plugging drive) or a shutdown and reboot(for cold-plugging drives). The drive can and should be readded to the zpool because it was part of the zpool. I always do a reboot because I've found that hot-plugging doesn't work 100% on all hardware that claims to support it. Unlike hardware RAIDs where a drive must be readded to the RAID after a lengthy resyncing to the rest of the RAID, ZFS will let you immediately bring that drive right back into the pool. It obviously will be out of sync with the rest of the zpool so you will need to perform a scrub.

If you are trying to test a disk as being failed you are better off wiping the beginning and end of the disk to make the disk look brand new. If the drive appears to be part of the zpool(or was ever part of the zpool) the GUI won't let you "readd" the disk because you should be doing an unmount and remount as discussed above.

Something else I'd like to point out since you mentioned clients...I wouldn't do anything less than a RAIDZ2 if you are selling this as a solution for someone with a backup system and backup solution with ZFS snapshots. There's plenty of anecdotal evidence from forum users that if 1 disk fails all it takes is another disk to have anything less than 100% readable and correct data for you to see file corruption since there would be no additional redundancy. RAIDZ2 provides exceptional extra protection because it allows for bit-rot and disk read errors without any data loss. Also, backups are a MUST. Read up on snapshots as those are the best way to go for keeping 2 machines in sync.

 

[c0re]

Hi cyberjock,

Thanks for the thoughtful reply. I agree that if having drives unplugged is a concern, you have bigger problems. That said, my goal is to actually test as many failure modes as a I can "just in case" so at least I know the right procedures. Hey, you can never be too careful! As far as maintaining data integrity, I also totally agree with you on RAIDz2, especially with drives reaching the capacities they are now. From what I've read, it seems like the best way to make sure your data is safe and in fact accurate is to use 2 disk redundancy and have a good scrub schedule in place.

After having reread your post a few times, from what I understand, a disk being disconnected should be treated in the same way as any case where you believe the state of one disk has become inconsistent for any reason, correct? With this being the case, would you recommend the following procedure for a disk (ada0) suspected to be in an inconsistent state:

1) Offline the device - "zpool offline -t pool ada0p2"
2) Shut down the system, verify all connections
3) Start up system and verify that disk is online again
4) Perform a scrub to clear inconsistencies

To be sure, I probably don't fully understand the difference between a scrub and resilvering. As I understand it, a scrub is normally used as for maintenance, whereas a resilvering is almost the same but is what takes place when you replace a failed disk.

In a case such as this one, will the scrub safely repair all inconsistencies? I do remember trying to follow a similar procedure to this, and ended up finding 3 CRC errors on the zpool status for the drive that was temporarily lost. What concerns me here is that, to see if the data had reached a consistent state following the scrub, I decided to kill a second (different) drive in a RAIDz1 array following the scrub that I thought would "repair" the inconsistencies.

In case its not clear the procedure I followed, it looked like this:

1) Unplug ada0
2) Write new data to array
3) Offline ada0
4) Shutdown, plug in ada0
5) Bring ada0 back online
6) Perform scrub (CRC errors reported on zpool status for ada0)
7) Unplug ada1
8) Write new data to array
9) Offline ada1
10) Shutdown, plug in ada1
11) Bring ada1 back online
12) Perform scrub
13) Data Loss: Data written to zpool following loss of ada0 and ada1 is largely lost/damaged

My major concern here is that that procedure clearly didn't actually bring ada0 into a consistent state. If it had, then there should have been no data loss. Or am I the one that is completely lost? lol

Also, point taken on clearing the beginning and end of the drive to better simulate drive failures. I'll try that as well!

Thanks again for reading!

 

[cyberjock]

If you followed steps 1 through 16 and actually allowed the scrub to finish completely you should have had no data loss. Since you had no data loss removing and readding ada0 but did with ada1 then I'd think that ada 0 may have problems. You didn't mention your zpool type, but if you used RAIDZ1 then as soon as a disk went offline any disks with any bitrot, bad sectors, or read errors will result in corruption. This is why I always use RAIDZ2 or RAIDZ3. In my opinion, a RAIDZ1 is only a hail mary pass if a disk fails. Because of how big disks are getting and their typical error rate even when a disk is in perfect working order you can have errors. Back in 2008 or so RAID5 was labeled as "dead" by some whitepapers because RAID5s were considered to not be recoverable from a failed disk any longer.

 

[Radu]

I have done the test with one of my drives. Offline from GUI in Volume Status. Pull the disc out, put it back in and Replace form GUI volume status menu. ERR: Disk replacement failed part of active pool must use -f

I have tried zpool online tank0 id, the disk was online but unavailable. After reboot the problem was solved. I guess this was similar to Deatach tank0 and auto import it again @cyberjock ? ... instead of reboot.
After off lining the disk and the failed replace, if you reboot, the disk will not be added tot the pool, because it is in offline mode and not replaced(failed replace). If you online the disk with the zpool online tank0 id and then reboot the disk is accepted back.

I did this on a brand new raidZ2 pool and after that the output of the zpool status -v tank0 is: see the attached pictures.
cyberjock can you tell me how to interpret the output of the command?
what does the "1" means between all the zeros from the first column and the status and action message. The disk that i have put back was resilvered? Why is da1p2 written between the gptid lines? What does it mean?

 

[cyberjock]

The 1 means you had a checksum failure(potentially silent corruption).

da1p2 means disk da1 partition 2. Did you replace the disk from the GUI in accordance with the manual or did you do it from the CLI?

 

[Radu]

My pool was ok, and i wanted to test the replacement feature. First i off lined the disk from GUI after that i pulled it out and put it back in. From GUI replace command and the error was that the disk is already a member and use -f ...
I tryed after this to online the disk from cli and the disk status became unavailable... after many online offline commands i have tried to online it and reboot. After that the disk was accepted back. I want to know why did i needed the reboot, hence my last port questions form this thread... detach and import of the pool works for this? I have read that if the disk is erased it is accepted back with no problem and resilvered. Is there any other way to but back the good disk without the reboot?

The output of the command zpool status -vx tank0 , at first, it was not like this, all the list was with gptid's not the drive id's. Why after the disc change appeared like that?

 

[cyberjock]

The GUI adds the disk back as gptids.. you probably didn't do that when you did your CLI stuff. ;)

It's just as you said.. once you remove a disk from the pool it must be wiped before you can use it in another pool. That's the only way to add it back without it causing problems.

 

[Radu]

mmm ... i removed a working disk from a working pool ... and i just want to put it back without restart or wiping it out ... eventually without resilvering because the disk it's fine. Can i do this somehow? ZFS can do that?

 

[cyberjock]

No because you'd never deliberately remove and then add a disk back without resilvering...

What you are doing you'd never actually do in a real system.

 

[TheSmoker]

mmm ... i removed a working disk from a working pool ... and i just want to put it back without restart or wiping it out ... eventually without resilvering because the disk it's fine. Can i do this somehow? ZFS can do that?

For testing you have always need to do:
- offline
- wipe
- replace
- resilver

There is no way around it. You need to "trick" zfs that your drive is a new drive.

You should start reading that manual... As i've told you so many times before ....