How to put same offline disk back

[zzhangz]

I have 4x2TB Raid Z, I tried to upgrade to 4x3TB Raid Z during holiday. What I planed to do was as following.
1. Select one current 2TB drive offline. shut off, replace this drive with a new 3TB
2. "Replace" it on Freenas, after 8 hrs rebuild, it's 3x2tb + 1x3tb Raid Z Array, with same total space about 6TB.
3. Repeat 1-2, until replaced all other 3 2tb drives to 3tb, then total would be about 9TB.

Actually I did same thing before when I upgraded 4x1TB to 4x2TB

Now nightmare happened.
I succefully replaced the 1st 2TB drive with a 3TB drive.
Then I was replacing 2nd 2TB, during the resilvering, the 1st 3TB drive which I had replaced was dead!
So I lost all data. On my hand I still had 4 original 2TB disks. During the whole resilvering process, I didn't write or change any files.

I'm wondering that if I could put original 4 2TB disks back to rebuild the array.

Thanks

 

[cyberjock]

I don't think there is a recovery option. Now you know why I never trust mirrors and RAIDZ.

I'd wait until one of the more knowledgable guys comment. Be careful what you do on your own without the advice of one of our pros since you may cause irreversible damage to the zpool.

I believe if you have disconnected the drives from the zpool you can't reconnect them. So the real question may be if you actually disconnected the drives.

Just don't panic and do anything crazy. If you lost your data you'll have plenty of time to panic after all options are exhausted. You do have backups, right?(I've asked this question 3-5 times in the last week.. wish I knew when people would learn..)

 

[bollar]

I'm with noobsauce80. Don't do anything with the array until you hear your options.

I doubt the original drive can be returned to the array. I'm sure the ZFS file database has changed significantly.

My guess is that recovering the failed drive will be your best option. If the drive will spin up, maybe a pro could make a block copy that you could use as a replacement and ZFS will be none the wiser.

 

[zzhangz]

Unfortunately I don't have backup and I have disconnected the drives, as well as the failed 3TB drive.

 

[Stephens]

I think I'll enjoy this thread (sorry OP), mostly because it's a learning opportunity and it has already caused me to think about a few things. For instance, why wouldn't just putting back in the old drives restore the pool? bollar mentions the "zfs file database" being different. Does that refer to the FreeNAS db, which does keep information about the pool? That won't really matter as you can go through some series of steps where you could just reimport the pool. But that just points out that ZFS itself keeps a database of what the pool looks like (which hard drives are part of it, etc.). Where is this stored? Since the ZFS pool is separate from OS, etc., it's actually stored on the disks themselves.

So that led me to some googling and reading, focused on where the metadata is stored. That led me to here, which provided some good information. If I needed to resolve this, I'd look into what information the uberblocks contain and what exactly identifies disks as part of a pool. Then I'd look at which ZFS (and other) commands could be used to recreate the original all-2TB pool.

I'm looking forward to seeing how this plays out.

@zzhangz: I'll chime in on the advice not to do anything until you really understand what you're doing. Also, did you run SMART long tests on your 3TB drives before using them? Maybe I should petition to get that advice added to the documentation or FAQ or something.

 

[bollar]

I think I'll enjoy this thread (sorry OP), mostly because it's a learning opportunity and it has already caused me to think about a few things. For instance, why wouldn't just putting back in the old drives restore the pool? bollar mentions the "zfs file database" being different. Does that refer to the FreeNAS db, which does keep information about the pool? That won't really matter as you can go through some series of steps where you could just reimport the pool. But that just points out that ZFS itself keeps a database of what the pool looks like (which hard drives are part of it, etc.). Where is this stored? Since the ZFS pool is separate from OS, etc., it's actually stored on the disks themselves.

So that led me to some googling and reading, focused on where the metadata is stored. That led me to here, which provided some good information. If I needed to resolve this, I'd look into what information the uberblocks contain and what exactly identifies disks as part of a pool. Then I'd look at which ZFS (and other) commands could be used to recreate the original all-2TB pool.

Nice link -- From my perspective, we're trying to take the system back to a state before a resilvering began, so a lot has changed to the file system since that good disk was removed from the array. My impression of ZFS is that it's just not going to be possible to take the current metadata and relate it back to the drive. Metadata was the word I was looking for, but I kept coming up with File Allocation Table, which I figured would just confuse people more.

 

[zzhangz]

I did NOT do smart long test before using the new disk, that's my fault!

 

[cyberjock]

I did NOT do smart long test before using the new disk, that's my fault!

Even long tests can't always find all of the problems. Check out this article titled "Ask Slashdot: Do You Test Your New Hard Drives?"

If you filter out all +3 and above comments you'll see that no matter what test you do nothing is 100% and even among geeks the "right" tests to perform varies widely from "nothing at all" to "I run program a, b, and c".

You may have been able to perform the long test for years without a problem, then suddenly have the disk fail when you tried to use it the first time. The best advice is to not trust a disk until you've got a backup :P

 

[Stephens]

@noobsauce80:

Nothing is 100%, not even backups. I've seen backups that have been tested for restorability be bad at the time they're needed. That doesn't change my recommendation to do backups. What we're talking about when I tell everyone to run SMART long tests on new drives is playing the odds. We're trying to figure out what we can reasonably do to improve our odds. Doing a long SMART test before you populate a drive with data greatly improves your odds of weeding out DOA and almost-DOA drives. Greatly. I read as much of the thread you linked to as I had the time for, but I didn't see anyone reputable suggesting you shouldn't test the drives. After all, if the drive is faulty going in, why wouldn't you want to know that up front before you start populating it? The question isn't, "Does SMART guarantee you have a perfect drive?" The question is, "Does SMART (and the associated long test) improve your odds of knowing if you have a faulty drive?" The answer seems almost unquestionably, "Yes," which is why I feel so strongly about it as to suggest it might be worthy advice to include in the documentation. Not everyone using FreeNAS is an IT guru and stuff like MiniDLNA is catering to a crowd that may be proficient users, but not necessarily detailed hardware and software tech geeks.

@bollar:

I think we're all in agreement on the goal, but I don't have enough metadata layout information (and time) available to say if it's possible. I don't know in totality what information needs to be written to during a resilver. For instance, if you have a 4-disc RAID-Z1 represented by 2TBa, 2TBb, 2TBc, and 2TBd, and replace 2TBa with 3TBa, then let it resilver, what does ZFS need to write to 2TBb, 2TBc, and 2TBd? If zzhangz hasn't written to the disks outside of the resilver, I can see where ZFS resilver might not have to write much (if anything) to the static drives (those not being upgraded at the moment). But I wouldn't bet a paycheck on it and I don't know how it works. I'm curious to find out.

If nobody knows the answer, it's something that can be tested. WARNING: Test should be on a disposable NAS as data loss can occur. Build a 3-drive RAID-Z1 with smaller drives, replace the 1st drive with a larger drive, let it resilver, then remove the larger drive, put the smaller drive back in and force an import (Paleon would have more details on exact steps to try), then see what happens.

As I said, I'm really curious to see how this works out.

 

[cyberjock]

Stephens - If you read that post there were people claiming disks were doing a click of death DURING a SMART test and SMART was still reporting the disks as good.

Sure, doing any test is better than no test, but that doesn't mean it will help in a statistically significant way.



I think that once you "disconnect" the drive from the zpool there is no chance to reimport it.

 

[paleoN]

I don't think there is a recovery option.

There may be a recovery option assuming the following is complete & true:

Then I was replacing 2nd 2TB, during the resilvering, the 1st 3TB drive which I had replaced was dead!

There is one drive that failed so the pool should still be alive, depending.

zzhangz, from a SSH session as root paste the output of:

Code:

zpool status -v

camcontrol devlist

glabel status

gpart show

Use [code][/code] tags to preserve the formatting.

 

[bollar]

I think we're all in agreement on the goal, but I don't have enough metadata layout information (and time) available to say if it's possible.

I think there may be ZFS super-humans who can pull this off -- like this guy: http://mbruning.blogspot.com/2009/12/zfs-data-recovery.html

I agree, this would be interesting. ServerFault.com has a couple of threads about these types of recovery, but none that I found that had a satisfying resolution.

 

[Stephens]

@paleon:

If I understand correctly, I think it's 2 failed drives as the pool currently exists. 3TBa is dead, but the 2TBa that used to occupy its slot is still available. 3TBb hasn't been completely built as it was in the process of resilvering when 3TBa died. But 2TBb that used to occupy its slot is still available. It doesn't seem there's anyway to recover the 4-disc RAIDZ1 pool with the 2 3TB drives in it. The question will be if there's any way to recover by going back to the old 2TB drives.

@zzhangz:

Don't get confused by my reply to Paleon. Do what he said.

@bollar:

Good link. I went ahead and e-mailed him a link to this thread. Who knows? Maybe he's in the holiday spirit and will provide some advice.

 

[paleoN]

It doesn't seem there's anyway to recover the 4-disc RAIDZ1 pool with the 2 3TB drives in it.

Yes, not with the 2 3TB drives.

3TBb hasn't been completely built as it was in the process of resilvering when 3TBa died. But 2TBb that used to occupy its slot is still available.

This is the key piece. If accurate, the 2TB is still part of the pool until resilvering is complete. We'll have to wait and see what zzhangz comes back with.

Good link. I went ahead and e-mailed him a link to this thread. Who knows? Maybe he's in the holiday spirit and will provide some advice.

Doubtful. See the third paragraph of this entry.

 

[zzhangz]

I tried to put the 1st 2TB and /or 2nd 2TB back and auto import, unsuccessful. How to do "force import"?

@noobsauce80:

Nothing is 100%, not even backups. I've seen backups that have been tested for restorability be bad at the time they're needed. That doesn't change my recommendation to do backups. What we're talking about when I tell everyone to run SMART long tests on new drives is playing the odds. We're trying to figure out what we can reasonably do to improve our odds. Doing a long SMART test before you populate a drive with data greatly improves your odds of weeding out DOA and almost-DOA drives. Greatly. I read as much of the thread you linked to as I had the time for, but I didn't see anyone reputable suggesting you shouldn't test the drives. After all, if the drive is faulty going in, why wouldn't you want to know that up front before you start populating it? The question isn't, "Does SMART guarantee you have a perfect drive?" The question is, "Does SMART (and the associated long test) improve your odds of knowing if you have a faulty drive?" The answer seems almost unquestionably, "Yes," which is why I feel so strongly about it as to suggest it might be worthy advice to include in the documentation. Not everyone using FreeNAS is an IT guru and stuff like MiniDLNA is catering to a crowd that may be proficient users, but not necessarily detailed hardware and software tech geeks.

@bollar:

I think we're all in agreement on the goal, but I don't have enough metadata layout information (and time) available to say if it's possible. I don't know in totality what information needs to be written to during a resilver. For instance, if you have a 4-disc RAID-Z1 represented by 2TBa, 2TBb, 2TBc, and 2TBd, and replace 2TBa with 3TBa, then let it resilver, what does ZFS need to write to 2TBb, 2TBc, and 2TBd? If zzhangz hasn't written to the disks outside of the resilver, I can see where ZFS resilver might not have to write much (if anything) to the static drives (those not being upgraded at the moment). But I wouldn't bet a paycheck on it and I don't know how it works. I'm curious to find out.

If nobody knows the answer, it's something that can be tested. WARNING: Test should be on a disposable NAS as data loss can occur. Build a 3-drive RAID-Z1 with smaller drives, replace the 1st drive with a larger drive, let it resilver, then remove the larger drive, put the smaller drive back in and force an import (Paleon would have more details on exact steps to try), then see what happens.

As I said, I'm really curious to see how this works out.

 

[zzhangz]

@paleon:

If I understand correctly, I think it's 2 failed drives as the pool currently exists. 3TBa is dead, but the 2TBa that used to occupy its slot is still available. 3TBb hasn't been completely built as it was in the process of resilvering when 3TBa died. But 2TBb that used to occupy its slot is still available. It doesn't seem there's anyway to recover the 4-disc RAIDZ1 pool with the 2 3TB drives in it. The question will be if there's any way to recover by going back to the old 2TB drives.

You are absolutely correct.

 

[zzhangz]

"This is the key piece. If accurate, the 2TB is still part of the pool until resilvering is complete"
I don't think so. I "offline" the 2TB drive, so it's not part the pool any more. I didn't see an option to put offline drive back. One chance is "replace", but it's resilvering, not simply "re-join" the pool again

 

[paleoN]

I don't think so. I "offline" the 2TB drive, so it's not part the pool any more.

You're wrong. A device that has been offlined is still part of the pool.

You won't be able to fix this via the GUI. Run the commands I requested above and I can likely help you get your pool back.

 

[zzhangz]

Code:

zpool status -v

camcontrol devlist

glabel status

gpart show

Use [code][/code] tags to preserve the formatting.

I put the 2nd 2TB drive back and run the following command.
The server now has 3 2TB drive.

Code:

zpool status -v
no pools available                                                              

camcontrol devlist
<ST2000DM001-9YN164 CC96>          at scbus0 target 0 lun 0 (pass0,ada0)        
<Hitachi HDS5C3020ALA632 ML6OA180>  at scbus1 target 0 lun 0 (pass1,ada1)       
<Hitachi HDS5C3020ALA632 ML6OA580>  at scbus2 target 0 lun 0 (pass2,ada2)       
<USB DISK 2.0 0403>                at scbus6 target 0 lun 0 (pass3,da0)         

glabel status
                                      Name  Status  Components                  
gptid/903af3f2-64a4-11e1-84d5-6c626d384807     N/A  ada0p2                      
gptid/8fb92911-64a4-11e1-84d5-6c626d384807     N/A  ada1p2                      
gptid/8ef6e986-64a4-11e1-84d5-6c626d384807     N/A  ada2p2                      
                             ufs/FreeNASs3     N/A  da0s3                       
                             ufs/FreeNASs4     N/A  da0s4                       
                    ufsid/4fd8a9a73e516cd1     N/A  da0s1a                      
                            ufs/FreeNASs1a     N/A  da0s1a                      
                            ufs/FreeNASs2a     N/A  da0s2a                      

gpart show
     4194432  3902834703     2  freebsd-zfs  (1.8T)                             
                                                                                
=>        34  3907029101  ada2  GPT  (1.8T)                                     
          34          94        - free -  (47k)                                 
         128     4194304     1  freebsd-swap  (2.0G)                            
     4194432  3902834703     2  freebsd-zfs  (1.8T)                             
                                                                                
=>     63  3981249  da0  MBR  (1.9G)                                            
       63  1930257    1  freebsd  (942M)                                        
  1930320       63       - free -  (31k)                                        
  1930383  1930257    2  freebsd  [active]  (942M)                              
  3860640     3024    3  freebsd  (1.5M)                                        
  3863664    41328    4  freebsd  (20M)                                         
  3904992    76320       - free -  (37M)                                        
                                                                                
=>      0  1930257  da0s1  BSD  (942M)                                          
        0       16         - free -  (8.0k)                                     
       16  1930241      1  !0  (942M)                                           
                                                                                
=>      0  1930257  da0s2  BSD  (942M)                                          
        0       16         - free -  (8.0k)                                     
       16  1930241      1  !0  (942M)                                           
                                                

 

[zzhangz]

Code:

camcontrol devlist
<ST2000DM001-9YN164 CC96>          at scbus0 target 0 lun 0 (pass0,ada0)        
<Hitachi HDS5C3020ALA632 ML6OA180>  at scbus1 target 0 lun 0 (pass1,ada1)       
<Hitachi HDS5C3020ALA632 ML6OA580>  at scbus2 target 0 lun 0 (pass2,ada2)       
<USB DISK 2.0 0403>                at scbus6 target 0 lun 0 (pass3,da0)         

drivers at ada0 and ada1 are original 2 2TB drive which have never been offlined.
drive at ada2 is a 2TB drive which has been offlined and replaced by a 3TB drive, I put it back to same position.