Cannot re-add disk to zpool

[BenTheGuy]

I had a zpool with 3 disks running raidz1, and it was working great. Through a series of stupid moves to fix another issue, I've now got 2 disks working in degraded state, and the 3rd disk formatted and physically back in the server. I figured I'd be able to resilver it and be good to go, but it's not happening. For one, the device is listed as a long number rather than ada1p2 or something similar. When I try to run replace, I get incorrect vdev specification, and it tells me ada1p2 is already in the pool. But if I try to clear ada1p2, it tells me no such device is in the pool. I tried scrubbing, that did nothing.

I've been looking for answers for hours, any help would be much appreciated. I'm a noob other than that I was able to get the intial setup working great for a month now till I started messing with it.

 

[paleoN]

When I try to run replace, I get incorrect vdev specification, and it tells me ada1p2 is already in the pool. But if I try to clear ada1p2, it tells me no such device is in the pool. I tried scrubbing, that did nothing.

This sounds like you are trying to replace the wrong disk. Don't do that. Output of:

Code:

zpool status -v
 
camcontrol devlist
 
gpart show
 
glabel status

Be sure to use [code][/code] tags.

 

[BenTheGuy]

This sounds like you are trying to replace the wrong disk. Don't do that. Output of:

Code:

zpool status -v
 
camcontrol devlist
 
gpart show
 
glabel status

Be sure to use [code][/code] tags.

Here you go:

Code:

  pool: BenStore                                                                
 state: DEGRADED                                                                
status: One or more devices could not be opened.  Sufficient replicas exist for 
        the pool to continue functioning in a degraded state.                   
action: Attach the missing device and online it using 'zpool online'.           
   see: http://illumos.org/msg/ZFS-8000-2Q                                      
  scan: scrub repaired 0 in 1h40m with 0 errors on Wed Sep 11 20:19:18 2013     
config:                                                                         
                                                                                
        NAME                                            STATE     READ WRITE CKS
UM                                                                              
        BenStore                                        DEGRADED     0     0    
 0                                                                              
          raidz1-0                                      DEGRADED     0     0    
 0                                                                              
            gptid/da17ff25-0e6a-11e3-b095-bc5ff4a47cad  ONLINE       0     0    
 0                                                                              
            12824015695766159589                        UNAVAIL      0     0    
 0  was /dev/gptid/da85b2b0-0e6a-11e3-b095-bc5ff4a47cad                         
            gptid/db6caa1f-0e6a-11e3-b095-bc5ff4a47cad  ONLINE       0     0    
 0                                                                              
                                                                                
errors: No known data errors                                                    
 
[root@freenas ~]# camcontrol devlist                                            
<WDC WD20EFRX-68AX9N0 80.00A80>    at scbus0 target 0 lun 0 (ada0,pass0)        
<WDC WD20EFRX-68AX9N0 80.00A80>    at scbus1 target 0 lun 0 (pass1,ada1)        
<WDC WD20EZRX-00DC0B0 80.00A80>    at scbus3 target 0 lun 0 (ada2,pass2)        
<Generic- Multi-Card 1.00>         at scbus4 target 0 lun 0 (pass3,da0)         
<Corsair Voyager 3.0 1.00>         at scbus5 target 0 lun 0 (pass4,da1)   
 
 
=>      63  15687617  da1  MBR  (7.5G)                                          
        63   1930257    1  freebsd  [active]  (942M)                            
   1930320        63       - free -  (31k)                                      
   1930383   1930257    2  freebsd  (942M)                                      
   3860640      3024    3  freebsd  (1.5M)                                      
   3863664     41328    4  freebsd  (20M)                                       
   3904992  11782688       - free -  (5.6G)                                     
                                                                                
=>        34  3907029101  ada0  GPT  (1.8T)                                     
          34          94        - free -  (47k)                                 
         128     4194304     1  freebsd-swap  (2.0G)                            
     4194432  3902834696     2  freebsd-zfs  (1.8T)                             
  3907029128           7        - free -  (3.5k)                                
                                                                                
=>        34  3907029101  ada2  GPT  (1.8T)                                     
          34          94        - free -  (47k)                                 
         128     4194304     1  freebsd-swap  (2.0G)                            
     4194432  3902834696     2  freebsd-zfs  (1.8T)                             
  3907029128           7        - free -  (3.5k)                                
                                                                                
=>      0  1930257  da1s1  BSD  (942M)                                          
        0       16         - free -  (8.0k)                                     
       16  1930241      1  !0  (942M)    
 
                                                                                
=>        34  3907029101  ada1  GPT  (1.8T)                                     
          34          94        - free -  (47k)                                 
         128     4194304     1  freebsd-swap  (2.0G)                            
     4194432  3902834696     2  freebsd-zfs  (1.8T)                             
  3907029128           7        - free -  (3.5k)                                
                                              
 
[root@freenas ~]# glabel status                                                 
                                      Name  Status  Components                  
                             ufs/FreeNASs3     N/A  da1s3                       
                             ufs/FreeNASs4     N/A  da1s4                       
gptid/da17ff25-0e6a-11e3-b095-bc5ff4a47cad     N/A  ada0p2                      
gptid/db6caa1f-0e6a-11e3-b095-bc5ff4a47cad     N/A  ada2p2                      
                            ufs/FreeNASs1a     N/A  da1s1a                      
gptid/a7d66096-1b48-11e3-a4cf-bc5ff4a47cad     N/A  ada1p1                      
gptid/a7e164c7-1b48-11e3-a4cf-bc5ff4a47cad     N/A  ada1p2
 

 

[paleoN]

Through a series of stupid moves to fix another issue, I've now got 2 disks working in degraded state, and the 3rd disk formatted and physically back in the server.

Elaborate on this and why/how is the 3rd disk formatted? Does this server have hot-swap? The old unavailable device was gptid/da85b2b0-0e6a-11e3-b095-bc5ff4a47cad and ada1p2 is now gptid/a7e164c7-1b48-11e3-a4cf-bc5ff4a47cad. If you attempted some sort of cloning a restart may just pick up the changes; depending. You can also manually trigger a resilver. The zpool clear command doesn't do what you think.

 

[BenTheGuy]

Elaborate on this and why/how is the 3rd disk formatted? Does this server have hot-swap? The old unavailable device was gptid/da85b2b0-0e6a-11e3-b095-bc5ff4a47cad and ada1p2 is now gptid/a7e164c7-1b48-11e3-a4cf-bc5ff4a47cad. If you attempted some sort of cloning a restart may just pick up the changes; depending. You can also manually trigger a resilver. The zpool clear command doesn't do what you think.

I needed a drive for a Windows PC right then, it was kind of an emergency situation and I didn't have any other options at the time. Now it's formatted with NTFS. I had assumed, perhaps incorrectly, that Freenas would format and resilver it.

By hot swap do you mean is AHCI enabled? Yes.

Obviously I did not follow best practices with any of this. At this point I just want to get it fixed and make sure I have a better solution on hand next time.

 

[BenTheGuy]

So the correct command to manually trigger the resolver would be...
Zpool replace BenStore ada1p1 ada1p2

?

 

[BenTheGuy]

Just got home and started messing with it again. I swear I've shut it down or rebooted it several times last night while it was in the exact same state it was when I got home, but this time when I rebooted it came up and resilvered and went back to healthy. Why now and not last night? Very strange. But it's working, thank you for your assistance.

 

[paleoN]

Why now and not last night? Very strange. But it's working, thank you for your assistance.

That's an encouraging sign. To be sure it's working:

Code:

zpool status -v

zpool scrub BenStore

I'm interested to see how it's currently listed in the pool and you need to scrub the pool to be sure that disk is correct.

 

[BenTheGuy]

I already have it scrubbing, and it does seem to be repairing some stuff. Here's the output:

Code:

   see: http://illumos.org/msg/ZFS-8000-9P                                      
  scan: scrub in progress since Thu Sep 12 17:00:02 2013                        
        413G scanned out of 1.72T at 253M/s, 1h30m to go                        
        141M repaired, 23.51% done                                              
config:                                                                         
                                                                                
        NAME                                            STATE     READ WRITE CKS
UM                                                                              
        BenStore                                        ONLINE       0     0    
 0                                                                              
          raidz1-0                                      ONLINE       0     0    
 0                                                                              
            gptid/da17ff25-0e6a-11e3-b095-bc5ff4a47cad  ONLINE       0     0    
 0                                                                              
            gptid/4f5ded37-1b52-11e3-a4cf-bc5ff4a47cad  ONLINE       0     0 2.3
2K  (repairing)                                                                 
            gptid/db6caa1f-0e6a-11e3-b095-bc5ff4a47cad  ONLINE       0     0    
 0                                                                              
                                                                                
errors: No known data errors    

 

[paleoN]

Here's the output:

Yet another different GPTID, gptid/4f5ded37-1b52-11e3-a4cf-bc5ff4a47cad? If you had written on the drive I would expect it to need to be repaired.