SOLVED Mixed messages in resilver

[cyberjock]

Yep. Let it rip!

 

[joeschmuck]

Nothing stands out about your drives. All but ada1 and ada3 have a rather high head cycle count but it's not alarming, but then again ada1 and ada3 do not have as many hours on them either. WDIdle can reduce that if you want.

 

[climb2bhi]

WDidle3 was done recently.

Here we go...

 

[climb2bhi]

Detached the "removed" drive, and now I can't replace the offline one as there is nothing in the drop down box.

I think I should have done it the other way.

 

[climb2bhi]

So now I have a offline disk with the only option offered being "replace" and no replacement available in the small window that opens because it is the replacement disk.

 

[cyberjock]

Do a reboot of the server. See if that lets you use the disk.

 

[climb2bhi]

With the drive in. Or since it is "offline" shutdown, pull it, reboot, Shutdown, install it, reboot?

 

[cyberjock]

no, just reboot the server. Nothing else. ;)

 

[climb2bhi]

Exelent. Now I have this...

With ada1 now available in the drop down. Select "Replace disk"?

 

[cyberjock]

Yep.

 

[climb2bhi]

Seems happy.
Now I can sleep.

Code:

Shell
[root@freenas ~]# zpool status                                                                                                     
  pool: NAS                                                                                                                        
state: DEGRADED                                                                                                                   
status: One or more devices is currently being resilvered.  The pool will                                                          
        continue to function, possibly in a degraded state.                                                                        
action: Wait for the resilver to complete.                                                                                         
  scan: resilver in progress since Fri Nov  7 22:01:16 2014                                                                        
        27.5G scanned out of 10.9T at 268M/s, 11h52m to go                                                                         
        4.58G resilvered, 0.25% done                                                                                               
config:                                                                                                                            
                                                                                                                                   
        NAME                                              STATE     READ WRITE CKSUM                                               
        NAS                                               DEGRADED     0     0     0                                               
          raidz2-0                                        DEGRADED     0     0     0                                               
            gptid/00780523-12d1-11e4-94f0-0030483594b4    ONLINE       0     0     0                                               
            gptid/0153dbc6-12d1-11e4-94f0-0030483594b4    ONLINE       0     0     0                                               
            gptid/023861ca-12d1-11e4-94f0-0030483594b4    ONLINE       0     0     0                                               
            gptid/02f20184-12d1-11e4-94f0-0030483594b4    ONLINE       0     0     0                                               
            replacing-4                                   OFFLINE      0     0     0                                               
              8687065241295420711                         OFFLINE      0     0     0  was /dev/gptid/03d9e2a5-12d1-11e4-94f0-0030483
594b4                                                                                                                              
              gptid/44737168-6704-11e4-8f3a-0030483594b4  ONLINE       0     0     0  (resilvering)                                
            gptid/04a1784e-12d1-11e4-94f0-0030483594b4    ONLINE       0     0     0                                               
                                                                                                                                   
errors: No known data errors

Thanks.

 

[climb2bhi]

OK. So 12 hours later I was here:

So since the guide says:
4. If the replaced disk continues to be listed after resilvering is complete, click its entry and use the
“Detach” button to remove the disk from the list.

The replaced disk was 86870.... listed as offline. So following the guide I detached it and was once again with a disk that could not be replaced as it was the replacement. Rebooted and the system automatically started a resilver with no input from me.

And that is where it stands 12 more hours away from finishing a resilver (the 3rd resilver in the last few days).

Somehow I have the feeling that I'm going around in circles and after this resilver I will again have a drive listed as offline or unavailable.

I'll check back in a day or two. For now I need a break from all of this.

 

[cyberjock]

You shouldn't feel like you are going around in circles. You *are* going around in circles. So at this point I really don't know what is wrong with your box. I'm virtually convinced that the disk you are trying to put in the pool is bad (or something else is interrupting the resilver).

Can you post the debug for your system ASAP and then provide another after the resilvering finishes (regardless of whether it fails or succeeds).

 

[climb2bhi]

Not sure how to post the debug for my system. If you can give me the shell commands or whatever is needed I'd be glad to.

This where I found it upon returning home after being away this weekend.

With this status:

Is the zpool online command an option?

Code:

[root@freenas ~]# zpool status                                                                                                     
  pool: NAS                                                                                                                        
state: DEGRADED                                                                                                                   
status: One or more devices has been removed by the administrator.                                                                 
        Sufficient replicas exist for the pool to continue functioning in a                                                        
        degraded state.                                                                                                            
action: Online the device using 'zpool online' or replace the device with                                                          
        'zpool replace'.                                                                                                           
  scan: resilvered 1.16T in 11h1m with 0 errors on Sat Nov  8 20:05:52 2014                                                        
config:                                                                                                                            
                                                                                                                                   
        NAME                                            STATE     READ WRITE CKSUM                                                 
        NAS                                             DEGRADED     0     0     0                                                 
          raidz2-0                                      DEGRADED     0     0     0                                                 
            gptid/00780523-12d1-11e4-94f0-0030483594b4  ONLINE       0     0     0                                                 
            gptid/0153dbc6-12d1-11e4-94f0-0030483594b4  ONLINE       0     0     0                                                 
            gptid/023861ca-12d1-11e4-94f0-0030483594b4  ONLINE       0     0     0                                                 
            gptid/02f20184-12d1-11e4-94f0-0030483594b4  ONLINE       0     0     0                                                 
            587882400106777379                          REMOVED      0     0     0  was /dev/gptid/44737168-6704-11e4-8f3a-003048359
4b4                                                                                                                                
            gptid/04a1784e-12d1-11e4-94f0-0030483594b4  ONLINE       0     0     0                                                 
                                                                                                                                   
errors: No known data errors

 

[cyberjock]

Right, that shows that the drive was added to the pool, but later kicked out. Why was it kicked out? Your guess is as good as mine.

Debug is a button in the WebGUI. Please see the manual for your version to invoke it. :)

 

[no_connection]

Probably not related but I would be a tiny bit concerned with 17*C being a bit cold.

 

[climb2bhi]

The Save Debug button just causes my web browser to go into a waiting for 192.168.....

I thought cold was good 17C =63f. Is that too cold? I can warm my system if necessary.

 

[climb2bhi]

BTW I still have the old drive to be returned to WD. It only has one small SMART error. So it could be used for testing.

 

[mjws00]

You are still degraded with 5 disks out of six, one showing removed, right? Or did you resolve it?

 

[climb2bhi]

That is correct. Degraded. 5 out of 6 are fine same one every time shows "Removed" after a resilver. ada1 under View Discs is missing serial number. But the serial number displays correctly when a resilver starts.