Replacing failing drive - State: Unavail

F

FaxeB

Cadet
Joined
May 7, 2019
Messages
6
Hello,
My system:
FreeNAS-9.10.2-U6 (561f0d7a1)
5 x 3 TB raidz2
8 GB RAM

Recently I got the alert: 'ada1 has too many errors'. Since the drive has warranty I ordered an RMA (WD red) and got it fast. The drive was already offline, so when reading https://www.ixsystems.com/documentation/freenas/9.10/storage.html#replacing-a-failed-drive I jumped directly to "shut down the system to physically replace the disk". I did so and when starting it up again the ada1 showed 'UNAVAIL'. After that I've been searching this forum up and down without any success - probably I've done something preventing me from succeeding... So now I'm back at this 'UNAVAIL' state and I'm asking for my next step. Some info:
Code:
[root@freenas ~]# zpool status                                                                                                     
  pool: freenas-boot                                                                                                               
 state: ONLINE                                                                                                                     
  scan: scrub repaired 0 in 0h2m with 0 errors on Tue Apr  2 03:47:23 2019                                                         
config:                                                                                                                             
                                                                                                                                    
        NAME        STATE     READ WRITE CKSUM                                                                                     
        freenas-boot  ONLINE       0     0     0                                                                                   
          da0p2     ONLINE       0     0     0                                                                                     
                                                                                                                                    
errors: No known data errors                                                                                                       
                                                                                                                                    
  pool: storage                                                                                                                     
 state: DEGRADED                                                                                                                   
status: One or more devices could not be opened.  Sufficient replicas exist for                                                     
        the pool to continue functioning in a degraded state.                                                                       
action: Attach the missing device and online it using 'zpool online'.                                                               
   see: http://illumos.org/msg/ZFS-8000-2Q                                                                                         
  scan: scrub repaired 0 in 6h15m with 0 errors on Sun Apr 28 06:15:02 2019                                                         
config:                                                                                                                             
                                                                                                                                    
        NAME                                          STATE     READ WRITE CKSUM                                                   
        storage                                       DEGRADED     0     0     0                                                   
          raidz2-0                                    DEGRADED     0     0     0                                                   
            ada0                                      ONLINE       0     0     0                                                   
            6753909526819114032                       UNAVAIL      0     0     0  was /dev/ada1                                     
            ada2                                      ONLINE       0     0     0                                                   
            ada3                                      ONLINE       0     0     0                                                   
            ada4                                      ONLINE       0     0     0                                                   
          gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e  ONLINE       0     0     0                                                   
                                                                                                                                    
errors: No known data errors                               

[root@freenas ~]# sysctl kern.disks                                                                                                 
kern.disks: da0 ada4 ada3 ada2 ada1 ada0   

[root@freenas ~]# gpart show ada1                                                                                                   
=>        34  5860533101  ada1  GPT  (2.7T)                                                                                         
          34          94        - free -  (47K)                                                                                     
         128     4194304     1  freebsd-swap  (2.0G)                                                                               
     4194432  5856338696     2  freebsd-zfs  (2.7T)                                                                                 
  5860533128           7        - free -  (3.5K)       
 
[root@freenas ~]# zpool replace 6753909526819114032 /dev/ada1                                                                       
cannot open '6753909526819114032': name must begin with a letter


I think I'll update the system when I'm back on track, but right now I'd like a hint or two to get there. Thank you!
 
pro lamer

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626
Your system is not raidz2. It's raidz2 striped with a single drive. If you lose that single drive you would loose the whole pool.

I'd recommend you considered some solution to the above first. The best would be a complete pool rebuild but if not possible then at least add a mirror drive to gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e.

Just be careful not to make things worse.

Sent from my phone
 
T

tfran1990

Patron
Joined
Oct 18, 2017
Messages
294
pro lamer said:
Your system is not raidz2. It's raidz2 striped with a single drive. If you lose that single drive you would loose the whole pool.
Click to expand...


How can you tell just by looking at his zpool status?
he must have use diskmanager when he did the replace and added a single disc mirror on accident?
 
M

myoung

Explorer
Joined
Mar 14, 2018
Messages
70
tfran1990 said:
How can you tell just by looking at his zpool status?
he must have use diskmanager when he did the replace and added a single disc mirror on accident?
Click to expand...

The indentation shows which disks are in which vdev. It looks like the raidZ2 vdev came from another system (all the labels are adaX). Then he accidentaly added the new disk as a new vdev through the GUI instead of replacing the bad disk (it has gptid label).

pro lamer said:
Your system is not raidz2. It's raidz2 striped with a single drive. If you lose that single drive you would loose the whole pool.

I'd recommend you considered some solution to the above first. The best would be a complete pool rebuild but if not possible then at least add a mirror drive to gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e.

Just be careful not to make things worse.

Sent from my phone
Click to expand...

He could update to 11.2 and zpool remove the new single disk vdev.

Can you post the output of zpool iostat -v and zpool history storage
 
F

FaxeB

Cadet
Joined
May 7, 2019
Messages
6
Thank you for this rapid response!
Code:
[root@freenas ~]# zpool iostat -v                                                                                                   
                                           capacity     operations    bandwidth                                                     
pool                                    alloc   free   read  write   read  write                                                   
--------------------------------------  -----  -----  -----  -----  -----  -----                                                   
freenas-boot                            2.03G  12.3G      1      0  72.1K  5.49K                                                   
  da0p2                                 2.03G  12.3G      1      0  72.1K  5.49K                                                   
--------------------------------------  -----  -----  -----  -----  -----  -----                                                   
storage                                 5.26T  11.1T     18     70   506K   565K                                                   
  raidz2                                5.26T  8.38T     18     53   502K   362K                                                   
    ada0                                    -      -      8     18   143K   181K                                                   
    6753909526819114032                     -      -      0      0      0      0                                                   
    ada2                                    -      -      5     16   131K   181K                                                   
    ada3                                    -      -      6     16   162K   181K                                                   
    ada4                                    -      -      9     18   174K   181K                                                   
  gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e   368M  2.72T      0     16  4.09K   203K                                             
--------------------------------------  -----  -----  -----  -----  -----  -----


and (this one is long - the old part is attached as file):

Code:
[root@freenas ~]# zpool history storage
History for 'storage':                                                                                                             

<snip to file>
                                                                                             
2018-12-23.00:00:09 zpool scrub storage                                                                                             
2018-12-30.00:00:04 zpool scrub storage                                                                                             
2019-01-06.00:00:11 zpool scrub storage                                                                                             
2019-01-13.00:00:10 zpool scrub storage                                                                                             
2019-01-20.00:00:13 zpool scrub storage                                                                                             
2019-01-27.00:00:11 zpool scrub storage                                                                                             
2019-02-03.00:00:10 zpool scrub storage                                                                                             
2019-02-10.00:00:09 zpool scrub storage                                                                                             
2019-02-17.00:00:09 zpool scrub storage                                                                                             
2019-02-24.00:00:09 zpool scrub storage                                                                                             
2019-03-03.00:00:10 zpool scrub storage                                                                                             
2019-03-10.00:00:11 zpool scrub storage                                                                                             
2019-03-17.00:00:09 zpool scrub storage                                                                                             
2019-03-17.23:04:25 zpool import -o cachefile=none -R /mnt -f storage                                                               
2019-03-17.23:04:25 zpool set cachefile=/data/zfs/zpool.cache storage                                                               
2019-03-17.23:14:07 zpool import -o cachefile=none -R /mnt -f storage                                                               
2019-03-17.23:14:07 zpool set cachefile=/data/zfs/zpool.cache storage                                                               
2019-03-24.00:00:13 zpool scrub storage                                                                                             
2019-03-31.00:00:12 zpool scrub storage                                                                                             
2019-04-07.00:00:10 zpool scrub storage                                                                                             
2019-04-14.00:00:09 zpool scrub storage                                                                                             
2019-04-21.00:00:09 zpool scrub storage                                                                                             
2019-04-28.00:00:09 zpool scrub storage                                                                                             
2019-05-01.19:32:31 zfs create -o casesensitivity=sensitive -o aclmode=restricted storage/burken/shared                             
2019-05-01.19:32:36 zfs set org.freenas:description= storage/burken/shared                                                         
2019-05-04.23:21:25 zpool import -o cachefile=none -R /mnt -f storage                                                               
2019-05-04.23:21:25 zpool set cachefile=/data/zfs/zpool.cache storage                                                               
2019-05-04.23:53:17 zpool online storage 6753909526819114032                                                                       
2019-05-04.23:59:57 zpool import -o cachefile=none -R /mnt -f storage                                                               
2019-05-04.23:59:57 zpool set cachefile=/data/zfs/zpool.cache storage                                                               
2019-05-05.00:20:12 zpool online storage 6753909526819114032                                                                       
2019-05-05.23:17:29 zpool import -o cachefile=none -R /mnt -f storage                                                   
2019-05-05.23:17:29 zpool set cachefile=/data/zfs/zpool.cache storage                                                               
2019-05-06.00:01:39 zpool add -f storage /dev/gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e                                           
2019-05-06.21:14:42 zpool online storage ada1                                                                                       
2019-05-06.21:53:44 zpool import -o cachefile=none -R /mnt -f storage                                                               
2019-05-06.21:53:44 zpool set cachefile=/data/zfs/zpool.cache storage                                                               
2019-05-06.22:09:31 zpool import -o cachefile=none -R /mnt -f storage                                                               
2019-05-06.22:09:31 zpool set cachefile=/data/zfs/zpool.cache storage                                                               
2019-05-08.21:12:27 zpool import -o cachefile=none -R /mnt -f storage                                                               
2019-05-08.21:12:27 zpool set cachefile=/data/zfs/zpool.cache storage
 

Attachments

  • zpool_history_storage.txt
    40.7 KB · Views: 356
T

tfran1990

Patron
Joined
Oct 18, 2017
Messages
294
He could update to 11.2 and zpool remove the new single disk vdev.
Click to expand...

I just started using freenas when 10.x came out. Has zpool remove been a thing before 11.2?
my guess is yes, but you mentioned updating to 11.2 to do so.
 
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
myoung said:
He could update to 11.2 and zpool remove the new single disk vdev.
Click to expand...
No, he can't--vdev removal only works when all vdevs are either single disks or mirrors--which makes it considerably less useful than it otherwise would have been.
 
M

myoung

Explorer
Joined
Mar 14, 2018
Messages
70
danb35 said:
No, he can't--vdev removal only works when all vdevs are either single disks or mirrors--which makes it considerably less useful than it otherwise would have been.
Click to expand...

Oh, dang! I guess that's not an option.
 
M

myoung

Explorer
Joined
Mar 14, 2018
Messages
70
tfran1990 said:
I just started using freenas when 10.x came out. Has zpool remove been a thing before 11.2?
my guess is yes, but you mentioned updating to 11.2 to do so.
Click to expand...

From 11.2 Release notes (Emphasis mine):

OpenZFS is up-to-date with Illumos and slightly ahead due to support for sorted scrubs which were ported from ZFS on Linux. Notable improvements include channel programs, data disk removal, more resilient volume import, the ability to import a pool with missing vdevs, pool checkpoints, improved compressed ARC performance, and ZIL batching. Note that many of these improvements need further testing so have not yet been integrated into the UI.
Click to expand...
 
M

myoung

Explorer
Joined
Mar 14, 2018
Messages
70
FaxeB said:
Thank you for this rapid response!
Code:
[root@freenas ~]# zpool iostat -v                                                                                                  
                                           capacity     operations    bandwidth                                                    
pool                                    alloc   free   read  write   read  write                                                  
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
freenas-boot                            2.03G  12.3G      1      0  72.1K  5.49K                                                  
  da0p2                                 2.03G  12.3G      1      0  72.1K  5.49K                                                  
--------------------------------------  -----  -----  -----  -----  -----  -----                                                  
storage                                 5.26T  11.1T     18     70   506K   565K                                                  
  raidz2                                5.26T  8.38T     18     53   502K   362K                                                  
    ada0                                    -      -      8     18   143K   181K                                                  
    6753909526819114032                     -      -      0      0      0      0                                                  
    ada2                                    -      -      5     16   131K   181K                                                  
    ada3                                    -      -      6     16   162K   181K                                                  
    ada4                                    -      -      9     18   174K   181K                                                  
  gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e   368M  2.72T      0     16  4.09K   203K                                            
--------------------------------------  -----  -----  -----  -----  -----  -----


and (this one is long - the old part is attached as file):

Code:
[root@freenas ~]# zpool history storage
History for 'storage':                                                                                                            

<snip to file>
                                                                                            
2018-12-23.00:00:09 zpool scrub storage                                                                                            
2018-12-30.00:00:04 zpool scrub storage                                                                                            
2019-01-06.00:00:11 zpool scrub storage                                                                                            
2019-01-13.00:00:10 zpool scrub storage                                                                                            
2019-01-20.00:00:13 zpool scrub storage                                                                                            
2019-01-27.00:00:11 zpool scrub storage                                                                                            
2019-02-03.00:00:10 zpool scrub storage                                                                                            
2019-02-10.00:00:09 zpool scrub storage                                                                                            
2019-02-17.00:00:09 zpool scrub storage                                                                                            
2019-02-24.00:00:09 zpool scrub storage                                                                                            
2019-03-03.00:00:10 zpool scrub storage                                                                                            
2019-03-10.00:00:11 zpool scrub storage                                                                                            
2019-03-17.00:00:09 zpool scrub storage                                                                                            
2019-03-17.23:04:25 zpool import -o cachefile=none -R /mnt -f storage                                                              
2019-03-17.23:04:25 zpool set cachefile=/data/zfs/zpool.cache storage                                                              
2019-03-17.23:14:07 zpool import -o cachefile=none -R /mnt -f storage                                                              
2019-03-17.23:14:07 zpool set cachefile=/data/zfs/zpool.cache storage                                                              
2019-03-24.00:00:13 zpool scrub storage                                                                                            
2019-03-31.00:00:12 zpool scrub storage                                                                                            
2019-04-07.00:00:10 zpool scrub storage                                                                                            
2019-04-14.00:00:09 zpool scrub storage                                                                                            
2019-04-21.00:00:09 zpool scrub storage                                                                                            
2019-04-28.00:00:09 zpool scrub storage                                                                                            
2019-05-01.19:32:31 zfs create -o casesensitivity=sensitive -o aclmode=restricted storage/burken/shared                            
2019-05-01.19:32:36 zfs set org.freenas:description= storage/burken/shared                                                        
2019-05-04.23:21:25 zpool import -o cachefile=none -R /mnt -f storage                                                              
2019-05-04.23:21:25 zpool set cachefile=/data/zfs/zpool.cache storage                                                              
2019-05-04.23:53:17 zpool online storage 6753909526819114032                                                                      
2019-05-04.23:59:57 zpool import -o cachefile=none -R /mnt -f storage                                                              
2019-05-04.23:59:57 zpool set cachefile=/data/zfs/zpool.cache storage                                                              
2019-05-05.00:20:12 zpool online storage 6753909526819114032                                                                      
2019-05-05.23:17:29 zpool import -o cachefile=none -R /mnt -f storage                                                  
2019-05-05.23:17:29 zpool set cachefile=/data/zfs/zpool.cache storage                                                              
2019-05-06.00:01:39 zpool add -f storage /dev/gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e                                          
2019-05-06.21:14:42 zpool online storage ada1                                                                                      
2019-05-06.21:53:44 zpool import -o cachefile=none -R /mnt -f storage                                                              
2019-05-06.21:53:44 zpool set cachefile=/data/zfs/zpool.cache storage                                                              
2019-05-06.22:09:31 zpool import -o cachefile=none -R /mnt -f storage                                                              
2019-05-06.22:09:31 zpool set cachefile=/data/zfs/zpool.cache storage                                                              
2019-05-08.21:12:27 zpool import -o cachefile=none -R /mnt -f storage                                                              
2019-05-08.21:12:27 zpool set cachefile=/data/zfs/zpool.cache storage
Click to expand...


We can see here that you accidentally added the new disk as a single disk vdev on Monday, 5/6/2019 via the GUI.
2019-05-06.00:01:39 zpool add -f storage /dev/gptid/5886bff8-6f81-11e9-ab7f-001d60a7173e
Click to expand...
 
danb35

danb35

Hall of Famer
Joined
Aug 16, 2011
Messages
15,504
myoung said:
added the new disk as a single disk vdev on Monday, 5/6/2019 via the GUI.
Click to expand...
...which meant both ignoring the documentation, and bypassing a bunch of warnings.
 
F

FaxeB

Cadet
Joined
May 7, 2019
Messages
6
Thank you for this analysis. Yes, I did a bad command when I was searching for the solution for the initial 'unavail' problem. When I read your answer I don't find any 'unstriping' to raidz1 or back to a degraded raidz2, so I'm open for more ideas. (And if none, I'll prepare for a rebuild.)
 
M

myoung

Explorer
Joined
Mar 14, 2018
Messages
70
FaxeB said:
Thank you for this analysis. Yes, I did a bad command when I was searching for the solution for the initial 'unavail' problem. When I read your answer I don't find any 'unstriping' to raidz1 or back to a degraded raidz2, so I'm open for more ideas. (And if none, I'll prepare for a rebuild.)
Click to expand...

Apparently you can't remove a vdev from a pool which contains a raidz vdev, as @danb35 pointed out above. I think your only good option is to backup and rebuild your pool.
 
F

FaxeB

Cadet
Joined
May 7, 2019
Messages
6
So, when rebuilding why not updating. I already have the disk and RAM, and got this proposal:

Board: Supermicro X9Sri-F
CPU: E5-2630L @ 2 GHz or E5-1620 @ 3,6 GHz
RAM: 8 x 4 GB ECC (PC3-8500-7-10-HP)
Disk: 5 x 3 TB RAIDZ2 WD-red
Network: Dual intel Gigabit LAN based on i350

Any comments?
 
pro lamer

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626
FaxeB said:
RAM: 8 x 4 GB ECC (PC3-8500-7-10-HP)
Click to expand...
Better go with 4pcs of 8GB so you have slots available for future

Sent from my phone
 
F

FaxeB

Cadet
Joined
May 7, 2019
Messages
6
Correct, but these 8 x 4 are "for free". That's why I start this way.
Any thought about the CPU choice E5-2630L @ 2 GHz or E5-1620 @ 3,6 GHz ? I read that higher GHz is better for Samba and I don't plan for any encryption or transcoding.
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Replacing failing drive - State: Unavail"

Similar threads

P
Drive faulted when trying to resilver
Replies
10
Views
2K
patrickRx
P
B
  • Locked
SSD dropped from mirror, tested OK, FLUSHCACHE errors in dmesg
Replies
4
Views
3K
kdragon75
kdragon75
adrianwi
Disk Faulted
Replies
8
Views
3K
adrianwi
adrianwi
K
  • Locked
FAILED SMART self-check
2
Replies
23
Views
8K
kam270
K
-Adam-
Problem with WD REDS
Replies
5
Views
2K
Yorick
Yorick
Top