Disk UNAVAIL

jringer77 · Jun 21, 2018

Hi,

I don't know what to do here. I'm not sure if the pool is missing a disk or if something else is going on. I should have a total of 9 disks, I think. 8xHDDs (4 per zpool), plus one SATA Flash boot drive (came with FreeNAS Mini XL).

I just set up my new FreeNAS Mini XL (diskless), by moving 8xHDDs and Config backup from old server (v9.3) to the new one (v11). Upon booting the new Mini, I have a missing or defective disk and no way to identify which HDD to re-seat or replace. All 8 have solid blue LEDs in front of the case.

I have replaced several failed HDDs in the past, but my experience and searching on this issue has failed me. Any help is appreciated.

Code:

[root@fs01 ~]# zpool status																										 
  pool: freenas-boot																												
 state: ONLINE																													 
  scan: scrub repaired 0 in 0 days 00:00:29 with 0 errors on Thu Jun 21 03:45:29 2018											   
config:																															 
																																   
	   NAME		STATE	 READ WRITE CKSUM																					 
	   freenas-boot  ONLINE	   0	 0	 0																					
		 ada2p2	ONLINE	   0	 0	 0																					 
																																   
errors: No known data errors																										
																																   
  pool: vol00																													   
 state: ONLINE																													 
  scan: scrub repaired 0 in 0 days 00:00:00 with 0 errors on Fri Jun  1 05:00:01 2018											   
config:																															 
																																   
	   NAME											STATE	 READ WRITE CKSUM												 
	   vol00										   ONLINE	   0	 0	 0												 
		 raidz2-0									  ONLINE	   0	 0	 0												 
		   gptid/62a321c8-6b56-11e3-b369-5404a6979f1e  ONLINE	   0	 0	 0												 
		   gptid/62fa27b2-6b56-11e3-b369-5404a6979f1e  ONLINE	   0	 0	 0												 
		   gptid/73b9713a-afc6-11e7-b6d7-5404a6979f1e  ONLINE	   0	 0	 0												 
		   gptid/63a050c3-6b56-11e3-b369-5404a6979f1e  ONLINE	   0	 0	 0												 
																																   
errors: No known data errors																										
																																   
  pool: vol01																													   
 state: DEGRADED																													
status: One or more devices could not be opened.  Sufficient replicas exist for													 
	   the pool to continue functioning in a degraded state.																	   
action: Attach the missing device and online it using 'zpool online'.															   
   see: http://illumos.org/msg/ZFS-8000-2Q																						 
  scan: scrub repaired 0 in 0 days 05:23:10 with 0 errors on Sat Jun  2 10:23:11 2018											   
config:																															 
																																   
	   NAME											STATE	 READ WRITE CKSUM												 
	   vol01										   DEGRADED	 0	 0	 0												 
		 raidz1-0									  DEGRADED	 0	 0	 0												 
		   gptid/4c61b137-9618-11e6-ba93-5404a6979f1e  ONLINE	   0	 0	 0												 
		   gptid/cd9f836c-5b86-11e6-a6a7-5404a6979f1e  ONLINE	   0	 0	 0												 
		   10594125888305849011						UNAVAIL	  0	 0	 0  was /dev/gptid/0e4c6836-4190-11e2-a6dd-5404a6979
f1e																																 
		   gptid/b2b026ce-3a68-11e8-b910-5404a6979f1e  ONLINE	   0	 0	 0												 
																																   
errors: No known data errors																										
[root@fs01 ~]#

jringer77 · Jun 21, 2018

It seems like I'm missing a physical disk here, like there should be ada[1-9] instead of ada[1-8].

Code:

[root@fs01 ~]# camcontrol devlist																								   
<WDC WD40EFRX-68N32N0 82.00A82>	at scbus1 target 0 lun 0 (pass1,ada1)															
<16GB SATA Flash Drive SFDK002A>   at scbus2 target 0 lun 0 (pass2,ada2)															
<Marvell Console 1.01>			 at scbus9 target 0 lun 0 (pass3)																 
<ST31000528AS CC3E>				at scbus10 target 0 lun 0 (pass4,ada3)														   
<ST31000528AS CC3E>				at scbus11 target 0 lun 0 (pass5,ada4)														   
<ST4000DM000-1F2168 CC54>		  at scbus12 target 0 lun 0 (pass6,ada5)														   
<ST4000DM000-1F2168 CC54>		  at scbus13 target 0 lun 0 (pass7,ada6)														   
<WDC WD1001FALS-00Y6A0 05.01D05>   at scbus14 target 0 lun 0 (pass8,ada7)														   
<ST4000DM000-1F2168 CC54>		  at scbus15 target 0 lun 0 (pass9,ada8)														   
[root@fs01 ~]#

jlpellet · Jun 21, 2018

If you go to the GUI Storage > View Disks, this should generate a table showing all disks by name & serial #. I expect you'll then be able to look at the disk labels & find the missing SN. Also, you should look at the BIOS boot to see if the system sees all disks. I don't use that hardware but I'd suspect a dislodged SATA or power cable to one drive. In summary, if the system is up, view disks noting the sn, power off, reseat all of the power & data cables, then see if the drive is seen on boot. Hope this helps. Good luck.

jringer77 · Jun 26, 2018

So... I wrote down all the SNs, shutdown, pulled each HDD to compare SN, and found the one not listed. I replaced it with a spare and now it's re-silvering. All should be well when that operation completes. Thanks!

jringer77 · Jun 27, 2018

Update: I replaced the disk that was missing. The new one briefly appeared as ada0 and started to resilver. However, resilvering now reports completed but the pool is still degraded. During the resilvering period, everything on the system ran slow, and SMB shares were unavailable. The new HDD does NOT appear listed at all. Could this be a hardware issue with the FreeNAS Mini XL (mobo, cables, etc.)?

I have:
- re-seated SATA & power cables.
- replaced the missing HDD w/ a new WD Red.
- rebooted at various times.

What other info can I provide?

Code:

[root@fs01 ~]# camcontrol devlist																								   
<WDC WD40EFRX-68N32N0 82.00A82>	at scbus1 target 0 lun 0 (pass0,ada0)															
<16GB SATA Flash Drive SFDK002A>   at scbus2 target 0 lun 0 (pass1,ada1)															
<Marvell Console 1.01>			 at scbus9 target 0 lun 0 (pass2)																 
<ST31000528AS CC3E>				at scbus10 target 0 lun 0 (pass3,ada2)														   
<ST31000528AS CC3E>				at scbus11 target 0 lun 0 (pass4,ada3)														   
<ST4000DM000-1F2168 CC54>		  at scbus12 target 0 lun 0 (pass5,ada4)														   
<ST4000DM000-1F2168 CC54>		  at scbus13 target 0 lun 0 (pass6,ada5)														   
<WDC WD1001FALS-00Y6A0 05.01D05>   at scbus14 target 0 lun 0 (pass7,ada6)														   
<ST4000DM000-1F2168 CC54>		  at scbus15 target 0 lun 0 (pass8,ada7)														   
<SanDisk Cruzer Fit 1.27>		  at scbus17 target 0 lun 0 (pass9,da0)															
[root@fs01 ~]#

jlpellet · Jun 28, 2018

I'm confused by the ada0 showing in camcontrol devlist vs text that it is not listed. However, I'd try 1) replacing the SATA cable, 2) swapping the power/data cables to another disk. This attempts to diagnose whether the problem stays with the disk or MB. The other step would be to reinstall FreeNAS to a different USB stick on a different USB port & reload config. I've never had SMB not show during a resilver. I don't know the hardware but have a vague recollection of a motherboard/chipset issue a while back that affected some systems so it might be worthwhile to post the system details here for those more familiar with the hw. Good luck.

jringer77 · Jul 5, 2018

jlpellet said:
I'm confused by the ada0 showing in camcontrol devlist vs text that it is not listed. However, I'd try 1) replacing the SATA cable, 2) swapping the power/data cables to another disk. This attempts to diagnose whether the problem stays with the disk or MB. The other step would be to reinstall FreeNAS to a different USB stick on a different USB port & reload config. I've never had SMB not show during a resilver. I don't know the hardware but have a vague recollection of a motherboard/chipset issue a while back that affected some systems so it might be worthwhile to post the system details here for those more familiar with the hw. Good luck.

Thanks for the troubleshooting help! Replacing the SATA cable worked (for now). The new disk showed up/mounted/attached, or whatever, and is now resilvering (again). This happened the first time, but hopefully with a different cable it will be permanent.

Code:

[root@fs01 ~]# zpool status -v																									
  pool: freenas-boot																												
 state: ONLINE																													
  scan: scrub repaired 0 in 0 days 00:00:30 with 0 errors on Fri Jun 29 03:45:30 2018											  
config:																															
																																  
	   NAME		STATE	 READ WRITE CKSUM																					
	   freenas-boot  ONLINE	   0	 0	 0																					
		 ada2p2	ONLINE	   0	 0	 0																					
																																  
errors: No known data errors																										
																																  
  pool: vol00																													  
 state: ONLINE																													
  scan: scrub repaired 0 in 0 days 00:00:01 with 0 errors on Sun Jul  1 05:00:02 2018											  
config:																															
																																  
	   NAME											STATE	 READ WRITE CKSUM												
	   vol00										   ONLINE	   0	 0	 0												
		 raidz2-0									  ONLINE	   0	 0	 0												
		   gptid/62a321c8-6b56-11e3-b369-5404a6979f1e  ONLINE	   0	 0	 0												
		   gptid/62fa27b2-6b56-11e3-b369-5404a6979f1e  ONLINE	   0	 0	 0												
		   gptid/73b9713a-afc6-11e7-b6d7-5404a6979f1e  ONLINE	   0	 0	 0												
		   gptid/63a050c3-6b56-11e3-b369-5404a6979f1e  ONLINE	   0	 0	 0												
																																  
errors: No known data errors																										
																																  
  pool: vol01																													  
 state: ONLINE																													
status: One or more devices is currently being resilvered.  The pool will														  
	   continue to function, possibly in a degraded state.																		
action: Wait for the resilver to complete.																						
  scan: resilver in progress since Thu Jul  5 18:47:35 2018																		
	   2.26T scanned at 3.55G/s, 32.3G issued at 50.8M/s, 5.36T total															
	   8.06G resilvered, 0.59% done, 1 days 06:34:32 to go																		
config:																															
																																  
	   NAME											STATE	 READ WRITE CKSUM												
	   vol01										   ONLINE	   0	 0	 0												
		 raidz1-0									  ONLINE	   0	 0	 0												
		   gptid/4c61b137-9618-11e6-ba93-5404a6979f1e  ONLINE	   0	 0	 0												
		   gptid/cd9f836c-5b86-11e6-a6a7-5404a6979f1e  ONLINE	   0	 0	 0												
		   gptid/b9532f85-79b6-11e8-be68-d05099c39d88  ONLINE	   0	 0	 7  (resilvering)								  
		   gptid/b2b026ce-3a68-11e8-b910-5404a6979f1e  ONLINE	   0	 0	 0												
																																  
errors: No known data errors																										
[root@fs01 ~]#

jringer77 · Jul 5, 2018

Original camcontrol devlist:

Code:

[root@fs01 ~]# camcontrol devlist																								
<WDC WD40EFRX-68N32N0 82.00A82>	at scbus1 target 0 lun 0 (pass0,ada0)															
<16GB SATA Flash Drive SFDK002A>   at scbus2 target 0 lun 0 (pass1,ada1)															
<Marvell Console 1.01>			 at scbus9 target 0 lun 0 (pass2)																
<ST31000528AS CC3E>				at scbus10 target 0 lun 0 (pass3,ada2)														
<ST31000528AS CC3E>				at scbus11 target 0 lun 0 (pass4,ada3)														
<ST4000DM000-1F2168 CC54>		  at scbus12 target 0 lun 0 (pass5,ada4)														
<ST4000DM000-1F2168 CC54>		  at scbus13 target 0 lun 0 (pass6,ada5)														
<WDC WD1001FALS-00Y6A0 05.01D05>   at scbus14 target 0 lun 0 (pass7,ada6)														
<ST4000DM000-1F2168 CC54>		  at scbus15 target 0 lun 0 (pass8,ada7)														
<SanDisk Cruzer Fit 1.27>		  at scbus17 target 0 lun 0 (pass9,da0)															
[root@fs01 ~]#

New camcontrol devlist:

Code:

[root@fs01 ~]# camcontrol devlist																								
<WDC WD4002FFWX-68TZ4N0 83.H0A83>  at scbus0 target 0 lun 0 (pass0,ada0)	<- now appears										  
<WDC WD40EFRX-68N32N0 82.00A82>	at scbus1 target 0 lun 0 (pass1,ada1)															
<16GB SATA Flash Drive SFDK002A>   at scbus2 target 0 lun 0 (pass2,ada2)															
<Marvell Console 1.01>			 at scbus9 target 0 lun 0 (pass3)																
<ST31000528AS CC3E>				at scbus10 target 0 lun 0 (pass4,ada3)														
<ST31000528AS CC3E>				at scbus11 target 0 lun 0 (pass5,ada4)														
<ST4000DM000-1F2168 CC54>		  at scbus12 target 0 lun 0 (pass6,ada5)														
<ST4000DM000-1F2168 CC54>		  at scbus13 target 0 lun 0 (pass7,ada6)														
<WDC WD1001FALS-00Y6A0 05.01D05>   at scbus14 target 0 lun 0 (pass8,ada7)														
<ST4000DM000-1F2168 CC54>		  at scbus15 target 0 lun 0 (pass9,ada8)														
<SanDisk Cruzer Fit 1.27>		  at scbus17 target 0 lun 0 (pass10,da0)														
[root@fs01 ~]#

Important Announcement for The TrueNAS Community.

Disk UNAVAIL

jringer77

Cadet

jringer77

Cadet

jlpellet

Patron

jringer77

Cadet

jringer77

Cadet

jlpellet

Patron

jringer77

Cadet

jringer77

Cadet

Similar threads

Important Announcement for The TrueNAS Community.