Replace failed drive -- Web UI Nonfunctional -- Command-line steps?

[adamgoldberg]

Hi,

I have a failed drive in a RaidZ2, and if I go to log into the web admin pages, I get "An error occurred. Sorry the page you are looking for is currently unavailable...."

Ugh.

zpool status shows which drive is kaput:

Code:

[adam@freenas] /var/log# zpool status -x
  pool: freenasz2
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
		the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: scrub repaired 0 in 3h50m with 0 errors on Sun Jul 29 03:50:23 2018
config:

		NAME											STATE	 READ WRITE CKSUM
		freenasz2									   DEGRADED	 0	 0	 0
		  raidz2-0									  DEGRADED	 0	 0	 0
			16120334606489822329						UNAVAIL	  0   297	 0  was /dev/gptid/1c157709-ccfb-11e5-a222-00259056ae2b
			gptid/1ca67e85-ccfb-11e5-a222-00259056ae2b  ONLINE	   0	 0	 0
			gptid/1d68594d-ccfb-11e5-a222-00259056ae2b  ONLINE	   0	 0	 0
			gptid/1e6240c9-ccfb-11e5-a222-00259056ae2b  ONLINE	   0	 0	 0

errors: No known data errors

So, what are my steps?

1. zpool offline? Is this right (it's either zpool offline freenasz2 1612.... or zpool offline raidz2-0 1612...)
zpool offline raidz2-0 16120334606489822329

2. shutdown

3. replace the drive, then

4. reboot, and ...
zpool replace ... what? what?

help?.?!?@ I'm going out of town early AM tomorrow, and I'd like to get this back up & running... .

 

[adamgoldberg]

Code:

[adam@freenas] /var/log# camcontrol devlist
<ST2000DM001-1CH164 CC24>		  at scbus0 target 0 lun 0 (pass0,ada0)
<ST2000DM001-1CH164 CC24>		  at scbus1 target 0 lun 0 (pass1,ada1)
<TOSHIBA DT01ACA200 MX4OABB0>	  at scbus2 target 0 lun 0 (pass2,ada2)
<TOSHIBA DT01ACA200 MX4OABB0>	  at scbus3 target 0 lun 0 (pass3,ada3)
<TSSTcorp CDDVDW SN-208BB SB00>	at scbus5 target 0 lun 0 (pass4,cd0)
<SanDisk Ultra Fit 1.00>		   at scbus7 target 0 lun 0 (pass5,da0)
<SanDisk Ultra Fit 1.00>		   at scbus8 target 0 lun 0 (pass6,da1)
[adam@freenas] /var/log# zpool list -v
NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
freenas-boot							28.6G  6.08G  22.5G		 -	  -	21%  1.00x  ONLINE  -
  mirror								28.6G  6.08G  22.5G		 -	  -	21%
	da1p2								   -	  -	  -		 -	  -	  -
	gptid/8774672d-da94-11e5-b452-00259056ae2b	  -	  -	  -		 -	  -	  -
freenasz2							   7.25T  3.20T  4.05T		 -	12%	44%  1.00x  DEGRADED  /mnt
  raidz2								7.25T  3.20T  4.05T		 -	12%	44%
	16120334606489822329					-	  -	  -		 -	  -	  -
	gptid/1ca67e85-ccfb-11e5-a222-00259056ae2b	  -	  -	  -		 -	  -	  -
	gptid/1d68594d-ccfb-11e5-a222-00259056ae2b	  -	  -	  -		 -	  -	  -
	gptid/1e6240c9-ccfb-11e5-a222-00259056ae2b	  -	  -	  -		 -	  -	  -
[adam@freenas] /var/log# lspci
00:00.0 Host bridge: Intel Corporation Xeon E3-1200 Processor Family DRAM Controller (rev 09)
00:19.0 Ethernet controller: Intel Corporation 82579LM Gigabit Network Connection (rev 05)
00:1a.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #2 (rev 05)
00:1c.0 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 1 (rev b5)
00:1c.4 PCI bridge: Intel Corporation 6 Series/C200 Series Chipset Family PCI Express Root Port 5 (rev b5)
00:1d.0 USB controller: Intel Corporation 6 Series/C200 Series Chipset Family USB Enhanced Host Controller #1 (rev 05)
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev a5)
00:1f.0 ISA bridge: Intel Corporation C202 Chipset Family LPC Controller (rev 05)
00:1f.2 SATA controller: Intel Corporation 6 Series/C200 Series Chipset Family SATA AHCI Controller (rev 05)
00:1f.3 SMBus: Intel Corporation 6 Series/C200 Series Chipset Family SMBus Controller (rev 05)
02:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
03:03.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)

 

[Chris Moore]

So, what are my steps?

1. zpool offline? Is this right (it's either zpool offline freenasz2 1612.... or zpool offline raidz2-0 1612...)
zpool offline raidz2-0 16120334606489822329

2. shutdown

3. replace the drive, then

4. reboot, and ...
zpool replace ... what? what?

Don't offline anything.
Did you reboot yet? Are you able to access the GUI?

 

[Chris Moore]

Web UI Nonfunctional -- Command-line steps?

Don't do it from the command line. Just give a reboot command and the GUI should come back up after the reboot.

 

[adamgoldberg]

Yes, I rebooted and it came back with a functional UI. I think something was hung messing with the failed drive, timeouts, etc.... things were hung up so bad that a command line shutdown command didn't work. I eventually used the IPMI BRS, and it came up with a working WebUI, and it's happily resilvering now.

 

[Chris Moore]

and it came up with a working WebUI, and it's happily resilvering now.

It is really complicated to do from the command line because FreeNAS partitions the drive to use part as swap space and the rest for data and uses the GPTID of the data partition to add it to the pool. The GUI automates all the steps and makes it easy.

 

[Chris Moore]

Yes, I rebooted and it came back with a functional UI.

I don't see what version of FreeNAS you are using. If it is the latest, 11.1-U5, then a drive failure should not cause the problem you had with the UI becoming unresponsive. This is the second time I have heard of that in the last couple days. You might want to file a bug report because they thought they had this fixed.

 

[adamgoldberg]

I'm still running 9.10.

 

[Chris Moore]

I'm still running 9.10.

Are you running any jails?
In the latest version, they have made changes to the way system swap space is handled that is supposed to eliminate the problem of a failed drive causing the UI to become un-responsive. If you want to take a chance on the update, you might have even better results with it.

 

[adamgoldberg]

Yes, a handful of jails. It seems like a major change, though, and I've been afraid to take the plunge. Should I be afraid?

 

[Chris Moore]

Yes, a handful of jails. It seems like a major change, though, and I've been afraid to take the plunge. Should I be afraid?

I only have my Plex jail now, still using the warden jail system. I was running a jail with VirtualBox since the days of FreeNAS 9.3, but that wouldn't work after the upgrade. Too many changes to the underlying operating system. They are completely redesigning the jail infrastructure using iocage. The direction from the developers is to rebuild your warden jails in iocage because that is the way forward. The system is stable, it just requires some changes and 11.1 is the intermediate step because the interface is still the same; in 11.2 they are changing the UI and the iocage jails are the default too.
What jails are you running?