After replacing faulted drive from boot pool can't access web interface

[zimon]

This is my setup:
I have a boot pool containing 2 drives which are mirrored. One usb drive and one SSD.
I bought 3 usb drives since I have heard that they often fail which already happend 2 times. I just replaced them and eveything worked again. Not this time.

This is what happened.

One drive dp0p2 was faulted so I replaced the usb drive and hit replace via the web ui. After a while I checked the status:

Code:

pool: freenas-boot

state: UNAVAIL

status: One or more devices is currently being resilvered.  The pool will

    continue to function, possibly in a degraded state.

action: Wait for the resilver to complete.

  scan: resilver in progress since Wed Feb 22 20:30:57 2017

        346M scanned out of 1.22G at 98.8K/s, 2h36m to go

        346M resilvered, 27.69% done

config:

    NAME                     STATE     READ WRITE CKSUM

    freenas-boot             UNAVAIL      0     0     0

      mirror-0               UNAVAIL      9     0     4

        2664745704883015641  REMOVED      0     0     0  was /dev/gptid/4efb264a-21cd-11e0-9bf5-3cd92b02910c

        replacing-1          FAULTED      0     0     0

          da0p2/old          FAULTED     20   275    50  too many errors

          da0p2              ONLINE       0     0     0  (resilvering)


errors: Permanent errors have been detected in the following files:


        <metadata>:<0x0>

        <metadata>:<0x1>

        <metadata>:<0xdc>

        freenas-boot/ROOT/FreeNAS-9.3-STABLE-201605170422@2016-01-10-01:45:28:<0x0>

        freenas-boot/ROOT/FreeNAS-9.3-STABLE-201605170422@2016-01-10-01:45:28:<0xfffffffffffffffe>

        freenas-boot/ROOT/FreeNAS-9.3-STABLE-201605170422@2016-01-10-01:45:28:<0xffffffffffffffff>

        freenas-boot/ROOT/FreeNAS-9.3-STABLE-201605170422:<0xaf75>

        freenas-boot/ROOT/FreeNAS-9.3-STABLE-201605170422:<0x11de8>

        freenas-boot/ROOT/FreeNAS-9.3-STABLE-201605170422:<0x32ec>

And I also cannot reach the web interface anymore. The percentage of the resilvering process does not seem to change (for the last hour)

My guess is that I removed the wrong drive. I am stupid as !@#$
But it gets better. I don't have a backup of my config and -of course- I formatted the removed drive immediately because I am stupid as !@#$.

You probably think that I do not deserve any help at all, but re-creating the status quo from a fresh install would take me weeks since, as you already know, I am really really stupid.

Strangely all of my "services" are working like plex, transmission, vpn, btsync and so on, so I was wondering (if my analysis is correct) if I somehow can get the config file to restore the complete setup?

Even though I do not deserve it, I really really need your help.

 

[DrKK]

Your boot pool is a mirrored pair of devices, one is an SSD connected to a SATA port, and one is ....an external USB drive? Is that correct?

Also, can you tell us your system parameters? CPU? Motherboard model and make? Amount of RAM, and so on?

 

[zimon]

Your boot pool is a mirrored pair of devices, one is an SSD connected to a SATA port, and one is ....an external USB drive? Is that correct?

Yes that is correct.

Also, can you tell us your system parameters? CPU? Motherboard model and make? Amount of RAM, and so on?

It is a Hp ProLiant Microserver N36L Athlon2 with 8GB RAM.
Unfortunatly I cannot ssh into it anymore... don't know why, it worked a few minuted ago. The services like plex are still running...

 

[Robert Trevellyan]

Try searching the forums for "config backup on pool" or similar. I'm pretty sure FreeNAS stores a copy of the configuration on the storage pool on a regular basis. If I'm correct, then you can do a clean install, import the pool, and recover the configuration.

 

[Sakuru]

Yes it does, in $NAME_OF_POOL/.system. You can browse these backups on a normal system at /var/db/system/configs-*.

 

[zimon]

phew, I will try that on the weekend. I really hope that works. If anyone has a good step by step guid on restoring a system after a clean install, I would be very thankful.

 

[zimon]

Well I now did a fresh install and re imported my old data pool. When I go to /var/db/system/ only one old and one new config is there:

Code:

[root@freenas /var/db/system]# ls -al                                          
total 51                                                                      
drwxr-xr-x  10 root  wheel    12 Feb 28 15:44 .                                
drwxr-xr-x  20 root  wheel  1728 Feb 28 15:44 ..                              
drwxr-xr-x   2 root  wheel     2 Jan 10  2016 configs-e2eccb3703ad46d2b19f2e4809
443384                                                                        
drwxr-xr-x   2 root  wheel     2 Feb 28 15:44 configs-ea02119b0df4495ba64ec1dbdd
61ed06                                                                        
drwxrwxr-x   2 root  wheel     3 Jan 11  2016 cores                            
-rw-r--r--   1 root  wheel     0 Jan 10  2016 nfs-stablerestart                
-rw-r--r--   1 root  wheel     0 Jan 10  2016 nfs-stablerestart.bak            
drwxr-xr-x   2 root  wheel     2 Jan 10  2016 rrd-e2eccb3703ad46d2b19f2e48094433
84                                                                            
drwxr-xr-x   2 root  wheel     2 Feb 28 15:44 rrd-ea02119b0df4495ba64ec1dbdd61ed
06                                                                            
drwxr-xr-x   4 root  wheel    29 Feb 19 21:43 samba4                          
drwxr-xr-x   2 root  wheel     2 Jan 10  2016 syslog-e2eccb3703ad46d2b19f2e48094
43384                                                                          
drwxr-xr-x   3 root  wheel     3 Feb 28 15:44 syslog-ea02119b0df4495ba64ec1dbdd6
1ed06

and when I go to my Pool /mnt/myPool there is no .system folder...

 

[Sakuru]

Both of those "configs-*" entries are folders. They contain folders named after the version of FreeNAS.

Code:

[root@nas] ~# ls -lh /var/db/system/configs-67525bb0db424eceb7b9950533ac42e8/
total 193
drwxr-xr-x  12 root  wheel	12B Jan 14 03:45 ./
drwxr-xr-x   7 root  wheel	 9B Feb 28 04:42 ../
drwxr-xr-x   2 root  wheel	 6B May  4  2016 FreeNAS-9.10-STABLE-201604261518 (881b70d)/
drwxr-xr-x   2 root  wheel	24B May 26  2016 FreeNAS-9.10-STABLE-201605021851 (35c85f7)/
drwxr-xr-x   2 root  wheel	53B Jul 16  2016 FreeNAS-9.10-STABLE-201605240427 (64fcd8e)/
drwxr-xr-x   2 root  wheel	21B Aug  4  2016 FreeNAS-9.10-STABLE-201606270534 (dd17351)/
drwxr-xr-x   2 root  wheel	56B Sep 27 03:45 FreeNAS-9.10.1 (d989edd)/
drwxr-xr-x   2 root  wheel	 9B Oct  4 03:45 FreeNAS-9.10.1-U1 (ff51a5d)/
drwxr-xr-x   2 root  wheel	65B Dec  6 03:45 FreeNAS-9.10.1-U2 (f045a8b)/
drwxr-xr-x   2 root  wheel	14B Dec 20 03:45 FreeNAS-9.10.1-U4 (ec9a7d3)/
drwxr-xr-x   2 root  wheel	26B Jan 13 03:45 FreeNAS-9.10.2 (a476f16)/
drwxr-xr-x   2 root  wheel	48B Feb 28 03:45 FreeNAS-9.10.2-U1 (86c7ef5)/

Yeah, .system is a little strange to wrap your mind around. I believe it's mounted to /var/db/system, but zfs list doesn't say much.

Code:

[root@nas] ~# zfs list | grep system
tank/.system											128M  24.1T   222K  legacy
tank/.system/configs-67525bb0db424eceb7b9950533ac42e8  31.7M  24.1T  31.7M  legacy
tank/.system/cores									 1.31M  24.1T  1.31M  legacy
tank/.system/rrd-67525bb0db424eceb7b9950533ac42e8	  41.8M  24.1T  41.8M  legacy
tank/.system/samba4									 725K  24.1T   725K  legacy
tank/.system/syslog-67525bb0db424eceb7b9950533ac42e8   52.3M  24.1T  52.3M  legacy

 

[zimon]

Ok This is what I get:

Code:

[root@freenas /var/db/system/configs-ea02119b0df4495ba64ec1dbdd61ed06]# zfs list | grep system									

ZFSPool/.system										   17.1M  2.92T   104K  legacy											

ZFSPool/.system/configs-e2eccb3703ad46d2b19f2e4809443384  14.7M  2.92T  14.7M  legacy											

ZFSPool/.system/configs-ea02119b0df4495ba64ec1dbdd61ed06	88K  2.92T	88K  legacy											

ZFSPool/.system/cores									  736K  2.92T   736K  legacy											

ZFSPool/.system/rrd-e2eccb3703ad46d2b19f2e4809443384		96K  2.92T	96K  legacy											

ZFSPool/.system/rrd-ea02119b0df4495ba64ec1dbdd61ed06		88K  2.92T	88K  legacy											

ZFSPool/.system/samba4									 500K  2.92T   500K  legacy											

ZFSPool/.system/syslog-e2eccb3703ad46d2b19f2e4809443384	656K  2.92T   656K  legacy											

ZFSPool/.system/syslog-ea02119b0df4495ba64ec1dbdd61ed06	196K  2.92T   196K  legacy	

And the problem is, that both configs-* folders are empty.

Code:

										

[root@freenas /var/db/system/configs-ea02119b0df4495ba64ec1dbdd61ed06]# cd ..													

[root@freenas /var/db/system]# cd configs-e2eccb3703ad46d2b19f2e4809443384/														

[root@freenas /var/db/system/configs-e2eccb3703ad46d2b19f2e4809443384]# ls -lh													

total 0																															

[root@freenas /var/db/system/configs-e2eccb3703ad46d2b19f2e4809443384]# c

Please, please don't tell me that there is no old config left for me...

 

[Sakuru]

Wow, I don't know why I didn't try this earlier. mount shows where everything is mounted
I'm hoping your older configs dataset is either mounted somewhere else or not mounted at all. It's 14.7 MB, there's something in there.

Code:

[root@nas] ~# mount | grep system
tank/.system on /var/db/system (zfs, local, nfsv4acls)
tank/.system/cores on /var/db/system/cores (zfs, local, nfsv4acls)
tank/.system/samba4 on /var/db/system/samba4 (zfs, local, nfsv4acls)
tank/.system/syslog-67525bb0db424eceb7b9950533ac42e8 on /var/db/system/syslog-67525bb0db424eceb7b9950533ac42e8 (zfs, local, nfsv4acls)
tank/.system/rrd-67525bb0db424eceb7b9950533ac42e8 on /var/db/system/rrd-67525bb0db424eceb7b9950533ac42e8 (zfs, local, nfsv4acls)
tank/.system/configs-67525bb0db424eceb7b9950533ac42e8 on /var/db/system/configs-67525bb0db424eceb7b9950533ac42e8 (zfs, local, nfsv4acls)

 

[zimon]

It only shows me those folders again, which are empty.

Code:

p.p1 {margin: 0.0px 0.0px 0.0px 0.0px; font: 11.0px 'Courier New'; color: #ffffff; -webkit-text-stroke: #ffffff; background-color: #000000} span.s1 {font-kerning: none}

[root@freenas /var/db/system/configs-e2eccb3703ad46d2b19f2e4809443384]# mount | 

grep system																	 

ZFSPool/.system on /var/db/system (zfs, local, nfsv4acls)					   

ZFSPool/.system/cores on /var/db/system/cores (zfs, local, nfsv4acls)		   

ZFSPool/.system/samba4 on /var/db/system/samba4 (zfs, local, nfsv4acls)		 

ZFSPool/.system/syslog-ea02119b0df4495ba64ec1dbdd61ed06 on /var/db/system/syslog

-ea02119b0df4495ba64ec1dbdd61ed06 (zfs, local, nfsv4acls)					   

ZFSPool/.system/rrd-ea02119b0df4495ba64ec1dbdd61ed06 on /var/db/system/rrd-ea021

19b0df4495ba64ec1dbdd61ed06 (zfs, local, nfsv4acls)							 

ZFSPool/.system/configs-ea02119b0df4495ba64ec1dbdd61ed06 on /var/db/system/confi

gs-ea02119b0df4495ba64ec1dbdd61ed06 (zfs, local, nfsv4acls)   

So when it's not mounted? How can I mount it then?

 

[Sakuru]

Try this

Code:

mkdir /mnt/configs-e2eccb3703ad46d2b19f2e4809443384
zfs set mountpoint=/mnt/configs-e2eccb3703ad46d2b19f2e4809443384 ZFSPool/.system/configs-e2eccb3703ad46d2b19f2e4809443384

 

[zimon]

hm again no success:

Code:

[root@freenas /mnt/configs-ea02119b0df4495ba64ec1dbdd61ed06]# mkdir /mnt/config-e2eccb3703ad46d2b19f2e4809443384					

[root@freenas /mnt/configs-ea02119b0df4495ba64ec1dbdd61ed06]# zfs set mountpoint=/mnt/config										

config-e2eccb3703ad46d2b19f2e4809443384/  configs-ea02119b0df4495ba64ec1dbdd61ed06/												 

[root@freenas /mnt/configs-ea02119b0df4495ba64ec1dbdd61ed06]# zfs set mountpoint=/mnt/config-e2eccb3703ad46d2b19f2e4809443384/ ZFSPo

ol/.system/configs-e2eccb3703ad46d2b19f2e4809443384																				 

[root@freenas /mnt/configs-ea02119b0df4495ba64ec1dbdd61ed06]# cd ..																 

[root@freenas /mnt]# cd config-e2eccb3703ad46d2b19f2e4809443384/																	

[root@freenas /mnt/config-e2eccb3703ad46d2b19f2e4809443384]# ls -la																 

total 8																															 

drwxrwxr-x  2 root  wheel	0 Mar  3 20:05 .																					   

drwxr-xr-x  6 root  wheel  320 Mar  3 20:05 ..																					  

I guess it is time to face the truth?

 

[Sakuru]

Huh, that's weird. I thought for sure there would be something in there. Unfortunately it does look like you'll have reconfigure from scratch. This would be a good time to grab one of those automatic config backup scripts ;)