SOLVED Mirrored USB boot sticks won't boot, best way forward?

[entilza72]

Hi team, I've done a bit of reading and just want to poll the collective wisdom before moving forward:

My existing FreeNAS system (latest stable) worked fine for years with dual boot USB sticks.

After a graceful automated UPS shutdown due to a power failure, I appear to have suffered both of my mirrored FreeNAS USB stick boot drives failing at the same time. The boot order is correct in the BIOS (initially I did get "This is a FreeNAS data disk and can not boot system..." etc, but I disabled the data disks from the boot order and physically disconnected them too). But regardless of which USB stick I select, I still get "Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key" from the BIOS.

I've swapped the USB ports, tried a third port, swapped both sticks, changed their boot order, tried with 1 stick (each) in different ports. All no good.

My assumption is it was a double failure (maybe one failed some time ago and I never noticed?) and not a USB port/bus fault, because the keyboard still works.

My questions are:
1. does my assumption of a double failure seem correct? If not, do you have a suggestion?
2. what is the best way to recover from a boot device failure? My only requirement is to protect the integrity of the HDD data.

My plan was to buy a third USB stick and do a clean install, then once the system was up with the HDD pool, I'd locate the config backup (I can't recall placing it) and restore it, or just rebuild everything from scratch. But how does this go with the existing data tanks on the HDDs? Does the act of booting a clean install put the tanks at risk?

Is there a better plan? Do you have any warnings? My main priority is preservation of the NAS storage data. Losing the NAS config is an inconvenience, but surmountable.

Thanks in advance,
Ent.

 

[blueether]

I see no issues with how you plan to move forward. the config can be pulled out of the pool the data set was working in and then can be restored via GUI - do a search and there are several threads with where it's saved

 

[entilza72]

Thanks blueether for giving it a look over and providing confidence. Normally I'd be straight into something like this, but the fear of losing everything is ... sobering. :-/

I'll report back.

 

[blueether]

a small ssd may be better than usb in the long run

 

[1kokies]

After a graceful automated UPS shutdown due to a power failure, I appear to have suffered both of my mirrored FreeNAS USB stick boot drives failing at the same time. The boot order is correct in the BIOS (initially I did get "This is a FreeNAS data disk and can not boot system..." etc, but I disabled the data disks from the boot order and physically disconnected them too). But regardless of which USB stick I select, I still get "Reboot and Select proper Boot device or Insert Boot Media in selected Boot device and press a key" from the BIOS.

@entilza72 if you can boot from the same mobo it would not be the bios issue. The data disks have no boot loader hence will not boot. Is your bios working well ? To check just go into it & see if settings have changed or some features acts funny, in which case need a bios flash ya.

I've swapped the USB ports, tried a third port, swapped both sticks, changed their boot order, tried with 1 stick (each) in different ports. All no good.

swapping the USB's into a laptop or friend's and assuming Windows go to Create and format hard disks partition to see if the USB's appear with a Freenas like partition (screenshot).

My questions are:
1. does my assumption of a double failure seem correct? If not, do you have a suggestion?
2. what is the best way to recover from a boot device failure? My only requirement is to protect the integrity of the HDD data.

My plan was to buy a third USB stick and do a clean install, then once the system was up with the HDD pool, I'd locate the config backup (I can't recall placing it) and restore it, or just rebuild everything from scratch. But how does this go with the existing data tanks on the HDDs? Does the act of booting a clean install put the tanks at risk?

Is there a better plan? Do you have any warnings? My main priority is preservation of the NAS storage data. Losing the NAS config is an inconvenience, but surmountable.

Thanks in advance,

i don't have mirrored USB but i religiously back up the config once there are changes (even if there isn't). Perhaps due to the power shock it may. If you have the config then all is saved, just install into a new USB and upload, it will re boot and appear like it was 5 minutes ago (need to change the IP's at console ya).

If no config then fresh install and import the pool via GUI, you will still get back your data. Few weeks ago a good fellow @PhiloEpisteme showed me how to export and import pool, data was intact but i had my ori config. It should work in your case.

 

[entilza72]

Hi all, thanks for the support. And thanks 1kokies for the info on what a FreeNAS partition looks like. I did put it in windows yesterday and of course got nothing, but I haven't viewed the partitions yet. Will do that tonight.

Rebuild attempt is planned for Saturday.

A couple of interesting bits:

1. I'm not yet convinced there isn't a hardware (mobo) root cause here. Both USB sticks are metal and were quite hot when I shut down. Since then, the mobo has been sitting there powered on at the wall, but switched off at the mobo, with the USB sticks attached. 24 hours later in this "switched off" state, they were SUPER hot - uncomfortable to the touch, but not skin burning. Clearly they were being powered.

This struck me as unusual. I guess the mobo still supplies USB power when "switched off" but perhaps its over-voltage? Maybe the mobo cooked something in the USBs?

2. The BIOS seemed fine yesterday - the time was correct and things seemed to be operating in the menus. Not a great test, of course.

3. SSD boot - I have been thinking about this because I've already had 1 USB fail previously, within a year. I am unkeen to lose a SATA port for future expansion (I'm already planning to use a SATA card). I don't think the mobo supports M3 and as 1kokies mentions I could go single disk if I back up regularly ... (what good was redundancy to me this time? :) ) Assuming there's no M3 port, What I would need is a SATA card that offers 2 internal ports (the number I am missing to complete my drive bay array) and two external ports. I'd use the 2 external ports for a 2x SSD arrangement.

Thanks for the continued interest.

Ent.

 

[1kokies]

1. I'm not yet convinced there isn't a hardware (mobo) root cause here. Both USB sticks are metal and were quite hot when I shut down. Since then, the mobo has been sitting there powered on at the wall, but switched off at the mobo, with the USB sticks attached. 24 hours later in this "switched off" state, they were SUPER hot - uncomfortable to the touch, but not skin burning. Clearly they were being powered.

This struck me as unusual. I guess the mobo still supplies USB power when "switched off" but perhaps its over-voltage? Maybe the mobo cooked something in the USBs?

that's funny and to be safe better to test the mobo. Through my builds it is worth the time & effort. IE before i even install a server mobo i test with various OS and hypervisors to ensure full compatibility and no defects. it also allows me to get familiar with the boards. Once deployed and installed it is very hard to trouble shoot, that's my process ya. Perhaps even though troublesome should remove the mobo & install Windows to test all the ports, usb, pcie etc etc. Just a suggestion ya. And i always stress on the PSU, a good one with protections OCP, OTP, multiple rails etc and enough power. Always used >850W for me.

SSD boot - I have been thinking about this because I've already had 1 USB fail previously, within a year. I am unkeen to lose a SATA port for future expansion (I'm already planning to use a SATA card). I don't think the mobo supports M3 and as 1kokies mentions I could go single disk if I back up regularly ... (what good was redundancy to me this time? :) ) Assuming there's no M3 port, What I would need is a SATA card that offers 2 internal ports (the number I am missing to complete my drive bay array) and two external ports. I'd use the 2 external ports for a 2x SSD arrangement.

could not get you here but in my case i boot 100% from USB. No mirror and as seen from screenshot the file is very small 260mb, so i did not use SSD. They will fail too, in which case we have to back up the config anyway. i would save on the SSD and buy better or bigger disks

 

[blueether]

If you dont have a backup of the config then it can be found at approx:
root@freenas:~ # ls -altr /var/db/system/configs-3bc724bbb18d493585697320a07060e7/FreeNAS-11.2-U5\ \(c129415c52\)/

when the pool is mounted

 

[1kokies]

If you don't have a backup of the config then it can be found at approx:
root@freenas:~ # ls -altr /var/db/system/configs-3bc724bbb18d493585697320a07060e7/FreeNAS-11.2-U5\ \(c129415c52\)/

when the pool is mounted

@blueether this is interesting, means without the config back up we can still retrieve the config once we import the pool in the new install ?

sorry for side tracking @entilza72

 

[entilza72]

swapping the USB's into a laptop or friend's and assuming Windows go to Create and format hard disks partition to see if the USB's appear with a Freenas like partition (screenshot).

I mounted both sticks in a Win 10 machine and checked the partitions. Both only show as one large partition. Cooked?

 

[1kokies]

looks like it, there should be a UEFI/EFI partition as per screenshot above thread

 

[blueether]

@blueether this is interesting, means without the config back up we can still retrieve the config once we import the pool in the new install ?

sorry for side tracking @entilza72

Yes, i've done it once when i fecked up a VM install

 

[entilza72]

If you don't have a backup of the config then it can be found at approx:
root@freenas:~ # ls -altr /var/db/system/configs-3bc724bbb18d493585697320a07060e7/FreeNAS-11.2-U5\ \(c129415c52\)/

when the pool is mounted

Sorry for the question @blueether but I seem to be missing the file structure for the old config:

I've mounted the old Tank (good to see you old friend!) and have the following:

[root@freenas /var/db/system]# ls -la total 56 drwxr-xr-x 10 root wheel 12 Oct 25 04:23 . drwxr-xr-x 21 root wheel 1920 Oct 25 04:23 .. drwxr-xr-x 2 root wheel 2 Oct 25 04:23 configs-3bc724bbb18d493585697320a07060e7 drwxr-xr-x 2 root wheel 2 Oct 12 2017 configs-76c11d7f8a944b3d8e42fe35420dbaa3 drwxrwxr-x 2 root wheel 5 Feb 17 2019 cores -rw-r--r-- 1 root wheel 0 Oct 12 2017 nfs-stablerestart -rw-r--r-- 1 root wheel 0 Oct 12 2017 nfs-stablerestart.bak drwxr-xr-x 2 root wheel 2 Oct 25 04:23 rrd-3bc724bbb18d493585697320a07060e7 drwxr-xr-x 2 root wheel 2 Oct 12 2017 rrd-76c11d7f8a944b3d8e42fe35420dbaa3 drwxr-xr-x 4 root wheel 14 Mar 10 2019 samba4 drwxr-xr-x 2 root wheel 2 Oct 25 04:23 syslog-3bc724bbb18d493585697320a07060e7 drwxr-xr-x 2 root wheel 2 Oct 12 2017 syslog-76c11d7f8a944b3d8e42fe35420dbaa3

I chose configs-76c11d7f8a944b3d8e42fe35420dbaa3 because its the one that WASN'T created today

However, it is empty.
So is configs-3bc724bbb18d493585697320a07060e7

A search under /var/db/system also came up empty:
[root@freenas /var/db/system]#find . -name "*FreeNAS*" [root@freenas /var/db/system]#

I'm running 11.1 (same version as the old config) if that helps? Did it move in 11.2?

Thanks for the help.

Ent.

 

[blueether]

I guess the next question is was the system data set on the pool on USB?

I have a 11.1 dir

Code:

root@freenas:~ # ls -al /var/db/system/configs-3bc724bbb18d493585697320a07060e7/
total 17
drwxr-xr-x  8 root  wheel    8 Jul  1 03:45 .
drwxr-xr-x  9 root  wheel   11 Oct 26 04:08 ..
drwxr-xr-x  2 root  wheel   16 Aug 20  2018 FreeNAS-11.1-RELEASE (dc7d195f4)
drwxr-xr-x  2 root  wheel   69 Dec  2  2018 FreeNAS-11.2-BETA2 (f14b2ed0e)
drwxr-xr-x  2 root  wheel  100 Apr 10  2019 FreeNAS-11.2-RC2 (0fe4fb6d6)
drwxr-xr-x  2 root  wheel   25 May 11 03:45 FreeNAS-11.2-U3 (e140b6b8a)
drwxr-xr-x  2 root  wheel   50 Jun 30 03:45 FreeNAS-11.2-U4.1 (e33ce960b8)
drwxr-xr-x  2 root  wheel  115 Oct 26 03:45 FreeNAS-11.2-U5 (c129415c52)

 

[entilza72]

Thanks for helping @blueether

Taking your lead, I did a find from the root folder (/) for *.db
Excluding the .db files in my tank (ones that are actual files that have been put on my network drives), this is the result:

Code:

./var/tmp/consul/raft/raft.db                                                                                                    
./var/db/services.db                                                                                                            
./usr/local/share/collectd/types.db                                                                                              
./usr/local/lib/perl5/5.24/perl/man/mandoc.db                                                                                    
./usr/local/lib/python2.7/test/185test.db                                                                                        
./usr/share/misc/vgrindefs.db                                                                                                    
./usr/share/misc/termcap.db                                                                                                      
./usr/share/openssl/man/mandoc.db                                                                                                
./usr/share/i18n/csmapper/mapper.dir.db                                                                                          
./usr/share/i18n/esdb/esdb.dir.db                                                                                                
./usr/share/i18n/esdb/esdb.alias.db                                                                                              
./usr/share/man/mandoc.db                                                                                                        
./conf/base/etc/spwd.db                                                                                                          
./conf/base/etc/pwd.db                                                                                                          
./conf/base/etc/login.conf.db                                                                                                    
./conf/base/etc/local/sasldb2.db                                                                                                
./conf/base/var/db/services.db                                                                                                  
./data/factory-v1.db                                                                                                            
./data/freenas-v1.db                                                                                                            
./etc/pwd.db                                                                                                                    
./etc/spwd.db                                                                                                                    
./etc/login.conf.db                                                                                                              
./etc/local/sasldb2.db

There appears to be no config!

If there's no config, how does this even give me config items from the GUI??!

I also did a "find" for folders with *configs-* and got the two empty ones we already knew about:

Code:

[root@freenas /]# find . -name "*configs-*"                                                                                       
./var/db/system/configs-76c11d7f8a944b3d8e42fe35420dbaa3                                                                         
./var/db/system/configs-3bc724bbb18d493585697320a07060e7                                                                         
[root@freenas /]#

All suggestions appreciated. I'm flummoxed!

Ent.

 

[entilza72]

Additional to the above:

I am trying to mount the old USB boot sticks to see if I can recover something there, but am getting unexpected errors when I try to mount any USB device. I eliminated the suspect (old) boot USB and used the new installation stick I just installed from. These are the errors I got:

Code:

(Known good install USB stick is in da2):

[root@freenas /mnt]# mount -o ro /dev/da2 /mnt/test                                                                                
mount: /dev/da2: Invalid argument         
                                                                                        
[root@freenas /mnt]# mount -o ro /dev/da2p1 /mnt/test                                                                              
mount: /dev/da2p1: Input/output error                
                                                                             
[root@freenas /mnt]# mount -o ro /dev/da2p2 /mnt/test                                                                              
mount: /dev/da2p2: Invalid argument           
                                                                                    
[root@freenas /mnt]# mount -o ro /dev/da2p3 /mnt/test                                                                              
mount: /dev/da2p3: Invalid argument
                                                                                                
[root@freenas /mnt]#

Also, the original boot USB sticks have the following partitions:

Code:

da1
da1p1
da1p2

 

[1kokies]

you meant that the mobo has problems ? Or the usb has problems?

 

[blueether]

I wondering if the system dataset was being saved to the USB and this quickend the death of the USB sticks...

If you cant recover the config db from the pool and you didn't save it I guess that leaves you with rebuilding it all from scratch, and then saving the config (to the pool?)

 

[entilza72]

I wondering if the system dataset was being saved to the USB and this quickend the death of the USB sticks...

If you can't recover the config db from the pool and you didn't save it I guess that leaves you with rebuilding it all from scratch, and then saving the config (to the pool?)

Hi @blueether I like the theory as it would certainly explain 3x dead USB sticks in a couple of years, except get this: I gave up and rebuilt my NAS config from scratch. After rebuilding, guess what's now sitting in a /var/db/system/configs-* folder ...

Code:

root@prd-nas-01:/var/db/system/configs-3bc724bbb18d493585697320a07060e7/FreeNAS-11.1-RELEASE (dc7d195f4) # ls -la
total 268
drwxr-xr-x  2 root  wheel       4 Oct 27 03:45 .
drwxr-xr-x  3 root  wheel       3 Oct 26 03:45 ..
-rw-r-----  1 root  wheel  921600 Oct 26 03:45 20191026%s.db
-rw-r-----  1 root  wheel  921600 Oct 27 03:45 20191027%s.db

Either something prevented the original config going in the folder that was made for it, or it somehow got removed when the OS melted down ... ??!

Needless to say, I have learnt my lesson and will be backing this file up in a cloud service

I'd like to thank @blueether and @1kokies for their help. I've learnt and also grown less afraid of catastrophic failure in FreeNAS, because it's not all that catastrophic

Ent.

 

[1kokies]

I'd like to thank @blueether and @1kokies for their help. I've learnt and also grown less afraid of catastrophic failure in FreeNAS, because it's not all that catastrophic :)

@entilza72 can thank members by clicking Thanks on the posts. And glad to see another confident user.