I have that sinking feeling.....

[jaywest]

in the production machine (same hard drives, newly installed ssd for boot), shell to /var/db/system has the one empty config and the other one dated today which only has a single folder with a single db in it. I'm not smart enough to know, but I'd think this is the current (mostly blank) config from fresh install just as I posted above.

However.... I took the failed usb doms and put them in the other identical server. One it wouldn't boot off of at all. The other it would boot off of, with a handful of bad errors (shortening expected read, misc bad stuff...) but it does get to the truenas prompt and I can get to a shell. My hope was if I could boot off the failed usb doms my config should be there? Note, this machine I got the usbdoms to boot on - does NOT have any hard drives, my pool isn't there.

My thought was to plug ethernet into it, do an ifconfig to bring up the IP at some other address, and maybe I could ssh or smth the config from there. However, in /var/db/system there is just one folder called samba4, and the tdb files in it seem to my eye to be related to my drive shares. There is also a services.db file in /var/db.

The issue is, I'm not familiar with the filesystem layout of truenas at this point to know if my config is there on the usb drive somewhere, and where to look. I assume it's not /var/db/system/ as the only thing there is that samba4 folder. Am I on a fools errand and should just start re-adding services and vms through the ui or is there a magic spot I'm not looking on those usb drives?

 

[Glorious1]

The config backups should be on your main/first pool. Why they aren't I don't know, but I would not expect to find them on the boot device.

 

[jaywest]

The config backups should be on your main/first pool. Why they aren't I don't know, but I would not expect to find them on the boot device.

I know the config backups won't be on the boot device, I was hoping the original running configs were.

It appears all my jails are present so the pool import must have found those. My samba shares are not, I assume I can just re-add them from the ui and point to the existing datasets.

Biggest concern to me are the vm's. I assume i can just readd those through the ui and it wont overwrite the data on the drives when creating them (or will at least give me a choice so I can say Noooooo) :D

 

[jaywest]

I'm not qualified to identify pitfalls, but it just seems hard to believe that two DOM devices would fail at the same time. Was your boot pool getting scrubs on some schedule?

EDIT:
Actually, reading this post by @jgreco, perhaps only one boot device has failed, and you just need to tell your BIOS to boot from the other one.
https://www.truenas.com/community/t...t-wont-boot-video-included.104920/post-722007

re: 2 doms failing at same time and scrubs

If I am not mistaken - I think that by default truenas sets a periodic scrub on the boot-pool at every 7 days (system->boot->actions->stats/settings). I thought modern ssd drives were ok to be scrubbed that often. Is the default setting a bad idea?

 

[jaywest]

Did you install on both DOMs at the same time? If you added the second drive to the boot pool after installation, you will have a mirrored pool allright, but the drive will not have UEFI or FreeBSD boot loader installed. You would need to copy those manually (just once) when adding the second drive.

ok this I did not know at all. I assumed after installing fresh on one ssd, as soon as I added the second ssd to boot-pool it mirrored the entire first ssd block for block so no copy of uefi/bootloader needed. Is there a good link you can suggest on adding the additional bits so I can just tell it to boot off the other drive?

 

[jaywest]

All recovered. I can't say enough thanks to all those who responded and offered advice. I really appreciate the time and effort!

I never did find the old config tar, so spent a day poking in all the corners to remember what I had configured. I think danb35 indicated that truenas automatically backed up the config to the pool at one point, but at later releases it stopped doing that. Not sure why they stopped - good feature I would think. All I know is at 12.0.8 there were no config backups at all. I have created a cron job that tars up the configuration files once a month and dumps them into {pool}/backup-configs so that next time this is a very quick operation.

I'm also going to set up IPMI so I have another route in.

Since I made all these backups "just to be safe" before beginning recovery, I was thinking now is the time to update it to 13x. Some googling suggests that the upgrade from 12 to 13 is pretty easy/painless/safe but a friend running truenas 12 just upgraded to 13 and his (firewall or vpn) vm no longer works correctly but he's not sure of cause yet. My networking is pretty much all on my router not truenas so maybe I'm safe. Can I do it completely in the gui by simply switching trains from 12 to 13 and re-checking for updates?

 

[Etorix]

I think danb35 indicated that truenas automatically backed up the config to the pool at one point, but at later releases it stopped doing that. Not sure why they stopped - good feature I would think.

TrueNAS still backs up to the system dataset, whose default location may now be the boot pool. You can change that under System to have the system dataset on a data pool.

 

[Patrick M. Hausen]

ok this I did not know at all. I assumed after installing fresh on one ssd, as soon as I added the second ssd to boot-pool it mirrored the entire first ssd block for block so no copy of uefi/bootloader needed. Is there a good link you can suggest on adding the additional bits so I can just tell it to boot off the other drive?

Please post the output of gpart show for both your boot pool drives.

 

[jaywest]

Please post the output of gpart show for both your boot pool drives.

Code:

root@truenas[~]# gpart show ada0
=>       40  250069600  ada0  GPT  (119G)
         40       1024     1  freebsd-boot  (512K)
       1064   33554432     3  freebsd-swap  (16G)
   33555496  216498176     2  freebsd-zfs  (103G)
  250053672      15968        - free -  (7.8M)


root@truenas[~]# gpart show ada1
=>       40  250069600  ada1  GPT  (119G)
         40       1024     1  freebsd-boot  (512K)
       1064   33554432     3  freebsd-swap  (16G)
   33555496  216498176     2  freebsd-zfs  (103G)
  250053672      15968        - free -  (7.8M)

 

[Patrick M. Hausen]

If ada0 is the disk with a correct boot loader and ada1 the one that needs an update, then either dd if=/dev/ada0p1 of=/dev/ada1p1 or gpart bootcode -b /boot/pmbr -p /boot/gptzfsboot -i 1 ada1.

 

[jaywest]

ah just dd the partition... seems strange to me that when you add the new drive to the boot pool that it's smart enough to create the partition table identical to the first drive, but only seems to mirror the zfs partition instead of mirroring the entire drive block for block. Good to know, thanks!

 

[Patrick M. Hausen]

I'll be checking the source code to find out if I was writing entire FUD or TrueNAS does indeed skip that part.

My initial idea was that it would do what it does when you attach a nee drive to a storage pool. I might be wrong, i.e. overly pessimistic here, but that would explain your observed behaviour.

Even with a boot loader on the second disk - I noticed you are using legacy boot - everything might depend on how exactly the first disk failed. If the BIOS recognises it as "present" but cannot read any data, specifically the boot loader, that can also cause the system to hang.
UEFI is way more robust in that regard. It's just FAT partitions and the system will happily boot from anything it finds. Or at least drop you into an EFI shell so you can tell it what to boot without a rescue system.

IMHO there is no reason to use anything but EFI today.