I shut down my TrueNAS machine for a day and now it won't boot. Video included.

[Sonicmixmaster]

I had an outage to my internet connection early this morning and I was going to be out for the entire day so I figured I will shut down some of my computers and give them a break until I return in the evening because of no internet. Sometime during the day while I was not home my internet started working so when I returned this evening I started my main pc and confirmed internet is back on then powered up the TrueNAS machine and another computer because I wanted to access some files on of the TrueNAS pool shares. But the network shares in windows My Computer were not showing as connected. First thing I did is in sequence reboot all my internet devices giving them time to start up. FIOS modem, pfSense, Wireless router, Gigabit switch and finally shut down with power button the TrueNAS machine and after 10 seconds turned it on again but I noticed that the TrueNAS shut down was immediate as if the software was not actually running. So I get a spare monitor and keyboard and connect to TrueNAS machine and see that during boot I get the following message. "This is a FreeNAS data disk and can not boot system. System halted."
What I want to know is if there is a way to repair the data on my TrueNAS boot drives without reinstalling from scratch. I am fairly certain that my data pools are fine but I am trying to find out how both SSDs refuse to boot when I set them up during my initial TrueNAS install in RAID 1 configuration. I don't think both SSDs died at the same exact time especially that in the morning they were working fine and the TrueNAS machine was working normally.

So to summarize, I need to fix the boot on the TrueNAS SSDs which are set up inside TrueNAS config as RAID1.
I added a media link above to this post about how the system boots and some bios info.
Thanks in advance to anyone with suggestions. BTW this is my first total failure of the nas in 2 years since I built it.

 

[joeschmuck]

I get the following message. "This is a FreeNAS data disk and can not boot system. System halted."

Your BIOS just needs to be reset to bootstrap from the correct boot drive. It's trying to boot form a Data Drive right now.

 

[joeschmuck]

Sorry, let me add some more information about your setup and what I believe is happening.

It appears that you setup a pair of boot SSD's and you think there is some sort of automatic failover to bootstrap the computer, but that is not the case, you need a dedicate RAID controller card to do that, TrueNAS cannot do that nor do most motherboards. If the boot drive fails to boot up, which it looks like, then it will either hang up on the boot drive or it will move on the the next boot drive as defined by the BIOS. Apparently this would be a data drive, not your other SSD.

In the BIOS, select the other SSD as the boot drive, it will hopefully boot up.

Odds are your first SSD just has corrupt data on it.

 

[Sonicmixmaster]

I did that already. I tried to boot from either of the Kingston drives even having only one connected. No difference.

 

[Sonicmixmaster]

Just a few minutes ago I took the 2 boot SSDs out and attached them to a windows 10 machine using a SATA/USB dock and they both show up as unallocated. This is a major concern for me with how reliable TrueNAS software is. How can this happen especially when during my initial TrueNAS install I designated them for the boot pool as RAID1 for redundancy. If TrueNAS is going to hose the drives at the same time with some kind of update during shut down then RAID1 makes no sense at all as I don't have a safe installation in case of NAS software failures. I might as well boot of a pair of USB flash drives and have the same exact sense of security. This is not good.

 

[jgreco]

I am fairly certain that my data pools are fine but I am trying to find out how both SSDs refuse to boot when I set them up during my initial TrueNAS install in RAID 1 configuration.

It is quite likely that only one is refusing to boot.

You may have misunderstood what "RAID 1" or mirroring of boot devices implies on a PC architecture system. It does not get you redundant booting. It only gets you a backup copy of the data. For truly redundant boot devices, you need to follow a strategy such as what I outline in this article.

Highly Available Boot Pool Strategy

I've pounded a few versions of this out over the years, but I hate explaining over and over. TrueNAS allows a ZFS boot mirror pool to be created to increase the reliability of the NAS. This sounds great in theory, but there's a flaw. Due to...

www.truenas.com

The BIOS on your typical PC is too stupid to have a user-specified fallback or failover boot device configuration. It is probably just picking a semi-random device to try to boot from.

You can:

1) Try swapping your two boot devices, which is likely to end up placing the backup (and presumably still healthy) boot device in the primary device's slot

-or-

2) Go into the BIOS and see if you can tell it to prefer booting from the backup device

-or-

3) Reinstall TrueNAS, being careful to avoid inadvertently overwriting any of your data disks

This all assumes that there isn't some other fundamental error in play. It is, for example, possible to install TrueNAS on a system, and then later on add additional disks to it, and for the system to pick those new disks up as the preferred boot media.

This is a major concern for me with how reliable TrueNAS software is.

This is unfair to TrueNAS. It is the PC BIOS that is deciding what drive to boot off of, not TrueNAS. Your complaint would be more correctly aimed at the designers of the PC architecture, who did not make arrangements for redundant boot options.

 

[Ericloewe]

The BIOS on your typical PC is too stupid to have a user-specified fallback or failover boot device configuration. It is probably just picking a semi-random device to try to boot from.

With open-source system firmware gaining traction, maybe we'll finally be rid of AMI's weird ideas of how to manage boot order by the time there's wide adoption *checks notes* sometime later this decade...? I guess it's better than never.

 

[jgreco]

With open-source system firmware gaining traction, maybe we'll finally be rid of AMI's weird ideas of how to manage boot order by the time there's wide adoption *checks notes* sometime later this decade...? I guess it's better than never.

(*spits*)
[derisively]Optimist.[/derisively]

 

[Sonicmixmaster]

I'm telling you. I did a simple proper shutdown from the TrueNAS web interface like I did many times in the past. This time around while machine was off and I was not home ghosts came over to kill both boot SSD drives where now they show up in windows as unallocated. I downloaded the free Minitool Partition Wizard and looked at the drives in a SATA/USB dock. Both show as unallocated. Initially 2 years ago during my original TrueNAS install I did a lot of research and was recommended that for boot I should have 2 SSDs in a RAID 1 pool for redundancy so this is what I did. And now neither of them boots separately with only 1 of them attached to the TrueNAS machine at a time. Minitool Partition wizard shows that there is a recoverable efi partition.

Here are some screenshots how the 2 boot SSDs look in couple windows softwares I have:
Disregard the drives other than the Kingstons which are 120GB. The others are part of my pc I'm using to type on this forum and for testing of these drives.
First 2 pictures show each Kingston as having 100% healthy state according to SMART data
Last picture is from Minitool Partition Wizard (disk 5 and 6)

 

[joeschmuck]

I did that already. I tried to boot from either of the Kingston drives even having only one connected. No difference.

I might as well boot of a pair of USB flash drives and have the same exact sense of security. This is not good.

Lets assume both drives are corrupt, then you will need to reinstall TrueNAS to at least one of them and then possibly restore the configuration file which hopefully you have created and saved. If not, you can either setup TrueNAS all over again or recover the configuration data from the pool which should be located here: (read the thread, the last two entries)

SOLVED - ***RESOLVED*** Recover Configuration.db From Failing Boot Device. Almost There. Help!

Hi all, I did a regular update to my FreeNAS server on Friday, and afterward I could not access it. A trip to the basement server rack and a reboot showed a Grub error on the boot USB stick, a Sandisk Cruzr. I found this thread in my search for a solution...

www.truenas.com

I have always promoted using a single SSD as the boot device and then performing regular config file backups. I get an email once a week that includes my config file attached. And reinstalling TrueNAS is a fairly quick process. It's not fun but then again it's not bad either.

 

[Sonicmixmaster]

Lets assume both drives are corrupt, then you will need to reinstall TrueNAS to at least one of them and then possibly restore the configuration file which hopefully you have created and saved. If not, you can either setup TrueNAS all over again or recover the configuration data from the pool which should be located here: (read the thread, the last two entries)

SOLVED - ***RESOLVED*** Recover Configuration.db From Failing Boot Device. Almost There. Help!

Hi all, I did a regular update to my FreeNAS server on Friday, and afterward I could not access it. A trip to the basement server rack and a reboot showed a Grub error on the boot USB stick, a Sandisk Cruzr. I found this thread in my search for a solution...

www.truenas.com

I have always promoted using a single SSD as the boot device and then performing regular config file backups. I get an email once a week that includes my config file attached. And reinstalling TrueNAS is a fairly quick process. It's not fun but then again it's not bad either.

Thanks for the info. I will attempt a new TrueNAS install soon as I dont have time right now. I still would like to know how can something like this happen? Is the latest version buggy? I did some kind of update 3 weeks ago. Does it do updates during shutdown? How can both drives show up as unallocated and 100% healthy at the same time? It is almost Halloween and this definitely seems spooky. If you didnt look at my previous post as you may have been typing a reply, I added 3 pictures.

 

[joeschmuck]

I saw your postings and have no idea why the drives wouldn't have any partition data, but then again I have not looked at a boot drive with MiniTool Partition Wizard (I use it too). Could be a Halloween curse.

 

[Sonicmixmaster]

If anyone wants me to try to post some output of utilities in Linux or something else then I can. I also have a M1 Mac mini. I have to go out now for the whole day but will try anything you guys suggest when I return. Thanks for your help. Here is what Minitool Partition Wizard partition recovery wizard shows.

Free version will not recover anything but I have the paid version installed on my other computer if you guys suggest it is worth attempting recovery using this software. Thank you all again for your help.

 

[Sonicmixmaster]

I have installed latest TrueNAS on another single SSD and confirmed with my M1 Mac Mini that data pools are fine. I am having a major problem getting access to the pools on my Windows 10 computer. I had all the info for the pool username and passwords saved and when I reinstalled on the single SSD everything was working on the Mac Mini without me changing anything. Any ideas why Mac works and windows is telling me to remove all the shares before I can make new ones? I am typing this post on the Mac as I was able to test the 2 pools I have and working. I have no clue how to remove the 2 shares for the 2 pools in windows. I did right click on the 2 mapped drives for the 2 pools and disconnected but that did not do anything as now I am unable to recreate the same 2 mapped drives as windows is telling me to contact your network administrator. Mac didn't ask for any of that.

 

[joeschmuck]

Any ideas why Mac works and windows is telling me to remove all the shares before I can make new ones?

Because Windoze is a PITA? I take it that you did not restore your TrueNAS configuration file and just created TrueNAS from scratch again?

 

[Sonicmixmaster]

Because Windoze is a PITA? I take it that you did not restore your TrueNAS configuration file and just created TrueNAS from scratch again?

I don't know how to get the configuration and I still have not touched the 2 RAID 1 SSDs which magically no longer boot. I installed the latest Core version on another SSD temporarily as I needed to get to some files on one of the pools. I am now in Florida for a few days to a few weeks I am not sure yet. I won't be able to do anything to the NAS until I come back but I am looking for any suggestions if you or others can give me. Thanks.

 

[jgreco]

RAID 1

RAID 1? Or ZFS mirrored?

 

[Sonicmixmaster]

RAID 1? Or ZFS mirrored?

I am not sure. Whatever the recommended option was for redundancy using 2 SSDs during my original install 2 years ago. I chose that during the setup. I assumed it would be indestructible if I always have 2 boot drives having the same thing on them.

 

[joeschmuck]

RAID 1? Or ZFS mirrored?

He had a mirrored boot pool during TrueNAS installation, from what it sounds like.

@Sonicmixmaster , did you read the link I provided a few days ago? It tells you where your configuration files are located.

 

[Sonicmixmaster]

He had a mirrored boot pool during TrueNAS installation, from what it sounds like.

@Sonicmixmaster , did you read the link I provided a few days ago? It tells you where your configuration files are located.

I have no partitions visible on both boot SSDs according to Minitool Partition Wizard so I don't know how else to get the configuration file.