FreeNAS won't boot after CPU upgrade

[GlueFactoryBJJ]

Hi Everyone!

I sure hope you can help with this one...

For the record, I've been using FreeNAS for over 3 years (since 9.x, I'm currently on 11.2-U5(?)) and I haven't tried doing much other than basic NAS functionality. So, I would consider myself to be both a Unix and FreeNAS n00b, though I have both used and worked in the computer industry since the late 1980s (first computer was a TRS-80 with 4K(!) of RAM and a cassette tape drive storage).

It has worked very well through an upgrade from a SATA to LSI 9206-8i HBA upgrade (FreeNAS detected the drives were connected to the LSI card and configured the 6 x 4TB WD Red ZFS2 array without issue). The switch from USB stick (failing) to HDD and then from HDD to the current SanDisk 120GB SSD. The replacement of a bad 4TB WD Red drive (rebuilt the array without complaint). And even the addition of two additional WD RED drives added to the ZFS array, though I had to back up the array, add the new drives, put the old and 2 new 4TB WD Red drives into a 8 x 4TB WD Red ZFS2 array, and copy the data back to FreeNAS server.

The long and short is that it works well, if a bit on the slow side, which is why I am wanting to upgrade from the current dual core CPU to a (new for this system, but used) quad core.

Anyway, my initial server configuration (before the CPU upgrade), which has worked flawlessly (if slowly) until now, is:

AMD A6-5400 CPU
Gigabyte F2A88XM-D3H motherboard (for the 8x SATA ports and support for 4x DDR3 memory)
2 x 8GB DDR3 1333
1 x DVD player
1 x Sandisk 120GB SSD
LSI 9206-8i HBA
8 x 4TB WD Red HDD in a ZFS2 array (32TB total, 24TB net storage)
Generic Silverstone case
550W 80plus power supply

The only thing that I have done between the above flawlessly working (and no errors on the console before the upgrade) is change the CPU from the A6-5400 (dual core with integrated graphics) to an A8-6500 (quad core with integrated graphics, $20 delivered).

So far as I know I have not encrypted these drives...(?)

After the "upgrade" the system will not complete the boot process (see the attached pics).

There are 7 pics below and they are in the order they come up during boot.

Pic #1:


Pic #2:


Pic #3:


Pic #4:


Pic #5:


Pic #6:


Pic #7:


The pics represent any "pause points" in the boot process that I could take pics of. With the old processor, the entire boot process would only take a couple minutes. The last page represents a process loop. It displays a bunch of data, leaving the last screen where it pauses for ~30 seconds before trying again. I have left it here for hours and it continues the cycle.

Do I have to reinstall the OS to get this to work with this new CPU? I've never run across something like this, so I'm at a loss (as I am with just about anything Unix/Linux).

Any help would be appreciated! Please note that "get another CPU/Motherboard/memory/etc." won't help as I had a working system before and don't have the money for other hardware. This was old hardware I had at the time and it has worked well up until now. I still have the old CPU, but would like to see if there is anything I can do to get this new CPU working.

Thanks in advance!

Scott

PS. I REALLY need to get this array working again because, like a moron, I forgot to backup about 5TB of home videos I hadn't edited and uploaded to YouTube yet! I don't have any other copies of these videos! Yeah, I know. I DO know better!

 

[GlueFactoryBJJ]

Oh, I mistyped the LSI controller number, it is a 9207-8i HBA, not a 9206-8i HBA as I posted above. Oops!

 

[GlueFactoryBJJ]

One final pic of the screen where it waits for 30 secs to try whatever it is trying before waiting again. I thought I had included this one before. DOH!

Pic #8:


Scott

 

[GlueFactoryBJJ]

No ideas?

 

[Evertb1]

Your hardware is not exactly the most suitable gear for FreeNAS (to say the least) but OK if that's what you want.

Anyway, if you change things on your configuration, hardware or software, you should make it an habit to dowload you configuration file and store it on a safe place. This before you do anything else. It can be done from the WebGUI. If something goes wrong during you upgrade or whatever, you can reinstall FreeNAS and then upload your configuration file as soon as FreeNAS is running again (if you get it running). It should hardly cost you 30 minutes or so.

If you don't have the configuration safed you should go back to the old configuration (with the old CPU) if that's still possible and see if you get that running again. If you are able to do that then download the configuration file, safe it and start over with your upgrade.

 

[GlueFactoryBJJ]

Thanks for the help! I'll give that a try later tonight/tomorrow. I guess I never expected FreeNAS (and FreeBSD) to be so sensitive to small hardware changes like a CPU. However, I'll take your suggestions to heart for future reference.

I had intended to back up the video files, but the drive I was going to back them up to was about 500GB too small. When I got up the next morning I had forgotten about making room on the drive and doing the backup before making the change. It's what I get for not writing down things that need to be done before upgrades! Sigh!

 

[Tsaukpaetra]

I'm not convinced it's a hardware problem unless your boot drive is corrupted somehow. Those "Can't find crypto util" messages indicate missing files. It might be possible to upgrade to the same version (You _do_ have a backup of your config database as mentioned above, right?).
Regardless, your data should be safe, so it should be a simple matter of getting your boot environment correct.

That's another thing to try, actually; see if in the boot menu you can go back to the prior boot environment (screenshot 4, option 4 there). That might be the fastest quick-check TBH. Well, unless you have no prior boot environment, but that sounds somewhat unlikely.

 

[GlueFactoryBJJ]

I'm not convinced it's a hardware problem unless your boot drive is corrupted somehow. Those "Can't find crypto util" messages indicate missing files. It might be possible to upgrade to the same version (You _do_ have a backup of your config database as mentioned above, right?).
Regardless, your data should be safe, so it should be a simple matter of getting your boot environment correct.

That's another thing to try, actually; see if in the boot menu you can go back to the prior boot environment (screenshot 4, option 4 there). That might be the fastest quick-check TBH. Well, unless you have no prior boot environment, but that sounds somewhat unlikely.

Unfortunately, no, I don't have a configuration backup. Frankly, I'm such a n00b with FreeNAS that I didn't even know it was an option (something I'll rectify as soon as I get the system working again).

FWIW, replacing the original CPU did not help. Same problem.

I'll try Screenshot 4, Option 4 and report back if it works.

BTW, if I were to do a new install, would FreeNAS recognize the RaidZ2 array? If so (and Option 4 not fixing the problem), would I be ahead to just redo the install to a different drive and once I get it up and running, do a disk copy to the SSD and go from there?

Thanks for the response!

Scott

 

[Tsaukpaetra]

So, our options are closing. Your system dataset should have been automatically moved to your first pool, which should contain a backup of your config (though not any encryption keys).

If all else fails, you should be able to clean install, import your primary pool, and recover it (details to be investigated if it comes to that), which should get you most of your configuration (like shares and VMs etc) back.

 

[GlueFactoryBJJ]

My main concern is getting the data back (so I can back it up like I should have before making ANY changes, DOH!). So, doing a new install isn't the worst thing.

What does concern me is that I don't think/remember encrypting the disks. I have had bad luck with encryption on disks (lost key means lost data) and my needs for storage aren't security related. I have the NAS to back up my family's computers and to store home video files that I haven't had a chance to edit/upload to YouTube. I'm hoping that the "problem" is just that FreeNAS is choking because it can't find the crypto library files, not because it needs them to access the array.

I'm waiting for the "Initial Installation" environment to boot now to see if it will fix the problem (hopefully).

Scott

 

[Tsaukpaetra]

I'm hoping that the "problem" is just that FreeNAS is choking because it can't find the crypto library files, not because it needs them to access the array.

Yeah, something just got boofed in the latest boot environment most likely. Your data isn't stored on the boot environment, just the config so once you do get booted things will start looking up.

I'm waiting for the "Initial Installation" environment to boot now to see if it will fix the problem (hopefully).

Ah... is there not one after that? You should be going back to the next-latest one, not the beginning of time....

 

[GlueFactoryBJJ]

Ok, this is new:

Pic #9:


Any ideas what is happening?

And then that finished and I'm back to screen #8. I guess a reinstall is in order?

Scott

 

[GlueFactoryBJJ]

Yeah, something just got boofed in the latest boot environment most likely. Your data isn't stored on the boot environment, just the config so once you do get booted things will start looking up.

Ah... is there not one after that? You should be going back to the next-latest one, not the beginning of time....

Unfortunately, I only had the initial and "default" environments.

Scott

 

[Tsaukpaetra]

The dots mean it was generating your SSH key, normally done once at install so that part is normal.

If it still got stuck then yeah, a reinstall will be necessary. Curious: can you SSHinto the system when it's stuck like that? If so, you might be able to pull the configuration database...

 

[GlueFactoryBJJ]

The dots mean it was generating your SSH key, normally done once at install so that part is normal.

If it still got stuck then yeah, a reinstall will be necessary. Curious: can you SSHinto the system when it's stuck like that? If so, you might be able to pull the configuration database...

The system did continue past the scrolling SSH key generation screen, but then got stuck on the "Crypto.util not found" with the 30 second delay retries (see Pic #8 above). So, at this point I'm assuming that something on the boot drive is hosed? On to a re-install?

Um, "SSH into"? Maybe? How would I do that? As I said earlier, Unix/Linux is not my skill set with computers. Sorry. :(

Scott

 

[blueether]

do you have windows 10? then you can use power shell sna ssh.exe or if you have an older version of windows get a copy of putty

 

[Tsaukpaetra]

The system did continue past the scrolling SSH key generation screen, but then got stuck on the "Crypto.util not found" with the 30 second delay retries (see Pic #8 above). So, at this point I'm assuming that something on the boot drive is hosed? On to a re-install?

Yeah, most likely, unfortunately.

Um, "SSH into"? Maybe? How would I do that? As I said earlier, Unix/Linux is not my skill set with computers. Sorry. :(

Thinking a little further, you probably didn't enable it as a service anyways so it's kinda pointless to pursue, but essentially you'd use a program such as WinSCP or FileZilla, point it at your NAS (logging in as root and your password) and get at what little is available. But, that does require a little forethought and foreknowledge to turn on and configure it.

So, I think the next step is to get a new boot drive and swap the current one out (just in case we can still get the config off it).
Then, clean install onto the new drive and import your primary pool.
From there, we'll adventure into the terminal and do the following:

Code:

cd /var/db/system
ls -l
cd configs-[LotsOfNumbersAndLetters]
ls
cd FreeNAS-[Whatever version you have]
ls
cp [LatestDatedFile.db] /data/freenas-v1.db
reboot

So, each of the commands above needs a bit of attention (hoping the first one works )

1. First one takes us into the System Dataset.
2. Then we take a directory listing. We're looking for configs, there should be at least one. We want the one that wasn't created today.
3. Next we change to that directory, you can use Tab to complete the name to make it easier.
4. Then we list that directory to see what FreeNAS version name it is (In my case it was a FreeNAS-11.3-MASTER-Numbers because I'm on Nightlies) There should only be one but if not choose the latest one.
5. Next we change to that directory (yay)
6. Listing out the directory again. We're looking for the .db file with the latest date, which should coincide with the last time the system was booted successfully (or last time the config changed, I don't remember which).
7. Using that name, we copy it over the currently installed config db file
8. Lastly, reboot

In theory, if all goes perfectly, you'll be back with a majority of your configuration. Let me know if you get that far, or if you get stuck somewhere.

 

[GlueFactoryBJJ]

Mentioning getting a new boot drive reminded me that I still had the original HDD the NAS built on 11.2-U6 and stuck it in. At first, the system would only recognize it as a data drive. So I thought I'd use EaseUS Partition Manager (PM) to erase the partitions and install 11.2-U7 on it. So I put the PM disc in the DVD drive and booted. I wasn't paying attention to the bootable disc "press a key to boot the CD/DVD..." prompt and it just passed on to the HDD and it started FreeNAS!

Long story, short, I was able to sign into the NAS and I'm backing up the data I was really worried about. After it is all backed up (about 10-15 more hours), I'm going to SpinRite the boot SSD (yes, I know about SSD write/durability issues) to ensure it is good and then do a disk copy to a mirrored pair of SSDs for the boot drive.

Back to an earlier post, how do I back up my config db file through the interface? Or do I have to copy it manually using the method above?

After I finish backing up the NAS to my HDDs, should I upgrade to 11.2-U7 or just do a clean install and import the RaidZ2 array?

I am still trying to figure out the CAUSE of the initial problem. If I do a clean install, I'm going to do it with the new CPU because I'm afraid that it may, somehow, have been the cause?

Thanks for all of your help so far!

Scott

PS. Now I'm DEFINITELY going to get BackBlaze to back up my drives!

PPS. Does anyone have any recommendations for software that will find duplicate files on a drive?

 

[Evertb1]

Sometimes reading a manual is not a bad idea. The FreeNAS manual is available by a click on the documentation button in the topmenu of the forum. FreeNAS is a very involved system and asks for users to aquire some knowledge to get the best out of it. For the most functions the manual is a wealth of information.

Paragraph 2.5 of the manual for example deals with upgrade of FreeNAS while paragraph 2.5.2 handles some tasks to perform before you upgrade. Tasks like saving the config file.

By the way. If you do a clean install and upload your config file you won't need to do an import. But please do some reading before you continue. It makes you feel better.

 

[blueether]

if you download the current config I would think that it will be from the hdd install not the current config from using the ssd, unless nothing has changed very much. Knowing this I would try to get the config from the system dataset listed above as that will be the config from the day before it went dirtyside up, unless the sysyem dataset was on the ssd...