SOLVED FreeNAS 11.2 Will Not Boot.

[Janky Jay]

So, I've read through many other posts about similar issues but mine seems a little more unique considering there is no crash. it simply does not boot at all (I'll explain more below).

Hardware:
SuperMicro X7DBE (Latest BIOS that I could find)
2x Intel 2.0GHz CPU (model I cannot remember at the moment but I'll update when I get a chance)
32GB ECC RAM (I've run multiple memtests and everything looks perfect)
LSI 9207-8i/Avago 2308 SAS HBA (firmware 20.00.04.00 so not the very latest but pretty close and I see no reason this would be the issue)
8x 4TB HGST 12Gb SAS HDDs in a single RAIDZ2 configuration

I first noticed this after an upgrade from 11.0 to 11.2 (can't remember which versions exactly as it was a while back and I was able to "magically" work around it somehow).
Everything was running fine until the update. After the update, instead of being prompted with the GRUB bootloader (as it's been removed) it started the little spinning bar (Ex: | / -- \ | / -- \ |, etc... I hope this makes sense?) and then just stops at apparently the same point every time (with the bar in the "--" position). So, figuring something went bad during the upgrade, I decided to do a fresh install and then just upload the config as I have plenty of config backups (and still do). This, however, failed as well many, many times. I was using a converted 11.2-U2 (at the time) ISO to IMG on USB to boot and install to another USB. Sometimes the installation would work, sometimes it wouldn't. I thought, "Maybe this is a USB issue? Brand? Quality? Size? I dunno...". So, I tried many different USB brands and sizes including:

Cheap MicroCenter 16GB and 32GB
PNY 16GB
Kingston 8GB and 16GB
Gigabyte 32GB
Samsung 32GB and 64GB
Transcend 16GB

I also tried burning the ISO to CD and installing from CD to USB but, alas, all with the exact same result. The install would be hit and miss (usually successful, though). However, upon booting from USB, the bootloader would start spinning (very fast, mind you. Much faster than my systems that *ARE* working) and it would halt in the same "--" position. There are no options to do anything as it doesn't make it further than the spinning bar stuck in the "--" position.

At this point, I figured, "Well, I'll just try the 9.10-RELEASE and see what happens.". To my amazement, the 9.10-RELEASE installs and boots perfectly every single time (I did this a lot trying to troubleshoot where everything was going wrong). Even after upgrading to the latest 9.10 after the initial install, everything worked perfectly. Of course, after this, I'm curious and think, "Well, let's shoot for an update from the latest 9.10 to 11.2-U2 and behold, everything works perfectly fine. Boots right up, I configure everything from scratch (just in case) and all is good. That is, until the 11.2-U3 update.

Once the 11.2-U3 update rolls around, I think I'm safe so I upgrade (which goes without a hitch) but, again, it's now stuck at the darn spinning bar in the same darn positiion! Aargh!

Again, like an insane person, I try multiple different USB stick brands and sizes with the exact same results as before only to end up having to do what I've done in the past. Start with 9.10-RELEASE, update to latest 9.10, then upgrade to the latest stable 11.2 train. This takes a while but it went without issue. However, being as I'd configured a bunch of stuff and didn't want to start from scratch again, I attempted to upload my config to the 11.2-U3 system after all the upgrades and *BAM*, stuck at the spinning bar *AGAIN*! There is absolutely nothing I can do at this point. There is no option to boot a previous version. No keystrokes prompt me for anything at all. It just simply attempts to run the bootloader and fails. I cannot generate any logs or troubleshoot what is wrong. I'm at a total loss as to what is going on aside from something in the 11.2+ bootloader perhaps doesn't like my hardware. However, why does it work before I put my previously working config into place? I've also noticed that while the GRUB boot loader stays in place (or at least the menu appears) when upgrading from 9.10 to the 11.2, everything works fine. If I try a fresh 11.2, I never get that menu. Also, that menu seems to go away after I've uploaded my latest working backup config when going the 9.10 -> blah blah blah -> 11.2-U3 route.

I do know that if I were to do this all over again (install 9.10-RELEASE, upgrade to latest 9.10, then upgrade to 11.2-U3 and reconfigure from scratch), that it would all work again. But, why? Why won't a fresh install of 11.2 boot? Why won't an upgraded from 9.10 to 11.2 work after a config upload boot? I know this isn't a hardware failure problem as the 9.10 version works EVERY. SINGLE. TIME. There appears to be a serious issue with the 11.2 bootloader that is preventing my system from booting like it should and I can't for the life of me figure out how to troubleshoot anything as once the new bootloader is in place, I cannot proceed.

Granted, this system *IS* is a backup system for a backup so it isn't incredibly time sensitive but I would really like to get this system back up and running again. Has anyone seen or heard of anything like this? Is there any possible way of troubleshooting this if I can't get the bootloader to behave at all? I'm all out of ideas so if anyone has any thoughts I am most certainly all ears. Also, if there is any additional information that I failed to mention or that you would like to see, please let me know.

Aside from this, FreeNAS has been running fantastically for years on this machine (and many others) and I'm very thankful for all the hard work and dedication to this project. I'm currently trying to get iX Systems in the door at my work which would be great for all parties involved! The trouble right now is getting all the NetApp and Nimble (now HPE, of course) fans to just simply listen and take a look at what FreeNAS/TrueNAS has to offer. I'm working on it, though!

 

[Snow]

Do you have any Option Rom's enabled? Disable all of them. Then try to load 11.2-U3 see if that helps. I had some problems with booting on 11.2-U3 on my main box as well. I've seen the Option Rom's do some funky stuff.

Side note may want to pick up a SSD for your boot drive as USB are no longer a Good choice in the newer versions of freenas. Due to the Increased read and write loads on the boot device.

 

[Janky Jay]

BIOS had been previously reset to factory defaults and left as-is. The only change was to disable every boot device aside from USB. USB boots perfectly fine every time. FreeNAS 9.10 boots perfectly fine every time. FreeNAS 11.2 and higher chokes on bootloader every time.

EDIT: I should add that FreeNAS 11.2 chokes on the bootloader every time _ONLY_ on fresh installs. If I get to 11.2-U3 via upgrading from 9.10, it boots fine every time (until I upload my saved config).

Also, thank you for the quick reply. :)

 

[Snow]

I think they may be enabled by default does not hurt to try and check only take 2-3 secs . you can go to system > advanced > save debug and see if you can find what is happening there ? it saved as Tgz let me see if I can find where the log is for booting.

it is in ixdiagnose\fndebug\Loader\Dump You may want to post this Dump and ask some one that has much more Knowledge of what's right in here and what's wrong hope this helps.

 

[Janky Jay]

Ah. You're talking about additional stuff within FreeNAS itself. The answer to that would be no. It was a standard install used only for zfs send/receive as a backup. No additional software changes or add-ons at all. I set the IP address, enabled what I needed to for either rsync or zfs send/receive and that is all. Also, there is no way for me to retrieve any logs. I am unable to boot into the system, unfortunately. Thanks again for the replies, though.

 

[Snow]

can you get it to load with 11.2-U2 ? sounds like you tried 11.2 &11.2-U3 ?

 

[Janky Jay]

I cannot get 11.2-U2 to boot either (fails the same way at the bootloader). The only way I would be able to get 11.2-U2 to boot is if I upgraded to it from 9.10. However, that doesn't seem to be an option as it will just updated to the latest in the train. So, it'll jump to 11.2-U3.

 

[Snow]

When did you update the bios for the motherboard, After you had the problem ? Just as a test try this, install freenas to one of your USBs off of a different PC then Install said usb with Ver 11.2-U3 and see if the other system boots off of it.

 

[Janky Jay]

It was actually quite a while back. I did look for another, newer BIOS for the motherboard but I cannot find one. In fact, the BIOS version I'm running now is newer than the newest version I could find anywhere online.

 

[Snow]

DId you try the work around posted above, did it boot? Oh yeah Make sure after you've install it to the Usb to make sure you do not boot off that usb with the install pc. if you did it is not that big of a deal just default the Config & reboot then upload your old config.

 

[Janky Jay]

DId you try the work around posted above, did it boot? Oh yeah Make sure after you've install it to the Usb to make sure you do not boot off that usb with the install pc. if you did it is not that big of a deal just default the Config & reboot then upload your old config.

Sorry. I didn't see your edit. Yes, it boots on other systems without any issues at all. I've tried this with both brand new 11.2-U3 USB sticks as well as the one with my config loaded on it. This is a very strange issue to me.

 

[Snow]

Sorry. I didn't see your edit. Yes, it boots on other systems without any issues at all. I've tried this with both brand new 11.2-U3 USB sticks as well as the one with my config loaded on it. This is a very strange issue to me.

Did it work for your system with the problems ? sorry did not see if it did.

 

[Janky Jay]

I'd actually tried that a few times before I posted my issue here but I just tried it again just in case. Still the same issue. It does not work in the system I'm having issues with. Still hangs at the same spot.

 

[Snow]

What Bios Ver is it ruining?

 

[Janky Jay]

RIght now, it's 2.1c. Most recent I was able to find was 2.1a (and it wasn't from SuperMicro so I was hesitant to try it anyway).

However, while I was sitting here thinking about what was happening, I may have resolved my issue...

I started thinking about the VERY fast spinning during boot which made me think, "Okay, the bootloader MUST be looking for something, can't find it, and immediately starts spinning like crazy and then failing... But, what the Hell is it looking for and not finding? Or, maybe, what is it finding that it shouldn't be?". So, I decided to take a closer look at what was happening during boot. I happened to see the HBA initialization and it got me to thinking that maybe the bootloader was seeing the additional disks and looking there for something instead of the USB itself. So, I thought, "To Hell with it. Let's just check the HBA settings and see what's going on in there.".

So, I reboot and used CTRL+C to enter the HBA configuration and everything looks pretty normal. However, I did notice that the "Boot Support" was set to "BIOS and OS" which I generally have set to either "BIOS Only" or "Disabled" in a case like this (a case where the HBA is not housing anything that should be booted from). So, I set it to "BIOS Only" and reboot. No changes, same issue. So, blah... But, I figured, well, just to be sure, I'll just disable "Boot Support" completely so the bootloader isn't seeing these storage drives and thinking they're something to use for booting. So, again, I go back into HBA configuration and set the "Boot Support" to "Disabled", save, and reboot. VOILA! Everything comes up completely normal!

I import my saved config (again, just to stress myself out), reboot, everything goes as planned and the system comes back up again (hooray!). I ran a few quick disk tests, checked the data and just for good measure, rebooted again. Sure enough, system boots without a single hiccup. So, this whole time I'm having these issues, it's because the bootloader was seeing my data/storage disks and either thought they were something they weren't or tried to use them as something they're not. Either way, this appears to be the culprit.

I'm very happy that everything is up and running again but it still makes me wonder what on Earth was going on with all my previous attempts at installations and upgrades/updates? Why did some work and some not? Why was FNAS 9.10 (even though it wasn't the original version of FNAS I installed) able to boot every single time without any issues? My guess is this goes back to the GRUB bootloader in the older versions. Perhaps GRUB was just a little smarter (or dumber?) about where to look or what to use. Not sure.

Anyhow, just an FYI to anyone who overlooked this (I feel like an idiot since I've done this 1000's of times in the past, literally), don't forget to disable your "Boot Support" on your HBA's, kids! Well, unless you're using them to actually boot.

Thanks again for trying to help and for your patience, Snow. I really do appreciate it!

 

[Snow]

No problem man I will say we have all done some thing like this and chased our tails. Some times walking away or even talking to some one about it helps. I had this same problem do to Opt Roms. I also change some watch dog power setting that would not let me boot with Ver 11. All I can say is I am happy your Stuff is up and working!!!

On a side note I would still replace your USB with a SSD you can find 120Gb for under 30 Bucks. Here is a pic of all the failed USB'S that have Cause me pain & had to reinstall FN. I am surprised they have not change the standard to show that USB'S are no longer supported with the newer versions of freenas.

 

[Janky Jay]

No problem man I will say we have all done some thing like this and chased our tails. Some times walking away or even talking to some one about it helps. I had this same problem do to Opt Roms. I also change some watch dog power setting that would not let me boot with Ver 11. All I can say is I am happy your Stuff is up and working!!!

On a side note I would still replace your USB with a SSD you can find 120Gb for under 30 Bucks. Here is a pic of all the failed USB'S that have Cause me pain & had to reinstall FN. I am surprised they have not change the standard to show that USB'S are no longer supported with the newer versions of freenas.

View attachment 30113

Thanks! And, wow. That's quite the pile of USB sticks. Not only are USB supported, but they're recommended. I do agree that anything larger (in size) is an incredible waste of space but due to failure rate, USB can be rough and also make troubleshooting a bit of a hassle. SSD's will certainly last longer but, I'm my case, unfortunately, I have no free bays for one (and no 2.5" bays either). So, I'd have to go external which can be tough in a tight rack. As long as I have backed up configs, I'm a happy camper! If I did have internal space, I'd certainly go with mirrored SSD's, though. Thanks again!

 

[Snow]

Double sided sticky tape to a 2.5 to 3.5 slim tray off amazon works best. As long as you have a open Sata port. Both my 4U and my 3U Have this done to them Ever since I switched from usb I have not had a boot failure. Like you I planed out all my bays so did not have a extra slot open for one. Tray link this way if I need to remove the SSD for a RMA I do not wreck it.

 

[Janky Jay]

Double sided sticky tape to a 2.5 to 3.5 slim tray off amazon works best. As long as you have a open Sata port. Both my 4U and my 3U Have this done to them Ever since I switched from usb I have not had a boot failure. Like you I planed out all my bays so did not have a extra slot open for one. Tray link this way if I need to remove the SSD for a RMA I do not wreck it.

If there were anywhere inside the chassis the place this, it would work. Unfortunately, no dice. And, of course, all my standard bays are taken. Unless you're mounting this outside the chassis all together and running the SATA cable into the chassis via a hole somewhere or something?

 

[SynbiosVyse]

Thanks! And, wow. That's quite the pile of USB sticks. Not only are USB supported, but they're recommended. I do agree that anything larger (in size) is an incredible waste of space but due to failure rate, USB can be rough and also make troubleshooting a bit of a hassle. SSD's will certainly last longer but, I'm my case, unfortunately, I have no free bays for one (and no 2.5" bays either). So, I'd have to go external which can be tough in a tight rack. As long as I have backed up configs, I'm a happy camper! If I did have internal space, I'd certainly go with mirrored SSD's, though. Thanks again!

USB sticks are not recommended anymore, you are reading outdated recommendations. You should not use a USB stick for FreeNAS, they are far too unreliable. See if you have an empty M.2 slot or two on your motherboard, SATA DOM, or do some expansion with a PCie slot or something. There are plenty of creative ways to get additional slots available.