Workaround/Semi-Fix for Mountroot Issues with 9.3!

Status
Not open for further replies.

DJL

Cadet
Joined
Dec 31, 2014
Messages
1
Thanks - this sorted out problems I had with the 9.3 RELEASE. Symptoms were different -

No handlers could be found for the logger "freenasOS.Configuration"
freenasOS.Installer.InstallerPackageNotFoundException

This was on an HP gen8 microserver with a Kingston DataTraveler USB drive.
 
J

jkh

Guest
If someone can get a specific USB device and a specific motherboard combo which exhibits this problem, and said combo can actually be purchased on the open market (DrKK's system was being built for someone else, for example, and is not something he can just send us) then please let me know and I'll whip out the company credit card to do so. None of the HW we have at iXsystems does this, and Sean even obtained a really "old and crappy" AMD system with an eye to reproducing this but, of course, it worked flawlessly. :(
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
@DrKK, Post your hardware that didn't work!
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
OK. It was a SuperMicro X10SLL-F that I bought at Micro Center. The boot device was an 8GB Kingston Micro DT attached to the rear USB ports. The have one more there. If you want me to buy it, and then get reimbursed, I will.

Problem disappeared when I put the boot device on the on-mobo-USB port in the middle of the mobo, but would not disappear on the other ports until I set the delay.
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630
I actually have a spare Kingston Micro DT 8GB thumb drive. It's not the same one as the guy had, but it's the same make/model as far as I can see.

Point being, all we need is the mobo.

But of course, you know, and I know, that when I go out and buy this mobo, it will immediately cease happening.
 

9C1 Newbee

Patron
Joined
Oct 9, 2012
Messages
485
I did NOT have the problem with my X9SCM-F-O booting on a Lexar 16G. It worked on the USB port on the front of the machine as well as the internal location.

Maybe if we gather as much information as we can on what DOES and does not work, it might provide a clue.

So, this issue is introduced due to GRUB?

If I get time, I will see if I run into this issue on the Dell T105. If I do, I may be heading that way soon. I could drop it off to you guys if you wish.
 

9C1 Newbee

Patron
Joined
Oct 9, 2012
Messages
485
I would assume the bios ver might be worth noting as well.
 

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175

Rich N

Cadet
Joined
Feb 10, 2014
Messages
6
Great job. I have not tried this yet but i was wondering if there is harm in setting up the turnable prior to upgrading?
 

Lazuruz

Cadet
Joined
Jan 1, 2015
Messages
1
Esteemed Community:

Many of you with bootable USB drives have been reporting that you are having difficulties getting FreeNAS 9.3 to boot, even on recommended hardware, and that the process dies at the infamous "mountroot" prompt. We have had this report now on at least three different recommended SuperMicro models (X10SLL, X10SLM, and X9DRD-7LN4F), and since I was personally affected as a friend of mine asked for a new FreeNAS build on bare metal (X10SLL), and I had the problem personally, we put in some effort to track it down. In the bug reports, I had called this a "fix", but, (after Jordan yelled at me for the hubris of calling it a "fix") we're going to downgrade to "workaround" until we get more clarity on what's going on, and the senseis at iXsystems can do more investigation.


But the good news is: many/most of you using recommended hardware getting the mountroot error on boot should now be able to boot, for the foreseeable future, using the following guide. Your hardware is fine; the problem has more to do with the peculiarities of the booting sequence, and the fact that you are using the USB bus to boot.

NOTE: Even if you're not using recommended/server grade hardware, there's a good chance this will work. This problem is one that will probably be ubiquitous across many motherboard models and chipsets. In any case, you can't hurt anything by trying this, if you're having a mountroot issue. Go ahead and try, and if it works, reply with your motherboard make and model so that others might search and find the discussion.

As is usual for me, I'll explain both what to do, and try to give some insight as to what the hell is going on while I am doing it. A disclaimer though: we are talking about parts of the process that I, personally, do not have a very good grip on, so I may say some horsecrap in my attempts to explain what's going on. Even so, the workaround process I outline below should solve your problem.

So here we go.

First of all, if I understand Cyberjock correctly, when you boot, this is a multi-staged process. At first, the BIOS recognizes your boot devices, and reads sectors specifically designated to get the initial stages of the operating system going. GRUB (which can then hand off to a variety of operating systems) then takes over, and at this point, your boot USB device is re-recognized in the context of a FreeBSD device (like /dev/ad0).

Here's the problem. Apparently, for whatever reason, there is a small bit of time/lag that occurs between when the GRUB takes over, and when your thumb drive is *actually* accessible in a /dev/da0 context. If, at the point that access is attempted, and it's not available, THAT is when you drop to mountroot, and then you're screwed.

So here is what you do:

Step 1: GRUB loads.
View attachment 5985

Step 2: When you get to the GRUB screen (this is the screen that pauses for 5 seconds with the FreeNAS 9.3 option highlighted), IMMEDIATELY HIT THE ESCAPE KEY to stop GRUB.

View attachment 5986

Step 3: Having done this, hit "e" to edit. You will now see a list of what we call loader tunables that over-ride default behaviors in the loader, and you will be in a rudimentary text editor similar to "nano" (if you are familiar with it).

View attachment 5987

Step 4: You will need to add your OWN tunable here, manually. That tunable is as follows:

set kFreeBSD.kern.cam.boot_delay="50000"

This is shown below. Note: do not scroll down too far, you want this entry to be in the "Normal Bootup" entry in this file. Doesn't matter really where within that, but do not go so far that you are outside of that zone.

View attachment 5988

This number 50000 is almost certainly much, much more than you actually need, but we think it would be impossible to need *more* than this, so let's just start with this number.

Step 5: At this time, press F10 to BOOT. For this one boot (and this one boot only), your tunable will be active. When prompted, select "normal boot". Now, the GRUB will pause for 50000 milliseconds (i.e., 50 seconds), which is obviously ample (obscenely ample) time to let the USB bus do whatever it has to do.

View attachment 5989

And you should boot now!!!! woo hoo!


Step 6: NOW, once you're booted, the first thing you're going to want to do is to go into your FreeNAS GUI, and go to system->tunables and add a tunable, with category "loader". The name of the loader tunable is exactly as above minus the kFreeBSD, in other words, kern.cam.boot_delay, and the value for the loader tunable will be 50000. Hit Save/OK (whatever), and now this should be a persistent setting. (In fact, you should actually see it, if you were to reboot and start the GRUB process and go into the edit screen).

Play with the numbers (i.e., lower it by 10000 at a time). More than likely, 10000 will not be enough, but 20000 or 30000 will.

You're all set! At least until such time as the devs determine if this is actually the appropriate long-term fix, or the devs figure something else out.

Please, if this has fixed your problem, let us know right here. Thanks to Cyberjock for working jointly on this with me.


Made an account for this.

Thank you!

Austin
 

derekzchu

Dabbler
Joined
Dec 5, 2014
Messages
23

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
You might want to try the USB 2.0 ports. The USB 3 drivers aren't enabled by default in FreeNAS, *but* the USB3 hardware *should* work at USB2 speeds. Of course there are no guarantees, hence the 'but' and 'should'.
 
Last edited:

derekzchu

Dabbler
Joined
Dec 5, 2014
Messages
23
that worked. whew thanks!
 

DrKK

FreeNAS Generalissimo
Joined
Oct 15, 2013
Messages
3,630

Chamrajnagar

Explorer
Joined
Jan 2, 2015
Messages
89
I ran into this issue with the latest release on a Supermicro A1SAI-2750F-O. Like derekzchu, I was using a USB3.0 port (had been hoping to use the motherboard's internal port). Changing to a USB2.0 port appears to have fixed the problem.
 

Rich N

Cadet
Joined
Feb 10, 2014
Messages
6
Rich, the answer to your question is "no", there's no harm in it.

BUT, the real answer is this: As far as I know the latest STABLE (get your install .iso from http://download.freenas.org/9.3/STABLE/) already has this worked in, as per the closed bug ticket.


DrKK, You are correct there are two STABLE releases with workaround integrated. I performed a CD upgrade (from bootable usb) and i'm now rocking 9.3 that boots!

Thanks for the response.
Rich
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
OK. It was a SuperMicro X10SLL-F that I bought at Micro Center. The boot device was an 8GB Kingston Micro DT attached to the rear USB ports. The have one more there. If you want me to buy it, and then get reimbursed, I will.

Problem disappeared when I put the boot device on the on-mobo-USB port in the middle of the mobo, but would not disappear on the other ports until I set the delay.
Your last sentence suggests there might be a compatability problem with one of the two usb controllers
on that board.

C222 chipset specs pdf

Table 5-46.
USB EHCI Host Controllers (D29:F0 and D26:F0)
The PCH contains two Enhanced Host Controller Interface (EHCI) host controllers which support up to fourteen USB 2.0 high-speed root ports. USB 2.0 allows data transfers up to 480 Mb/s. USB 2.0 based Debug Port is also implemented in the PCH.

This is just a plumber who is shooting in the dark here, but just thought I'd ask the question...
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Not exactly. When I hear "compatibility" I think something along the lines of the hardware and software not being compatible because, for example, drivers aren't functional with the hardware resulting in problems.

This is more of a timing issue. On bootup lots of things are going on throughout the bootup sequence. The problem seems to be that the USB devices aren't actually detected and available to the system yet at the time in the bootup that the devices need to be available to continue the bootup sequence. So the box ends up at the mountroot prompt because the OS wants to continue booting, but no boot media is available yet.

All this workaround does is insert a delay in the bootup sequence to slow down the bootup at a particular step to ensure the bootup device has been detected. Its definitely not the most ideal because we're throwing an arbitrary delay in there (in this case kFreeBSD.kern.cam.boot_delay="50000" adds up to 50 seconds of delay). It's not the cleanest way for things to work. Ideally you'd delay the boot only as long as it takes for the boot device to be available to continue to the next step, but that's not what this does.

There obviously should be a better way to handle his, but nobody knows what it is at this time. I'm pretty sure jkh is going to look into this further. I just don't know what the long-term fix is or should be. That's for the devs to figure out ;)

There's no reason that setting the boot delay can't be a long term fix except for the fact that it's a pretty sloppy solution.
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
This is more of a timing issue. On bootup lots of things are going on throughout the bootup sequence. The problem seems to be that the USB devices aren't actually detected and available to the system yet at the time in the bootup that the devices need to be available to continue the bootup sequence. So the box ends up at the mountroot prompt because the OS wants to continue booting, but no boot media is available.
Perhaps compatability was a bad choice of words, I just thought it was odd that when the boot drive was place
in the internal USB port, the issue disappeared. Perhaps, while shooting in the dark, I managed to shoot myself
in the foot! :D
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Perhaps compatability was a bad choice of words, I just thought it was odd that when the boot drive was place
in the internal USB port, the issue disappeared. Perhaps, while shooting in the dark, I managed to shoot myself
in the foot! :D

That pretty much mimics what DrKK saw. More than likely the internal USB plug is just detected early enough in the sequence that it does work, while the external ports are one of the last devices detected in the system.
 
Status
Not open for further replies.
Top