SOLVED IPMI (SOL) Console Redirection Issues & GUI Enable Serial Console - Port Not Available

[zenon1823]

I have been working on this for 2 days and have exhausted every post here and in several freeBSD forums trying to figure this out and I'm out of ideas. I am working on a new build with no configuration or data so I can try anything suggested. Once this system is installed and working properly I'll add move my drives into the disk shelf and import the pools. I won't go into detail of every challenge I ran into or everything I've done as that would be a book, so I'll include here what I think is relevant to the current status. This is the first IPMI and SOL configuration I've attempted on my own so I'm learning as I go.

System Details:
Dell R210 ii Chassis | e3-1220v2 | 8GB ram | 2x 32gb USB keys on internal ports - boot mirror | Dell H200E crossflashed to LSI9200-8e | Data drives will be in a Dell MD1200 disk shelf
Installing/Installed Freenas 11.2-u5

I believe the problems stem from a mismatch in baud rate or serial communication parameters, but I am not freeBSD savvy enough to figure out the details.
Below is my bios SOL configuration. There are no issues with my bios SOL (that Ive identified).

I am initiating my SOL connections from an ubuntu VM using ipmitool via an ssh session.

I was not able to install freenas via SOL, I was able to see the GRUB install menu via SOL, but when I would select "installer (Serial Console)" it was hit an miss to even get to the blue screen for install. I did a couple times but wasn't able to progress through the steps reliably.
Comments on this are welcome as I'd like to know how to do it, but I gave up on this and installed it via local console, and am now attempting to get SOL working reliably going forward.

Currently it defaults to boot without console, my SOL console displays everything upto and through the bootloader screen and then is blank when it moves forward from the bootloader. This is perfectly expected behavior - All Good

If I select to boot with Serial Console from the bootloader I get boot messages on the local console and eventually to the running screen with options 1-11, but the SOL console advances blankly for a few screens then it seems to die. I cannot ~. out of the SOL console or ctrl-c, I need to kill that ssh session. After logging in with a new ssh session if I run the command to deactivate the SOL it usually tells me the SOL payload is already deactivated. If i activate a new SOL I can see the running screen with options 1-11 and use it normally, a reboot kills the SOL connection and I need to re-establish as described above.
While this is not the end of the world since I have SOL though Grub and after it finishes loading, it seems something is definitely not right. and I am blind to troubleshooting a boot issue remotely,

Here is what I tried next:
In the freebsd handbook (section 26.6.3) it suggests adding the following to /boot/loader.conf to change the default 9600 baud setting.

Code:

boot_multicons="YES"
boot_serial="YES"
comconsole_speed="115200"
console="comconsole,vidconsole"

With that change when I let it boot the default (non-console) option in Grub I see some output duplicated on both the local and SOL console while other output is only on one of the consoles. In the end both consoles display the run screen with options 1-11 so this seems to be the best implementation of this I have found, but I really don't know if there are possible nuance problems I may have created as issue down the road. (like when I upgrade)

I was hoping to achieve a mirror of the local console over SOL, but it seems that in my reading this is not how it works as it treats serial and local as its own tty sessions. If so thats fine but I thought id put out the question.

Now lastly in the freenas web gui under System>Advanced there is the option to "Enable Serial Console" Its unchecked and I cannot enable it because when I check it asks me to select a serial port, but the drop down is empty for choseing a serial port. What does this do exactly, it would seem I have serial console so is it a bug or what have I done?

Sorry for the length, but I hope I included sufficient detail, and hopefully this will help someone save alot of time in the future.

 

[zenon1823]

I'm going to bump this, but also to ask if the admins feel it should be moved to the networking forum? When I started writing the post I was struggling to successfully install over an IPMI console which is why its in the installation forum, but as I continued troublshooting on my own its evolved to more of a general Serial-Over-Lan networking question.

Thanks

 

[dlavigne]

Were you able to resolve this?

 

[zenon1823]

No Im still looking forward suggestions re 3 questions:

Unable to install via SOL

Post Install Console boot kills SOL, SOL can be reestablished after fully booted.

My manual edits to /boot/loader.conf work, but uncertain if they will persist through an upgrade or cause issues during upgrade.

 

[zenon1823]

I'm going to bump this thread and ask for some insight, as I'm done testing and ready to build this system for production now.
Initially I put posted in this forum because of issues installing over IMPI, that remains a problem, and I welcome ideas, but I'm more interested in proper SOL functionality. Admins - Please move this to appropriate forum if you feel that's a better location.

To summarize the important issue details:

If I select to boot with Serial Console from the bootloader. The local console shows boot messages and eventually the screen with options 1-11, but the SOL console advances blankly for a few screens then dies. I cannot ~. out of the SOL console or ctrl-c, I need to kill the ssh session running the SOL connection. After logging in with a new ssh session and activating a new SOL session I can see the running screen with options 1-11 and use it normally, a reboot kills the SOL connection and I need to re-establish as described above.

According to the freebsd handbook section 26.6.3 it suggests adding the following to /boot/loader.conf to change the default 9600 baud setting.

boot_multicons="YES" boot_serial="YES" comconsole_speed="115200" console="comconsole,vidconsole"

With that change when I let it boot the default (non-console) option in Grub I see some output duplicated on both the local and SOL console while some output is only on one of the consoles. In the end both consoles display the run screen with options 1-11 so this seems to be the best implementation of this I have found.

I can't imagine this would be cause any stabilty risks, but is this a good solution long term and are there risks to upgrading freenas with this custom loader.conf file.

 

[jgreco]

Well, it could stop working at some point, but probably isn't actively dangerous. FreeNAS will eventually overwrite your loader.conf so you have to expect that. I don't happen to remember the specific cases where this will happen, but very likely upon update...

I was one of the folks who instigated the serial-and-VGA console craze in FreeBSD but it has been a bit hit-or-miss over the years.

Not really clear why you want to use SOL for this. If you have full KVM IPMI, SOL is pretty crappy in comparison. It mainly has the advantage of being able to capture panic messages etc.

 

[Ericloewe]

Big advantage of the serial console: The output can actually be captured in a reasonable format. Small size and being able to grep stuff can save a lot of time when things have gone wrong, which they probably have if you're using SoL.

Now, some comments about FreeNAS: At least on a good, working install, activating the serial console should just be a matter of turning it on in the GUI and everything should just work, once you've figured out which port is what in IPMI. There is a catch: It's fairly likely that some messages will not end up on the serial console. Last time I filed a bug for this, it was deemed too difficult to solve - I should note that it used to work before an update, so it's clearly not difficult-difficult, just too-much-research-for-my-deadline-difficult.

 

[zenon1823]

Thanks guys,
I don't have a full KVM or the full Dell Enterprise Idrac, this is for my house and the R210 only has the BMC so I'm working with what I've got. My reason for wanting it is that I travel a fair bit I like to have an Oh S*** measure available, and its also nice to not have to travel 4 flights of stairs so im a little lazy too. I was afraid that it would overwrite it on update, and since that when its likely for something to go wrong, it wasn't what I wanted to hear (even though its what I expected).

Eric, you actually reminded me of one of the questions I noticed while messing around with this config. I don't have the option to enable SOL in the GUI, when I click the check mark for it it asks me to select a serial port, but no serial ports are listed to choose from. Have I likely ran across a bug or have I likely messed something else up?

 

[Ericloewe]

Probably not a bug, it's more likely that the serial ports are disabled or otherwise obfuscated by the BMC. Maybe a very non-standard address or something...

 

[zenon1823]

I finally got back to this project today and as a final chapter and to perhaps help someone else I thought I'd report back. I installed FreeNAS using the latest release 11.2U6 (I was using u5 previously). And the Serial console install option worked without issue, following installation my SOL terminal was no longer locking up on boot. I was still getting some weird overwriting of lines rather then proper scrollback, but after a couple hours I thought maybe I should update Putty and viola, I have perfectly functioning SOL.
**LESSON LEARNED - Dont Take Putty Forgranted

Funny enough the first thing I did was look at the loader.conf and sure enough it has the exact configuration I was editing manually previosuly. There was absolutly no change in my hardware or configuration, so I can't explain what changed between u5 and u6 with regard to this and there is no reference in the changelog that would make me think this was addressed, but I'm very happy regardless.

Eric, I'm still confuzzled about the SOL option in the GUI, my gui does not allow me to enable SOL because there are no ports available to choose from, but yet it works perfectly. Everything looks perfectly normal wrt bios/BMC config, my bios SOL setting is redirecting to COM1 and COM1 is using 3F8 IRQ4. I'm not overly concerned since is working without me doing any customization to the loader.conf, but curious none the less.

 

[Ericloewe]

Maybe it's a GUI bug. Do you get a graphical or a text mode console on the monitor?

 

[zenon1823]

My bios config is shown in the first image of the OP.

1st screen capture below:
I'm getting a text mode console over SOL

2nd screen capture below:
The BMC boot summary confirms standard port/IRQ addressing 3F8

3rd screen capture below:
The FreeNAS 11.2U6 config for serial console. The error in red appears when I try and save the settings, as you can see there are no serial ports available in the drop down. (whats weird and different from U5 is that when I go into this screen there is already a check in "Enable Serial Console" so I could only save the settings on the screen if I remove that check. In U5 this was unchecked after install and I wasn't able to put a check in.) So in summary, its checked now, but if I try and save the settings as is it give me an error. I haven't tried saving it with the check removed as I don't want to risk losing the functionality since I don't know how to get it back short of reinstalling.

This is feeling like a bug, or at the very least an outling configuration anomaly. Thoughts?

 

[dlavigne]

It may be this bug which has been fixed for the upcoming 11.3: https://jira.ixsystems.com/browse/NAS-101984.

 

[zenon1823]

Thanks DLavigne, Is there an 11.3 beta/alpha you would like me to validate against? This machine is not in prod yet so I can update it and do some testing if its helpful. I'm not concerned really as I have the functionality i need, but happy to help the community while this machine is still in testing.

 

[dlavigne]

That would be great. Right now you can try installing a nightly (say, to a USB stick) to run your test. We're still a couple of weeks away from BETA1 (ETA is Nov 5).

 

[zenon1823]

Is changing the train in the gui to nightly and upgrading going to be just as effective of a test? Thats easier for me then getting to the server physically?

 

[dlavigne]

If you do that, be sure to reboot back into your former 11.2 boot environment after your test. Otherwise you won't be able to update to 11.3 once the 11.3 train is created (that will happen with BETA1).

 

[zenon1823]

I just applied updated to 11.3 by way of nightlies as you suggested and I'm happy to report that my SOL worked seemlesssly through and after the upgrade, additionally the issues where the serial ports were not displaying in the gui is also resolved.

Also I really like the feel of 11.3 excited for it to be GA. The reporting is especially improved over the lag and display errors in 11.2.