Issues with serial console + PuTTY - and became unable to complete boot or get to single user/boot environment choices

[Stilez]

I'm trying to set up a serial console in parallel to the usual VGA console, to capture system/kernel messages for bughunting. There have been a couple of kernel panics that didn't create a textdump/crashdump and I've been suggested to do this to capture better info.

I've *somewhat* got it working but can't seem to fix 3 problems. maybe someone else understands serial better than I do:

1. When I hit shell, I find the environment variable TERM is incorrectly "vt100". I copied this config directly from my SSH login config in PuTTY and it definitely defines TERM=putty. and works correectly in SSH. I've checked, tried all the things, I can't seem to specify my terminal when opening a serial connection in PuTTY, I have to manually type TERM=putty and reload my shell startup script (source SCRIPT), to get "normal" look and feel back. How do I ensure the term type is correct when it starts a PuTTY serial console session?
   
   
2. Any full window command (mc, ee) reverts the screen size to a tiny size, doesn't use the PuTTY console window full size, or recognise console window resizing. Again, config copied from SSH config where it works fine.
   
   
3. Function keys don't work, so I can't exit "mc" using F10 for example

Those seem to be my problems with the console itself.

Once they are solved, how do I ensure console messages currently shown on my VGA console 1st window, also get sent to the serial console (if connected)?

 

[Samuel Tai]

Setting it in PuTTY itself doesn't work?

 

[Stilez]

Setting it in PuTTY itself doesn't work?

No. It doesn't. I've got that entry from the SSH login which I copied as a template, and rechecked it. Its not doing anything. It still shows TERM=vt100, ncurses screen size issues, function key fails, when I log in, which breaks things.

 

[Samuel Tai]

OK, this appears to be set in /etc/ttys. All the serial terminals default to vt100.

 

[Stilez]

That looks like the one. Thanks!!
The screen size one, I found is because serial doesn't have a resize negotiation. There are scripts that have the console report its size to set it in the ttys settings.

So its getting there

 

[Stilez]

OK, this appears to be set in /etc/ttys. All the serial terminals default to vt100.

Now I need help.
So, serial console is working in PuTTY. That is, I can connect from putty with "serial" and I get a session via ttyu0 - serial UART port.

I then enabled serial console in the Web UI, saved and rebooted - and my entire NAS is prompty unusable.

It boots, and thats reflected on the serial console. But it never displays the usual startup type options so I can't select "Single User", boot environment, rollback, or any kind of prompt.

Then after it displays this on the srial console, nothing more is displayed on the serial console again:

Setting currdev to zfs:boot-pool/ROOT/12.0-BETA2.1:​

Loading /boot/defaults/loader.conf​

Loading /boot/defaults/loader.conf​

Loading /boot/device.hintsg pools: Main_pool boot-pool​

Loading /boot/loader.confns​

Loading /boot/loader.conf.local​

|​

Activity continues on the VGA console. It appears to boot all the way into init, past the point where it recognises disks, brings the networking up and creates devices "GEOM_MIRROR: Device mirror/swapN launched" and then GEOM_ELI: to create them as encrypted.

And then, nothing more. Like, absolutely nothing. The keyboard remains responsive to caps lock, but not to CTRL-D/Z/C. CTRL-ALT-DEL forces a reboot and the usual "nodes shutting down" messages are shown on VGA but not serial, if that helps. Nothing more at any time appears on the serial console, however many minutes I wait, and SSH won't connect, confirming its not just a lack of console echo.

I got back in by disabling the serial ports in firmware/BIOS, but now what?

What's gone on, and what's my fix?

 

[Samuel Tai]

This article on serial console in XigmaNAS (which is another FreeBSD-based NAS) suggests some additional loader tunables:

NAS4Free Serial Console on Supermicro A1SRi

Having Supermicro A1SRi motherboard is awesome - especially when it comes to remote management over IPMI. However, as I have NAS4Free on it, one thing annoyed me all the time - why I don't see anything in IPMI Text Console tab? It's not that I really need it as iKVM works for me, but I…

www.medo64.com

Code:

boot_multicons=YES
console="vidconsole comconsole"

You may also have to see if the BIOS has taken over COM1 or COM2 for console redirection.

 

[Stilez]

This article on serial console in XigmaNAS (which is another FreeBSD-based NAS) suggests some additional loader tunables:

NAS4Free Serial Console on Supermicro A1SRi

Having Supermicro A1SRi motherboard is awesome - especially when it comes to remote management over IPMI. However, as I have NAS4Free on it, one thing annoyed me all the time - why I don't see anything in IPMI Text Console tab? It's not that I really need it as iKVM works for me, but I…

www.medo64.com

Code:

boot_multicons=YES
console="vidconsole comconsole"

I've added those 2, and now get the preboot firmware and other text (oroms etc) without boot freezing up. But I lose the boot menu (single user/boot environment), and early in the boot the serial console stops and I cant get it to restart.

This is its final serial output:

Code:

READY TO BOOT...
Press DEL to run Setup
Press F11 to invoke Boot Menu
Press F12 to boot from PXE/LAN                                               B2

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0xFFFF,0x0)/HD(1,GPT,EAC56DF9-D012-11EA-ADF0-0007433962F0,0x28,0x82000)
   BootCurrent: 0009
   BootOrder: 0009[*] 0008 0003 0004 000a
   Probing 35 block devices...not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
good
not supported
good
Consoles: EFI console
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool My_pool
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk18p2:
FreeBSD/amd64 EFI loader, Revision 1.1
not supported
   Command line arguments: loader.efi
   EFI version: 2.40
   EFI Firmware: American Megatrends (rev 5.11)
   Console: efi (0x20000000)
   Load Path: HD(2,GPT,EAD0B19C-D012-11EA-ADF0-0007433962F0,0x82028,0x4A08000)
   Load Device: PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0xFFFF,0x0)/HD(2,GPT,EAD0B19C-D012-11EA-ADF0-0007433962F0,0x82028,0x4A08000)
   BootCurrent: 0009
   BootOrder: 0009[*] 0008 0003 0004 000a
   BootInfo Path: HD(1,GPT,EAC56DF9-D012-11EA-ADF0-0007433962F0,0x28,0x82000)/\EFI\BOOT\BOOTX64.EFI
Ignoring Boot0009: Only one DP found
Trying ZFS pool
Setting currdev to zfs:boot-pool/ROOT/12.0-BETA2.1:
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hintsg pools: My_pool boot-pool
Loading /boot/loader.confns
Loading /boot/loader.conf.local
console vidconsole is invalid!
Available consoles:
    efi
    comconsole
    nullconsole
¦¦  spinconsole

I'm impressed at the message "console vidconsole is invalid!" at the same time all this is being displayed on the VGA console.....?

You may also have to see if the BIOS has taken over COM1 or COM2 for console redirection.

What does this actually mean? Ive never used serial console this way before.

 

[Samuel Tai]

Ah, try console="efi comconsole" instead, since the available consoles don't list vidconsole.

 

[Stilez]

Ah, try console="efi comconsole" instead, since the available consoles don't list vidconsole.

Only slight improvement. It got rid of the "vidcontrol" error and I get the single user boot menu back, but the serial console still hangs unresponsively at the exact same place:

Code:

READY TO BOOT...
Press DEL to run Setup
Press F11 to invoke Boot Menu
Press F12 to boot from PXE/LAN                                               B2

>> FreeBSD EFI boot block
   Loader path: /boot/loader.efi

   Initializing modules: ZFS UFS
   Load Path: \EFI\BOOT\BOOTX64.EFI
   Load Device: PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0xFFFF,0x0)/HD(1,GPT,EAC56DF9-D012-11EA-ADF0-0007433962F0,0x28,0x82000)
   BootCurrent: 0009
   BootOrder: 0009[*] 0008 0003 0004 000a
   Probing 35 block devices...not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
not supported
good
not supported
good
Consoles: EFI console
ZFS: i/o error - all block copies unavailable
ZFS: can't read MOS of pool My_pool
    Reading loader env vars from /efi/freebsd/loader.env
Setting currdev to disk18p2:
FreeBSD/amd64 EFI loader, Revision 1.1
not supported
   Command line arguments: loader.efi
   EFI version: 2.40
   EFI Firmware: American Megatrends (rev 5.11)
   Console: efi (0x20000000)
   Load Path: HD(2,GPT,EAD0B19C-D012-11EA-ADF0-0007433962F0,0x82028,0x4A08000)
   Load Device: PciRoot(0x0)/Pci(0x1F,0x2)/Sata(0x2,0xFFFF,0x0)/HD(2,GPT,EAD0B19C-D012-11EA-ADF0-0007433962F0,0x82028,0x4A08000)
   BootCurrent: 0009
   BootOrder: 0009[*] 0008 0003 0004 000a
   BootInfo Path: HD(1,GPT,EAC56DF9-D012-11EA-ADF0-0007433962F0,0x28,0x82000)/\EFI\BOOT\BOOTX64.EFI
Ignoring Boot0009: Only one DP found
Trying ZFS pool
Setting currdev to zfs:boot-pool/ROOT/12.0-BETA2.1:
Loading /boot/defaults/loader.conf
Loading /boot/defaults/loader.conf
Loading /boot/device.hintsg pools: My_pool boot-pool
Loading /boot/loader.confns
Loading /boot/loader.conf.local
¦

If it helps, that file's contents are:

Code:

 # cat /boot/loader.conf.local
hw.igb.max_interrupt_rate="32768"
fuse_load="YES" # Load fuse KLD (for NTFS)
cc_htcp_load="YES" # Load htcp KLD (for TCP advanced congestion handler for 10G)
hw.cxgbe.largest_rx_cluster="4000" # Required <= 4096 in 11.2, Win=4000, as NIC drivers not handling 9k jumbo properly. See "intro", docs
kern.vt.fb.default_mode="1600x1200" # Video  - part of res (was 1280x1024). Set general default screen res (all screens)
kern.vt.fb.modes.LVDS-1="1600x1200" # Video  - part of res (was 1280x1024). Set built-in screen res
hw.vga.textmode="1" # Video  - part of res
verbose_loading="YES" # verbose loader, per /boot/defaults/loader.conf
boot_verbose="-v" # verbose loader, per /boot/defaults/loader.conf
kern.msgbufsize="4194304" # Increase dmesg buffer to 4MB to fit verbose boot output (DFLT=524288)
vfs.zfs.arc.meta_min="80000000000" # DRAFT v12 - FAIRLY HAPPY: (80 GB)
vfs.zfs.dirty_data_max="24000000000" # DRAFT v12-TO CHECK (24 GB) - only what can write in 20 s, or will buffer+write out hard+stall things
vfs.zfs.dirty_data_max_max="35000000000" # DRAFT v12 - FAIRLY HAPPY: (35 GB) - why limit on startup (dirty_data_max should kick in as well)
vfs.zfs.zio.taskq_batch_pct="80" # DRAFT v12 - FAIRLY HAPPY: - but would like to increase in future
vfs.zfs.l2arc.rebuild_enabled="1" # Persistent L2ARC enabled
vfs.zfs.l2arc.headroom="0" # DFLT=2. Realtes to L2ARC scanning ARC for data about to be evicted, to cache it in advance. >0 sets a limit how far to look. 0 allows scanning of entire ARC
boot_multicons="YES" # console - enable serial multiconsole
console="efi,comconsole" # Console - active consoles (efi,vidconsole,comconsole). Some userland messages only show on the FIRST of these (e.g. application, rc.d, and script)
kernel="kernel"
kern.cam.ctl.ha_id=0
loader_logo="TrueNASCore"

 

[Samuel Tai]

OK, you're running the TrueNAS beta. I'll move this thread over there.

 

[Samuel Tai]

Dunno if this will work, but try adding another loader tunable:

kern.console=ttyu0,ttyv0,/uart,ucom,ttyv0,

Replace ttyu0 with ttyu1 if you're using COM2.