Updated to 13.x core, removed old boot envs, stuck at mountroot

bkw777

Cadet
Joined
Aug 7, 2022
Messages
7
I have a 4-bay iXsystems FreeNAS mini from 2017 running 13.x core.
I've been running and updating this box since 2017 with no problems. As far as I can tell all drives and hardware are fine.

Recently I updated to 13.something core (I can't see the exact version since I can't boot)
I had updated and rebooted with no problem at least a couple times without a problem, so the update to 13.0 or 13.1 or whatever it was seemed to have gone fine.

The last things I was doing before it failed to reboot was I had tried to install pkg and ports on the host and had created a new base jail to run unifi controller, which was all actually working fine. I had that running for a couple days before the problem.

The problem happened when I noticed that the host console was outputting a constant repeating error message about a killed syslog-ng process. I assume that was caused either by the 13.x upgrade or by my messing with pkg and ports. I didn't intentionally touch anything related to syslog but who knows maybe just running pkg upgrade or installing some other pkgdid something.

Anyway, the last thing I did before failing to boot was I noticed the syslog-ng process on the host console log window in the gui, removed all old boot environments (12.x and lower) from the gui, and then did an intentional/graceful reboot using the reboot button in the gui, and that reboot halted at mountroot> and I've never been able to get past that since then. It looks like it's successfully loading the bootloader and even the kernel, and only failing at mounting to root fs.

What do I do now?
Here is a capture from the serial console:

Code:
F  ______               ____   _____ _____ 
- |  ____|             |  _ \ / ____|  __ \
  | |___ _ __ ___  ___ | |_) | (___ | |  | |
  |  ___| '__/ _ \/ _ \|  _ < \___ \| |  | |
  | |   | | |  __/  __/| |_) |____) | |__| |
  | |   | | |    |    ||     |      |      |
  |_|   |_|  \___|\___||____/|_____/|_____/      ```                        `
 +-------------------------- Welcome to FreeBSD ----------------------------+/
 |                                              +o   .--`         /y:`      |.
 |  1. Boot FreeBSD [Enter]                      yo`:.            :o      `+|
 |  2. Boot FreeBSD (Serial Console)              y/               -/`   -o/|
 |  3. Escape to loader prompt                   .-                  ::/sy+:|
 |  4. Reboot                                    /                     `--  |
 |                                              `:                          |`
 |  Options:                                    `:                          |`
 |  5. Kernel: default/kernel (1 of 2)           /                          |
 |  6. Boot Options                              .-                        -|
 |                                                --                      -.|
 |                                                 `:`                  `:` |
 |                                                   .--             `--.   |
 |                                                      .---.....----.      |
 +--------------------------------------------------------------------------+
                                                                                

Loading kernel...
/boot/kernel/kernel text=0x1830d8 text=0xe03b10 text=0x8270fc data=0x140 data=0x25d3bc+0x5a1c44 syms=[0x8+0x1b3090+0x8+0x20d623]
Loading configured modules...
/boot/entropy size=0x1000
can't find '/etc/hostid'
KDB: debugger backends: ddb
KDB: current backend: ddb
---<<BOOT>>---
Copyright (c) 1992-2021 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
    The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 13.1-RELEASE n245376-eba770b30ff TRUENAS amd64
FreeBSD clang version 13.0.0 (git@github.com:llvm/llvm-project.git llvmorg-13.0.0-0-gd7b669b3a303)
VT(vga): resolution 640x480
CPU: Intel(R) Atom(TM) CPU  C2750  @ 2.40GHz (2400.10-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x406d8  Family=0x6  Model=0x4d  Stepping=8
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x43d8e3bf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,SSE4.1,SSE4.2,MOVBE,POPCNT,TSCDLT,AESNI,RDRAND>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x101<LAHF,Prefetch>
  Structured Extended Features=0x2282<TSCADJ,SMEP,ERMS,NFPUSG>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16596303872 (15827 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  TIANO   >
WARNING: L1 data cache covers fewer APIC IDs than a core (0 < 1)
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 8 core(s)
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23
Launching APs: 3 1 2 5 7 4 6
random: entropy device external interface
kbd1 at kbdmux0
vtvga0: <VT VGA driver>
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS>
padlock0: No ACE support.
acpi0: <ALASKA A M I >
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> mem 0xdf740000-0xdf75ffff irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
pcib2: <ACPI PCI-PCI bridge> mem 0xdf720000-0xdf73ffff irq 20 at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> mem 0xdf500000-0xdf51ffff irq 22 at device 0.0 on pci2
pci3: <ACPI PCI bus> on pcib3
pcib4: <PCI-PCI bridge> irq 23 at device 1.0 on pci3
pci4: <PCI bus> on pcib4
ahci0: <Marvell 88SE9172 AHCI SATA controller> port 0xc040-0xc047,0xc030-0xc033,0xc020-0xc027,0xc010-0xc013,0xc000-0xc00f mem 0xdf410000-0xdf4101ff irq 23 at device 0.0 on pci4
ahci0: AHCI v1.00 with 2 6Gbps ports, Port Multiplier supported with FBS
ahci0: quirks=0x1000000<IOMMU_BUSWIDE>
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
pcib5: <PCI-PCI bridge> irq 23 at device 5.0 on pci3
pci5: <PCI bus> on pcib5
pcib6: <PCI-PCI bridge> irq 23 at device 0.0 on pci5
pci6: <PCI bus> on pcib6
vgapci0: <VGA-compatible display> port 0xb000-0xb07f mem 0xde000000-0xdeffffff,0xdf000000-0xdf01ffff irq 23 at device 0.0 on pci6
vgapci0: Boot video device
pcib7: <PCI-PCI bridge> irq 21 at device 7.0 on pci3
pci7: <PCI bus> on pcib7
igb0: <Intel(R) I210 (Copper)> port 0xa000-0xa01f mem 0xdf300000-0xdf37ffff,0xdf380000-0xdf383fff irq 21 at device 0.0 on pci7
igb0: EEPROM V3.16-0 eTrack 0x800004d6
igb0: Using 1024 TX descriptors and 1024 RX descriptors
igb0: Using 4 RX queues 4 TX queues
igb0: Using MSI-X interrupts with 5 vectors
igb0: Ethernet address: d0:50:99:c2:b5:23
pcib8: <PCI-PCI bridge> irq 23 at device 9.0 on pci3
pci8: <PCI bus> on pcib8
igb1: <Intel(R) I210 (Copper)> port 0x9000-0x901f mem 0xdf200000-0xdf27ffff,0xdf280000-0xdf283fff irq 23 at device 0.0 on pci8
igb1: EEPROM V3.16-0 eTrack 0x800004d6
igb1: Using 1024 TX descriptors and 1024 RX descriptors
igb1: Using 4 RX queues 4 TX queues
igb1: Using MSI-X interrupts with 5 vectors
igb1: Ethernet address: d0:50:99:c2:b5:24
pcib9: <ACPI PCI-PCI bridge> mem 0xdf700000-0xdf71ffff at device 4.0 on pci0
pci9: <ACPI PCI bus> on pcib9
ahci1: <Marvell 88SE9230 AHCI SATA controller> port 0xd050-0xd057,0xd040-0xd043,0xd030-0xd037,0xd020-0xd023,0xd000-0xd01f mem 0xdf610000-0xdf6107ff irq 23 at device 0.0 on pci9
ahci1: AHCI v1.20 with 8 6Gbps ports, Port Multiplier not supported
ahci1: quirks=0x1000900<NOBSYRES,ALTSIG,IOMMU_BUSWIDE>
ahcich2: <AHCI channel> at channel 0 on ahci1
ahcich3: <AHCI channel> at channel 1 on ahci1
ahcich4: <AHCI channel> at channel 2 on ahci1
ahcich5: <AHCI channel> at channel 3 on ahci1
ahcich6: <AHCI channel> at channel 4 on ahci1
ahcich7: <AHCI channel> at channel 5 on ahci1
ahcich8: <AHCI channel> at channel 6 on ahci1
ahcich9: <AHCI channel> at channel 7 on ahci1
pci0: <base peripheral, IOMMU> at device 15.0 (no driver attached)
ehci0: <Intel Avoton USB 2.0 controller> mem 0xdf763000-0xdf7633ff irq 23 at device 22.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
ahci2: <Intel Avoton AHCI SATA controller> port 0xe0d0-0xe0d7,0xe0c0-0xe0c3,0xe0b0-0xe0b7,0xe0a0-0xe0a3,0xe040-0xe05f mem 0xdf762000-0xdf7627ff irq 19 at device 23.0 on pci0
ahci2: AHCI v1.30 with 4 3Gbps ports, Port Multiplier not supported
ahcich10: <AHCI channel> at channel 0 on ahci2
ahcich11: <AHCI channel> at channel 1 on ahci2
ahcich12: <AHCI channel> at channel 2 on ahci2
ahcich13: <AHCI channel> at channel 3 on ahci2
ahci3: <Intel Avoton AHCI SATA controller> port 0xe090-0xe097,0xe080-0xe083,0xe070-0xe077,0xe060-0xe063,0xe020-0xe03f mem 0xdf761000-0xdf7617ff irq 19 at device 24.0 on pci0
ahci3: AHCI v1.30 with 2 6Gbps ports, Port Multiplier not supported
ahcich14: <AHCI channel> at channel 0 on ahci3
ahcich15: <AHCI channel> at channel 1 on ahci3
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
apei0: <ACPI Platform Error Interface> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel Avoton/Rangeley SoC watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xcb000-0xcbfff pnpid ORM0000 on isa0
superio0: <Nuvoton NCT6776> at port 0x2e-0x2f on isa0
wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at WDT ldn 0x08 on superio0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounter "TSC-low" frequency 1199999726 Hz quality 1000
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
Trying to mount root from zfs:freenas-boot/ROOT/13.0-U1.1 []...
ugen0.1: <Intel EHCI root HUB> at usbus0
Mounting from zfs:freenas-boot/ROOT/13.0-U1.1 failed with error 2: unknown file system.
uhub0 on usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ada0 at ahcich3 bus 0 scbus3 target 0 lun 0
ada0: <16GB SATA Flash Drive SFDK002A> ACS-2 ATA SATA 3.x device
ada0: Serial Number E0117209800000000086
ada0: 600.000MB/s transfers (SATA 3.x, UDMA5, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 15272MB (31277232 512 byte sectors)
ada1 at ahcich10 bus 0 scbus10 target 0 lun 0
ada1: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device
ada1: Serial Number WD-WCC7K3SESA43
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 3815447MB (7814037168 512 byte sectors)
ada1: quirks=0x1<4K>
ada2 at ahcich11 bus 0 scbus11 target 0 lun 0
ada2: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device
ada2: Serial Number WD-WCC7K1RCRT25
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 3815447MB (7814037168 512 byte sectors)
ada2: quirks=0x1<4K>
ada3 at ahcich12 bus 0 scbus12 target 0 lun 0
ada3: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device
ada3: Serial Number WD-WCC7K6TSYF4Y
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 3815447MB (7814037168 512 byte sectors)
ada3: quirks=0x1<4K>
ada4 at ahcich13 bus 0 scbus13 target 0 lun 0
ada4: <WDC WD40EFRX-68N32N0 82.00A82> ACS-3 ATA SATA 3.x device
ada4: Serial Number WD-WCC7K6AKH5CJ
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 3815447MB (7814037168 512 byte sectors)
ada4: quirks=0x1<4K>
pass1 at ahcich9 bus 0 scbus9 target 0 lun 0
pass1: <Marvell Console 1.01> Removable Processor SCSI device
pass1: Serial Number HKDP221516WL
pass1: 150.000MB/s transfers (SATA 1.x, UDMA4, ATAPI 12bytes, PIO 8192bytes)
Root mount waiting for: usbus0
Root mount waiting for: usbus0
uhub0: 8 ports with 8 removable, self powered
Root mount waiting for: usbus0
ugen0.2: <vendor 0x8087 product 0x07db> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x8087 product 0x07db, class 9/0, rev 2.00/0.02, addr 2> on usbus0
uhub1: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.3: <vendor 0x05e3 USB2.0 Hub> at usbus0
uhub2 on uhub1
uhub2: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 3> on usbus0
uhub2: MTT enabled
uhub2: 4 ports with 4 removable, self powered
Root mount waiting for: usbus0
ugen0.4: <vendor 0x04d9 RPI Wired Keyboard 4> at usbus0
ukbd0 on uhub2
ukbd0: <vendor 0x04d9 RPI Wired Keyboard 4, class 0/0, rev 2.00/1.10, addr 4> on usbus0
kbd2 at ukbd0
uhid0 on uhub2
uhid0: <vendor 0x04d9 RPI Wired Keyboard 4, class 0/0, rev 2.00/1.10, addr 4> on usbus0
ugen0.5: <American Megatrends Inc. Virtual Hub> at usbus0
uhub3 on uhub1
uhub3: <7-port Hub> on usbus0
Root mount waiting for: usbus0
uhub3: 5 ports with 5 removable, self powered
ugen0.6: <American Megatrends Inc. Virtual Keyboard and Mouse> at usbus0
ukbd1 on uhub3
ukbd1: <Keyboard Interface> on usbus0
kbd3 at ukbd1
ums0 on uhub3
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0

Loader variables:
  vfs.root.mountfrom=zfs:freenas-boot/ROOT/13.0-U1.1

Manual root filesystem specification:
  <fstype>:<device> [options]
      Mount <device> using filesystem <fstype>
      and with the specified (optional) option list.

    eg. ufs:/dev/da0s1a
        zfs:zroot/ROOT/default
        cd9660:/dev/cd0 ro
          (which is equivalent to: mount -t cd9660 -o ro /dev/cd0 /)

  ?               List valid disk boot devices
  .               Yield 1 second (for background tasks)
  <empty line>    Abort manual input

mountroot> ?

List of GEOM managed disk devices:
  diskid/DISK-WD-WCC7K6AKH5CJp2 diskid/DISK-WD-WCC7K6AKH5CJp1 gptid/f56a0216-71e1-11e7-bcde-d05099c2b523 gptid/f5561875-71e1-11e7-bcde-d05099c2b523 diskid/DISK-WD-WCC7K6TSYF4Yp2 diskid/DISK-WD-WCC7K6TSYF4Yp1 gptid/f44af09c-71e1-11e7-bcde-d05099c2b523 gptid/f4376e2c-71e1-11e7-bcde-d05099c2b523 diskid/DISK-WD-WCC7K1RCRT25p2 diskid/DISK-WD-WCC7K1RCRT25p1 gptid/f32ee1d8-71e1-11e7-bcde-d05099c2b523 gptid/f31c3673-71e1-11e7-bcde-d05099c2b523 diskid/DISK-WD-WCC7K6AKH5CJ ada4p2 ada4p1 diskid/DISK-WD-WCC7K6TSYF4Y ada3p2 ada3p1 diskid/DISK-WD-WCC7K1RCRT25 ada2p2 ada2p1 diskid/DISK-WD-WCC7K3SESA43p2 diskid/DISK-WD-WCC7K3SESA43p1 gptid/f215e5ce-71e1-11e7-bcde-d05099c2b523 gptid/f20454d6-71e1-11e7-bcde-d05099c2b523 ada4 ada3 ada2 diskid/DISK-WD-WCC7K3SESA43 ada1p2 ada1p1 diskid/DISK-E0117209800000000086p2 diskid/DISK-E0117209800000000086p1 gptid/e9b0c5ff-68a0-11e7-b857-d05099c2b523 gptid/e9ae3b5e-68a0-11e7-b857-d05099c2b523 ada1 diskid/DISK-E0117209800000000086 ada0p2 ada0p1 ada0

mountroot>


There is no usb devices plugged in. Until just now I'd never opened the case even. There's no usb boot drive inside. It's a small pcb plugged into an sata plug on the mother board, which appears to be a 16G ssd. The hardware, other than the main drives, is all right from iXsystems new and until just now I'd never even broken the warranty sticker. Inside everything looks fine, not even dusty, fan works etc. It's been powered from a apc smart-ups 2u rackmount the whole time and there was no storm or power event recently. I believe the problem must be the result of deleting the old boot environments. Though I don't know why since I'm definitely running 13.x and I definitely only removed some 12.x and 11.x environments. The main drives are 4 x 4T wd red 3.5" NASware (well you can see those in the console log so nevermind...)

It looks like, the kernel says it's 13.1, but the failing mountroot says it's trying to mount something with 13.0 in the name. No idea how that came about.

What can I do from here?
I actually don't mind reinstalling the host and all jails as long as I can preserve the main zfs pool. I don't have a recent config backup, but I do have some old config backups, from before the 13.0 update. The few jails I have don't have anything irreplaceable in the jails themselves. I can lose the host and all jails but do not want to lose the main zfs data pool.
 

bkw777

Cadet
Joined
Aug 7, 2022
Messages
7
I ended up removing the 16G Apacer sata disk-on-module, ejecting the main drives, installing a normal sandisk ssd, installing 13.0 fresh on that. Rebooted with the main drives re-inserted and imported the pool, and everything is fine.

My jails even still work just like nothing happened. I had to re-create the host configs like adding a couple users and setting their home dir to point where it was in the main pool, configuring some services I had been using like rsync and samba, setting a couple jails to start at boot, etc, but all that was actually trivial.

Now that I see how that went, I know I could put the apacer boot disk back in and install on that again if I feel like. I only took it out out of caution.

I think the original cause of the problem might have been me messing with pkg and port config files in the host environment, particularly editing /usr/local/etc/pkg/*.conf to switch from quarterly to latest main pkg repo and then running pkg update.
 
Top