---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
	The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-PRERELEASE 4912790fb32(HEAD) TRUENAS amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
WARNING: WITNESS option enabled, expect reduced performance.
VT(vga): text 80x25
CPU: Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz (2900.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 137438953472 (131072 MB)
avail memory = 133814722560 (127615 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <DELL   PE_SC3  >
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads
random: unblocking device.
ioapic1: MADT APIC ID 1 != hw id 15
ioapic2: MADT APIC ID 2 != hw id 15
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 32-55 on motherboard
ioapic2 <Version 2.0> irqs 64-87 on motherboard
Launching APs: 14 23 21 12 10 7 3 18 6 1 20 11 22 13 19 2 9 5 4 15 16 17 8
Timecounter "TSC-low" frequency 1450033843 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
nexus0
vtvga0: <VT VGA driver> on motherboard
aesni0: <AES-CBC,AES-CCM,AES-GCM,AES-ICM,AES-XTS> on motherboard
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <DELL PE_SC3> on motherboard
acpi0: Power Button (fixed)
apei0: <ACPI Platform Error Interface> on acpi0
ipmi0: <IPMI System Interface> port 0xca8,0xcac irq 10 on acpi0
ipmi0: KCS mode found at io 0xca8 on acpi
cpu0: <ACPI CPU> on acpi0
atrtc0: <AT realtime clock> port 0x70-0x7f irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x5f irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 53 at device 1.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
bge0: <Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000> mem 0xd90a0000-0xd90affff,0xd90b0000-0xd90bffff,0xd90c0000-0xd90cffff irq 34 at device 0.0 numa-domain 0 on pci1
bge0: APE FW version: NCSI v1.5.14.0
bge0: CHIP ID 0x05720000; ASIC REV 0x5720; CHIP REV 0x57200; PCI-E
miibus0: <MII bus> numa-domain 0 on bge0
brgphy0: <BCM5720C 1000BASE-T media interface> PHY 1 on miibus0
brgphy0:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
<6>bge0: Using defaults for TSO: 65518/35/2048
<6>bge0: Ethernet address: c8:1f:66:f8:a9:8d
bge1: <Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000> mem 0xd90d0000-0xd90dffff,0xd90e0000-0xd90effff,0xd90f0000-0xd90fffff irq 36 at device 0.1 numa-domain 0 on pci1
bge1: APE FW version: NCSI v1.5.14.0
bge1: CHIP ID 0x05720000; ASIC REV 0x5720; CHIP REV 0x57200; PCI-E
miibus1: <MII bus> numa-domain 0 on bge1
brgphy1: <BCM5720C 1000BASE-T media interface> PHY 2 on miibus1
brgphy1:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
<6>bge1: Using defaults for TSO: 65518/35/2048
<6>bge1: Ethernet address: c8:1f:66:f8:a9:8e
pcib2: <ACPI PCI-PCI bridge> irq 53 at device 1.1 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
bge2: <Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000> mem 0xd91a0000-0xd91affff,0xd91b0000-0xd91bffff,0xd91c0000-0xd91cffff irq 35 at device 0.0 numa-domain 0 on pci2
bge2: APE FW version: NCSI v1.5.14.0
bge2: CHIP ID 0x05720000; ASIC REV 0x5720; CHIP REV 0x57200; PCI-E
miibus2: <MII bus> numa-domain 0 on bge2
brgphy2: <BCM5720C 1000BASE-T media interface> PHY 1 on miibus2
brgphy2:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
<6>bge2: Using defaults for TSO: 65518/35/2048
<6>bge2: Ethernet address: c8:1f:66:f8:a9:8b
bge3: <Broadcom NetXtreme Gigabit Ethernet, ASIC rev. 0x5720000> mem 0xd91d0000-0xd91dffff,0xd91e0000-0xd91effff,0xd91f0000-0xd91fffff irq 38 at device 0.1 numa-domain 0 on pci2
bge3: APE FW version: NCSI v1.5.14.0
bge3: CHIP ID 0x05720000; ASIC REV 0x5720; CHIP REV 0x57200; PCI-E
miibus3: <MII bus> numa-domain 0 on bge3
brgphy3: <BCM5720C 1000BASE-T media interface> PHY 2 on miibus3
brgphy3:  10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow
<6>bge3: Using defaults for TSO: 65518/35/2048
<6>bge3: Ethernet address: c8:1f:66:f8:a9:8c
pcib3: <ACPI PCI-PCI bridge> irq 53 at device 2.0 numa-domain 0 on pci0
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
pcib4: <ACPI PCI-PCI bridge> irq 53 at device 2.2 numa-domain 0 on pci0
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
mps0: <Avago Technologies (LSI) SAS2308> port 0xfc00-0xfcff mem 0xddff0000-0xddffffff,0xddf80000-0xddfbffff irq 42 at device 0.0 numa-domain 0 on pci4
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
pcib5: <ACPI PCI-PCI bridge> irq 53 at device 3.0 numa-domain 0 on pci0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
pcib6: <PCI-PCI bridge> irq 16 at device 17.0 numa-domain 0 on pci0
pci6: <PCI bus> numa-domain 0 on pcib6
pci0: <simple comms> at device 22.0 (no driver attached)
pci0: <simple comms> at device 22.1 (no driver attached)
ehci0: <Intel Patsburg USB 2.0 controller> mem 0xdf8fd000-0xdf8fd3ff irq 23 at device 26.0 numa-domain 0 on pci0
usbus0: EHCI version 1.0
usbus0 numa-domain 0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
pcib7: <ACPI PCI-PCI bridge> at device 28.0 numa-domain 0 on pci0
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
pcib8: <ACPI PCI-PCI bridge> irq 19 at device 28.7 numa-domain 0 on pci0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
pcib9: <PCI-PCI bridge> at device 0.0 numa-domain 0 on pci8
pci9: <PCI bus> numa-domain 0 on pcib9
pcib10: <PCI-PCI bridge> at device 0.0 numa-domain 0 on pci9
pci10: <PCI bus> numa-domain 0 on pcib10
pcib11: <PCI-PCI bridge> at device 0.0 numa-domain 0 on pci10
pci11: <PCI bus> numa-domain 0 on pcib11
vgapci0: <VGA-compatible display> mem 0xd8000000-0xd8ffffff,0xdeffc000-0xdeffffff,0xde000000-0xde7fffff irq 19 at device 0.0 numa-domain 0 on pci11
vgapci0: Boot video device
pcib12: <PCI-PCI bridge> at device 1.0 numa-domain 0 on pci9
pci12: <PCI bus> numa-domain 0 on pcib12
ehci1: <Intel Patsburg USB 2.0 controller> mem 0xdf8fe000-0xdf8fe3ff irq 22 at device 29.0 numa-domain 0 on pci0
usbus1: EHCI version 1.0
usbus1 numa-domain 0 on ehci1
usbus1: 480Mbps High Speed USB v2.0
pcib13: <PCI-PCI bridge> at device 30.0 numa-domain 0 on pci0
pci13: <PCI bus> numa-domain 0 on pcib13
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
pcib14: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pci14: <ACPI PCI bus> numa-domain 1 on pcib14
pcib15: <ACPI PCI-PCI bridge> irq 85 at device 1.0 numa-domain 1 on pci14
pci15: <ACPI PCI bus> numa-domain 1 on pcib15
pcib16: <ACPI PCI-PCI bridge> irq 85 at device 2.0 numa-domain 1 on pci14
pci16: <ACPI PCI bus> numa-domain 1 on pcib16
ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xdcc0-0xdcdf mem 0xd4e80000-0xd4efffff,0xd4ff8000-0xd4ffbfff irq 72 at device 0.0 numa-domain 1 on pci16
ix0: Using 2048 TX descriptors and 2048 RX descriptors
ix0: queue equality override not set, capping rx_queues at 6 and tx_queues at 6
ix0: Using 6 RX queues 6 TX queues
ix0: Using MSI-X interrupts with 7 vectors
ix0: allocated for 6 queues
ix0: allocated for 6 rx queues
<6>ix0: Ethernet address: 00:1b:21:bd:d2:84
ix0: PCI Express Bus: Speed 5.0GT/s Width x8
ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver> port 0xdce0-0xdcff mem 0xd4f00000-0xd4f7ffff,0xd4ffc000-0xd4ffffff irq 76 at device 0.1 numa-domain 1 on pci16
ix1: Using 2048 TX descriptors and 2048 RX descriptors
ix1: queue equality override not set, capping rx_queues at 6 and tx_queues at 6
ix1: Using 6 RX queues 6 TX queues
ix1: Using MSI-X interrupts with 7 vectors
ix1: allocated for 6 queues
ix1: allocated for 6 rx queues
<6>ix1: Ethernet address: 00:1b:21:bd:d2:86
ix1: PCI Express Bus: Speed 5.0GT/s Width x8
pcib17: <ACPI PCI-PCI bridge> irq 85 at device 3.0 numa-domain 1 on pci14
pci17: <ACPI PCI bus> numa-domain 1 on pcib17
pcib18: <ACPI PCI-PCI bridge> irq 85 at device 3.2 numa-domain 1 on pci14
pci18: <ACPI PCI bus> numa-domain 1 on pcib18
pcib19: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci19: <ACPI PCI bus> numa-domain 0 on pcib19
pci19: <dasp, performance counters> at device 14.1 (no driver attached)
pci19: <dasp, performance counters> at device 19.1 (no driver attached)
pci19: <dasp, performance counters> at device 19.4 (no driver attached)
pci19: <dasp, performance counters> at device 19.5 (no driver attached)
pcib20: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pci20: <ACPI PCI bus> numa-domain 1 on pcib20
pci20: <dasp, performance counters> at device 14.1 (no driver attached)
pci20: <dasp, performance counters> at device 19.1 (no driver attached)
pci20: <dasp, performance counters> at device 19.4 (no driver attached)
pci20: <dasp, performance counters> at device 19.5 (no driver attached)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ichwd0: <Intel Patsburg watchdog timer> on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xd2000-0xd2fff,0xd3000-0xd3fff,0xec000-0xeffff pnpid ORM0000 on isa0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est0 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est1 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est2 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est3 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est4 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est5 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est6 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est7 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est8 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est9 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est10 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est11 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est12 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est13 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est14 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est15 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est16 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est17 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est18 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est19 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est20 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est21 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est22 attach returned 6
est: CPU supports Enhanced Speedstep, but is not recognized.
est: cpu_vendor GenuineIntel, msr 245900002000
device_attach: est23 attach returned 6
Timecounters tick every 1.000 msec
can't handle raw ops yet!!!
can't handle raw ops yet!!!
can't handle raw ops yet!!!
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ipmi0: IPMI device rev. 1, firmware rev. 2.65, version 2.0, device support mask 0xdf
ipmi0: Number of channels 6
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
ugen1.1: <Intel EHCI root HUB> at usbus1
ugen0.1: <Intel EHCI root HUB> at usbus0
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e8ecd0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e31b at xpttargettraverse+0x6b
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e89cd0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e884d0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witnesda1 at mps0 bus 0 scbus0 target 9 lun 0
da1: <ATA Samsung SSD 860 4B6Q> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number S3YHNX0K848696D     
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 238475MB (488397168 512 byte sectors)
s_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e8c4d0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e8d4d0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
da3 at mps0 bus 0 scbus0 target 12 lun 0
da3: <ATA WDC WD20EFRX-68A 0A80> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number WD-WCC1T0337779
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors)
da3: quirks=0x8<4K>
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e974d0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
da4 at mps0 bus 0 scbus0 target 13 lun 0
da4: <ATA WDC WD20EFRX-68A 0A80> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number WD-WCC1T0363055
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors)
da4: quirks=0x8<4K>
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e92cd0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_da5 at mps0 bus 0 scbus0 target 14 lun 0
da5: <ATA WDC WD20EFRX-68A 0A80> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number WD-WCC1T0366936
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors)
da5: quirks=0x8<4K>
periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e8fcd0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffda6 at mps0 bus 0 scbus0 target 15 lun 0
da6: <ATA WDC WD20EFRX-68A 0A80> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number WD-WMC301613321
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 1907729MB (3907029168 512 byte sectors)
da6: quirks=0x8<4K>
ffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e91cd0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
da7 at mps0 bus 0 scbus0 target 16 lun 0
da7: <ATA WDC WD20EFRX-68A 0A80> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number WD-WMC301627367
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 1907729MB (3907029168 512 byte sectors)
da7: quirks=0x8<4K>
uma_zalloc_arg: zone "kenv" with the following non-sleepable locks held:
exclusive sleep mutex CAM device lock (CAM device lock) r = 0 (0xfffff80102e904d0) locked @ /truenas-releng/freenas/_BE/os/sys/cam/cam_xpt.c:2340
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80ae02dd at witness_warn+0x40d
#2 0xffffffff80e40d28 at uma_zalloc_arg+0x38
#3 0xffffffff80a2713a at getenv_string_buffer+0x3a
#4 0xffffffff80a27867 at getenv_quad+0x17
#5 0xffffffff80a27832 at getenv_int+0x12
#6 0xffffffff803d9867 at daregister+0x1e7
#7 0xffffffff803721ab at cam_periph_alloc+0x57b
#8 0xffffffff803d91c2 at daasync+0x2c2
#9 0xffffffff8037c85a at xptsetasyncfunc+0x11a
#10 0xffffffff8037e595 at xptdevicetraverse+0xa5
#11 0xffffffff8037e37b at xpttargettraverse+0xcb
#12 0xffffffff8037de5f at xptbustraverse+0x6f
#13 0xffffffff8037c6ce at xpt_register_async+0x1fe
#14 0xffffffff803d8ea9 at dainit+0x19
#15 0xffffffff80371c03 at periphdriver_init+0x43
#16 0xffffffff8037c3f2 at xpt_finishconfig_task+0x12
#17 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
da0 at mps0 bus 0 scbus0 target 8 lun 0
da0: <ATA ST2000VN004-2E41 SC60> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number Z527W2B0
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 11 lun 0
da2: <ATA Hitachi HDS72302 A5C0> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number MN1220F32D8GWD
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors)
da8 at mps0 bus 0 scbus0 target 17 lun 0
da8: <ATA Hitachi HDS72302 A5C0> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number MN1220F32E1HXD
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 1907729MB (3907029168 512 byte sectors)
da9 at mps0 bus 0 scbus0 target 18 lun 0
da9: <ATA Hitachi HDS72302 A5C0> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number MN1220F32D5T8D
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 1907729MB (3907029168 512 byte sectors)
Trying to mount root from zfs:boot-pool/ROOT/default []...
can't handle raw ops yet!!!
WARNING: WITNESS option enabled, expect reduced performance.
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
uhub2 numa-domain 0 on uhub1
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087 product 0x0024> at usbus1
uhub3 numa-domain 0 on uhub0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <vendor 0x0424 product 0x2512> at usbus0
uhub4 numa-domain 0 on uhub2
uhub4: <vendor 0x0424 product 0x2512, class 9/0, rev 2.00/b.b3, addr 3> on usbus0
uhub4: MTT enabled
uhub4: 1 port with 1 removable, self powered
ugen0.4: <no manufacturer Gadget USB HUB> at usbus0
uhub5 numa-domain 0 on uhub2
uhub5: <no manufacturer Gadget USB HUB, class 9/0, rev 2.00/0.00, addr 4> on usbus0
uhub5: 6 ports with 6 removable, self powered
ugen0.5: <Avocent Keyboard/Mouse Function> at usbus0
ukbd0 numa-domain 0 on uhub5
ukbd0: <Keyboard> on usbus0
kbd2 at ukbd0
ugen0.6: <SIGMACHIP USB Keyboard> at usbus0
ukbd1 numa-domain 0 on uhub4
ukbd1: <SIGMACHIP USB Keyboard, class 0/0, rev 1.10/1.10, addr 6> on usbus0
kbd3 at ukbd1
lock order reversal:
 1st 0xfffff80102761228 db->db_rwlock (db->db_rwlock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:1256
 2nd 0xfffff81044a38928 dn->dn_mtx (dn->dn_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dnode.c:2299
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82dbc5a8 at dnode_block_freed+0xb8
#3 0xffffffff82d8b48a at dbuf_read_impl+0x3ca
#4 0xffffffff82d81ab2 at dbuf_read_+0x622
#5 0xffffffff82dbcf18 at dnode_hold_impl+0x2e8
#6 0xffffffff82d98e28 at dmu_buf_hold_noread+0x28
#7 0xffffffff82d9902c at dmu_buf_hold+0x1c
#8 0xffffffff82ed48de at zap_lockdir+0x2e
#9 0xffffffff82ed607c at zap_lookup_norm+0x3c
#10 0xffffffff82ed6031 at zap_lookup+0x11
#11 0xffffffff82e2e9ba at spa_ld_trusted_config+0x4a
#12 0xffffffff82e2d3b3 at spa_ld_mos_with_trusted_config+0x33
#13 0xffffffff82e2b57c at spa_load_impl+0x7c
#14 0xffffffff82e23bc6 at spa_load+0x56
#15 0xffffffff82e23325 at spa_load_best+0x65
#16 0xffffffff82e1f541 at spa_open_common+0x131
#17 0xffffffff82dea021 at dsl_pool_hold+0x21
lock order reversal:
 1st 0xfffff81044243098 db->db_mtx (db->db_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:2819
 2nd 0xfffff80100bf8928 dn->dn_mtx (dn->dn_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:2847
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d8d1df at dbuf_dirty_compute_state+0x11f
#3 0xffffffff82d8359f at dbuf_dirty+0xdf
#4 0xffffffff82dc91d6 at dsl_dataset_sync+0x56
#5 0xffffffff82de86fa at dsl_pool_sync+0xca
#6 0xffffffff82e28eb4 at spa_sync+0x1014
#7 0xffffffff82e3f914 at txg_sync_thread+0x484
#8 0xffffffff80a360a0 at fork_exit+0x80
#9 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
lock order reversal:
 1st 0xfffff81044a6a000 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfslock order reversal:
lock order reversal:
 1st 0xfffff81044a6a480 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs 1st 0xfffff81044a6a5c0 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/multilist.c:288
 2nd 0xfffff8106a431810 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dnode.c:275
stack backtrace:
/multilist.c:288
 2nd 0xfffff81044243250 db->db_mtx (db->db_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:5504
stack backtrace:
/multilist.c:288
 2nd 0xfffff80100bf8928 dn->dn_mtx (dn->dn_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1513
stack backtrace:
 1st 0xfffff81044a6a500 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/multilist.c:288
 2nd 0xfffff80103e3d490 buf->b_evict_lock (buf->b_evict_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/arc.c:6654
stack backtrace:
#0 0xffffffff80adf361 at witnes#0 0xffffffff80adf361 at witnes#0 0xffffffff80adf361 at witnes#0 0xffffffff80adf361 at witness_debugger+0x71
s_debugger+0x71
#1 0xffffffff80#1 0xffffffff80s_debugger+0x71
#1 0xffffffff80s_debugger+0x71
#1 0xffffffff80a80687 at _sx_slock_int+0x67
a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82dba358 at dnode_ve#2 0xffffffff82d8a26c at dbuf_sync_lea7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d70c4a at arc_releaseda7f5c7 at _sx_xlock+0x67
#2 0xffffffff82da1f5d at sync_dnodes_rify+0x98
#3 0xffffffff82dc0c1eaf+0x16c
#3 0xffffffff82d89ccb +0x2a
#3 0xffffffff82dc0c34 at task+0x12d
#3 0xffffffff82d2036 at dnode_sync+0x7e
#4 0xffffffff82da1e8f at sync_dnodes_task+at dbuf_sync_list+0xbb
#4 0xffffffff82dc1ab7 at dnode_sync+0xfdnode_sync+0x94
#4 0xffffffff82da1e8f at sync_dnodes_task+0x5ff at taskq_run+0x1f
#4 0xffffffff80ad1e88 at taskqueue_run_loc0x5f
#5 0xffffffff82d2036f at t17
#5 0xffffffff82da1e8f at syn
#5 0xffffffff82d2036f at taskqked+0x168
#5 0xffffffff80ad2e04c_dnodes_task+0x5f
askq_run+0x1f
#6 0xfffffff#6 0xffffffff80ad1e88 at taskqueue_run_locked+0x_run+0x1f
#6 0xffffffff80ad1e88 at taskqueue_thread_loop+0x94
#6 0xffffffff80a360a0 at fork_ef82d2036f at taskq_run+0x1f
#7 0xffffffff80ad1e88 at taskqueue168
#7 0xffffffff80ad2e04 at ta at taskqueue_run_locked+0x168
#7 0xffffffff80ad2e04 at taskquxit+0x80
#7 0xffffffff80fc09fe at fork_trampoline+0xe
skqueue_thread_loop+0x94
#8 0xf_run_locked+0x168
fffffff80a360a0 at fork_exit+0x#8 0xffffffff80ad2e04 at taskqueue_thread_loeue_thread_loop+0x94
#8 0xfffffop+0x94
#9 0xffffffff80a360a0 a80
#9 0xffffffff80fc09fe at fork_trampoline+0xe
fff80a360a0 at fork_exit+0x80
#9 0xffffffff80fc09fe at fork_trampoline+0xe
t fork_exit+0x80
#10 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff81044a6a340 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/multilist.c:288
 2nd 0xfffff801026e6800 ms->ms_lock (ms->ms_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/metaslab.c:5991
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82e07b37 at metaslab_check_free_impl+0x107
#3 0xffffffff82e09c5e at metaslab_check_free+0x5e
#4 0xffffffff82effde7 at zio_free+0x47
#5 0xffffffff82dc4565 at dsl_dataset_block_kill+0x725
#6 0xffffffff82dc2122 at free_blocks+0xd2
#7 0xffffffff82dc24f4 at dnode_sync_free_range+0xf4
#8 0xffffffff82e14272 at range_tree_walk+0x62
#9 0xffffffff82dc117a at dnode_sync+0x5da
#10 0xffffffff82da1e8f at sync_dnodes_task+0x5f
#11 0xffffffff82d2036f at taskq_run+0x1f
#12 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#13 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#14 0xffffffff80a360a0 at fork_exit+0x80
#15 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff81044a6a340 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/multilist.c:288
 2nd 0xffffffff830a42b8 buf_hash_table.ht_locks[i].ht_lock (buf_hash_table.ht_locks[i].ht_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/arc.c:1005
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d6bfda at buf_hash_find+0xba
#3 0xffffffff82d6ec7f at arc_freed+0x2f
#4 0xffffffff82efff0c at zio_free_sync+0x6c
#5 0xffffffff82effe3b at zio_free+0x9b
#6 0xffffffff82dc4565 at dsl_dataset_block_kill+0x725
#7 0xffffffff82dc2122 at free_blocks+0xd2
#8 0xffffffff82dc24f4 at dnode_sync_free_range+0xf4
#9 0xffffffff82e14272 at range_tree_walk+0x62
#10 0xffffffff82dc117a at dnode_sync+0x5da
#11 0xffffffff82da1e8f at sync_dnodes_task+0x5f
#12 0xffffffff82d2036f at taskq_run+0x1f
#13 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#14 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#15 0xffffffff80a360a0 at fork_exit+0x80
#16 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff81044a6a340 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/multilist.c:288
 2nd 0xfffff8106a835720 dn->dn_dbufs_mtx (dn->dn_dbufs_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dnode_sync.c:463
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82dc0971 at dnode_evict_dbufs+0x41
#3 0xffffffff82dc1270 at dnode_sync+0x6d0
#4 0xffffffff82da1e8f at sync_dnodes_task+0x5f
#5 0xffffffff82d2036f at taskq_run+0x1f
#6 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#7 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#8 0xffffffff80a360a0 at fork_exit+0x80
#9 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff8105e336ba8 zfs (zfs) @ /truenas-releng/freenas/_BE/os/sys/kern/vfs_lookup.c:693
 2nd 0xfffffe0115fc7c10 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/os/freebsd/zfs/zfs_znode.c:947
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d4bac6 at zfs_zget+0x56
#3 0xffffffff82d321b9 at zfs_dirent_lookup+0x169
#4 0xffffffff82d3231b at zfs_dirlook+0xdb
#5 0xffffffff82d48ac0 at zfs_lookup+0x440
#6 0xffffffff82d41362 at zfs_freebsd_cachedlookup+0x72
#7 0xffffffff811c902e at VOP_CACHEDLOOKUP_APV+0xce
#8 0xffffffff80b34dac at vfs_cache_lookup+0xac
#9 0xffffffff811c8e2e at VOP_LOOKUP_APV+0xce
#10 0xffffffff80b3e181 at lookup+0x491
#11 0xffffffff80b3d849 at namei+0x459
#12 0xffffffff80b44f6a at vfs_mountroot+0x10ea
#13 0xffffffff80a0d817 at start_init+0x27
#14 0xffffffff80a360a0 at fork_exit+0x80
#15 0xffffffff80fc09fe at fork_trampoline+0xe
<118>Starting devd.
<118>Autoloading module: uhid.ko
<118>Autoloading module: ums.ko
uhid0 numa-domain 0 on uhub4
uhid0: <SIGMACHIP USB Keyboard, class 0/0, rev 1.10/1.10, addr 6> on usbus0
ums0 numa-domain 0 on uhub5
ums0: <Mouse> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
ums1 numa-domain 0 on uhub5
ums1: <Mouse REL> on usbus0
ums1: 3 buttons and [XYZ] coordinates ID=0
<118>Autoloading module: uhid.ko
<118>Autoloading module: ums.ko
<118>Autoloading module: uhid.ko
<118>Autoloading module: wmt.ko
<118>Starting ums0 moused.
<118>Starting ums1 moused.
<118>Starting zfsd.
<6>lo0: link state changed to UP
<118><118>middlewared: starting                                       middlewared: loaded plugin middlewared.plugins.dyndns       middlewared: loaded plugin middlewared.plugins.s3           middlewared: loaded plugin middlewared.plugins.keychain     middlewared: loaded plugin middlewared.plugins.trueview     middlewared: loaded plugin middlewared.plugins.nfs          middlewared: loaded plugin middlewared.plugins.ftp          middlewared: loaded plugin middlewared.plugins.replication  middlewared: loaded plugin middlewared.plugins.keyvalue     middlewared: loaded plugin middlewared.plugins.system       middlewared: loaded plugin middlewared.plugins.init_shutdownmiddlewared: loaded plugin middlewared.plugins.ldap         middlewared: loaded plugin middlewared.plugins.ups          middlewared: loaded plugin middlewared.plugins.cache        middlewared: loaded plugin middlewared.plugins.cron         middlewared: loaded plugin middlewared.plugins.rsync        middlewared: loaded plugin middlewared.plugins.migration    middlewared: loaded plugin middlewared.plugins.kerberos     middlewared: loaded plugin middlewared.plugins.ntp          middlewared: loaded plugin middlewared.plugins.account      middlewared: loaded plugin middlewared.plugins.vpn          middlewared: loaded plugin middlewared.plugins.zfs          middlewared: loaded plugin middlewared.plugins.cloud_sync   middlewared: loaded plugin middlewared.plugins.nis          middlewared: loaded plugin middlewared.plugins.smart        middlewared: loaded plugin middlewared.plugins.crypto       middlewared: loaded plugin middlewared.plugins.stats        middlewared: loaded plugin middlewared.plugins.disk         middlewared: loaded plugin middlewared.plugins.mdns         middlewared: loaded plugin middlewared.plugins.vmware       middlewared: loaded plugin middlewared.plugins.bootenv      middlewared: loaded plugin middlewared.plugins.support      middlewared: loaded plugin middlewared.plugins.ec2          middlewared: loaded plugin middlewared.plugins.activedirectomiddlewared: loaded plugin middlewared.plugins.auth         middlewared: loaded plugin middlewared.plugins.snmp         middlewared: loaded plugin middlewared.plugins.ssh          middlewared: loaded plugin middlewared.plugins.config       middlewared: loaded plugin middlewared.plugins              middlewared: loaded plugin middlewared.plugins.boot         middlewared: loaded plugin middlewared.plugins.filesystem   middlewared: loaded plugin middlewared.plugins.usage        middlewared: loaded plugin middlewared.plugins.snapshot     middlewared: loaded plugin middlewared.plugins.jail_freebsd middlewared: loaded plugin middlewared.plugins.afp          middlewared: loaded plugin middlewared.plugins.sysdataset   middlewared: loaded plugin middlewared.plugins.lldp         middlewared: loaded plugin middlewared.plugins.multipath    middlewared: loaded plugin middlewared.plugins.ipmi         middlewared: loaded plugin middlewared.plugins.directoryservmiddlewared: loaded plugin middlewared.plugins.iscsi        middlewared: loaded plugin middlewared.plugins.network      middlewared: loaded plugin middlewared.plugins.device       middlewared: loaded plugin middlewared.plugins.zettarepl    middlewared: loaded plugin middlewared.plugins.update       middlewared: loaded plugin middlewared.plugins.tunables     middlewared: loaded plugin middlewared.plugins.etc          middlewared: loaded plugin middlewared.plugins.unscheduled_rmiddlewared: loaded plugin middlewared.plugins.pool         middlewared: loaded plugin middlewared.plugins.smb          middlewared: loaded plugin middlewared.plugins.alert        middlewared: loaded plugin middlewared.plugins.tftp         middlewared: loaded plugin middlewared.plugins.webdav       middlewared: loaded plugin middlewared.plugins.idmap        middlewared: loaded plugin middlewared.plugins.service      middlewared: loaded plugin middlewared.plugins.pwenc        middlewared: loaded plugin middlewared.plugins.acme_protocolmiddlewared: loaded plugin middlewared.plugins.mail         middlewared: loaded plugin middlewared.plugins.api_key      middlewared: loaded plugin middlewared.plugins.vm           middlewared: loaded plugin middlewared.plugins.webui        middlewared: loaded plugin middlewared.plugins.kmip.sync    middlewared: loaded plugin middlewared.plugins.kmip.update  middlewared: loaded plugin middlewared.plugins.kmip         middlewared: loaded plugin middlewared.plugins.kmip.zfs_keysmiddlewared: loaded plugin middlewared.plugins.kmip.connectimiddlewared: loaded plugin middlewared.plugins.kmip.sed_keysmiddlewared: loaded plugin middlewared.plugins.datastore.conmiddlewared: loaded plugin middlewared.plugins.datastore.wrimiddlewared: loaded plugin middlewared.plugins.datastore    middlewared: loaded plugin middlewared.plugins.datastore.evemiddlewared: loaded plugin middlewared.plugins.datastore.filmiddlewared: loaded plugin middlewared.plugins.datastore.reamiddlewared: loaded plugin middlewared.plugins.datastore.schmiddlewared: loaded plugin middlewared.plugins.datastore.utimiddlewared: loaded plugin middlewared.plugins.service_     middlewared: loaded plugin middlewared.plugins.disk_.swap_mimiddlewared: loaded plugin middlewared.plugins.disk_.smart  middlewared: loaded plugin middlewared.plugins.disk_.swap_remiddlewared: loaded plugin middlewared.plugins.disk_.dumpdevmiddlewared: loaded plugin middlewared.plugins.disk_.overpromiddlewared: loaded plugin middlewared.plugins.disk_.disk_evmiddlewared: loaded plugin middlewared.plugins.disk_.smart_amiddlewared: loaded plugin middlewared.plugins.disk_.remove_middlewared: loaded plugin middlewared.plugins.disk_.identifmiddlewared: loaded plugin middlewared.plugins.disk_.format middlewared: loaded plugin middlewared.plugins.disk_.swap_comiddlewared: loaded plugin middlewared.plugins.disk_.sync   middlewared: loaded plugin middlewared.plugins.disk_.encryptmiddlewared: loaded plugin middlewared.plugins.disk_.smartctmiddlewared: loaded plugin middlewared.plugins.disk_        middlewared: loaded plugin middlewared.plugins.disk_.hddstanmiddlewared: loaded plugin middlewared.plugins.disk_.wipe   middlewared: loaded plugin middlewared.plugins.disk_.temperamiddlewared: loaded plugin middlewared.plugins.disk_.label_fmiddlewared: loaded plugin middlewared.plugins.disk_.zfs_guimiddlewared: loaded plugin middlewared.plugins.disk_.disk_inmiddlewared: loaded plugin middlewared.plugins.disk_.info   middlewared: loaded plugin middlewared.plugins.reporting.conmiddlewared: loaded plugin middlewared.plugins.reporting.plumiddlewared: loaded plugin middlewared.plugins.reporting.rrdmiddlewared: loaded plugin middlewared.plugins.reporting.evemiddlewared: loaded plugin middlewared.plugins.reporting.plumiddlewared: loaded plugin middlewared.plugins.reporting.updmiddlewared: loaded plugin middlewared.plugins.reporting    middlewared: loaded plugin middlewared.plugins.vm_          middlewared: loaded plugin middlewared.plugins.vm_.events   middlewared: loaded plugin middlewared.plugins.pool_.import_middlewared: loaded plugin middlewared.plugins.pool_.attach_middlewared: loaded plugin middlewared.plugins.pool_.replacemiddlewared: loaded plugin middlewared.plugins.pool_.import_middlewared: loaded plugin middlewared.plugins.pool_        middlewared: loaded plugin middlewared.plugins.pool_.encryptmiddlewared: loaded plugin middlewared.plugins.pool_.format_middlewared: loaded plugin middlewared.plugins.pool_.info   middlewared: loaded plugin middlewared.plugins.pool_.expand middlewared: loaded plugin middlewared.plugins.replication_ middlewared: loaded plugin middlewared.plugins.replication_.middlewared: loaded plugin middlewared.plugins.smb_.registrymiddlewared: loaded plugin middlewared.plugins.smb_.passdb  middlewared: loaded plugin middlewared.plugins.smb_.admin_grmiddlewared: loaded plugin middlewared.plugins.smb_.status  middlewared: loaded plugin middlewared.plugins.smb_.sid     middlewared: loaded plugin middlewared.plugins.smb_         middlewared: loaded plugin middlewared.plugins.smb_.groupmapmiddlewared: loaded plugin middlewared.plugins.smb_.sharesecmiddlewared: loaded plugin middlewared.plugins.smb_.util_sd middlewared: loaded plugin middlewared.plugins.boot_        middlewared: loaded plugin middlewared.plugins.boot_.format middlewared: loaded plugin middlewared.plugins.boot_.boot_lomiddlewared: loaded plugin middlewared.plugins.truecommand  middlewared: loaded plugin middlewared.plugins.truecommand.cmiddlewared: loaded plugin middlewared.plugins.truecommand.umiddlewared: loaded plugin middlewared.plugins.truecommand.emiddlewared: loaded plugin middlewared.plugins.truecommand.pmiddlewared: loaded plugin middlewared.plugins.truecommand.wmiddlewared: loaded plugin middlewared.plugins.zettarepl_.utmiddlewared: loaded plugin middlewared.plugins.zettarepl_.stmiddlewared: loaded plugin middlewared.plugins.zettarepl_   middlewared: loaded plugin middlewared.plugins.cloud_sync_.cmiddlewared: loaded plugin middlewared.plugins.cloud_sync_  middlewared: loaded plugin middlewared.plugins.update_.instamiddlewared: loaded plugin middlewared.plugins.update_.uploamiddlewared: loaded plugin middlewared.plugins.update_.trainmiddlewared: loaded plugin middlewared.plugins.update_      middlewared: loaded plugin middlewared.plugins.update_.pendimiddlewared: loaded plugin middlewared.plugins.update_.utilsmiddlewared: loaded plugin middlewared.plugins.update_.downlmiddlewared: loaded plugin middlewared.plugins.update_.utilsmiddlewared: loaded plugin middlewared.plugins.account_     middlewared: loaded plugin middlewared.plugins.account_.syncmiddlewared: loaded plugin middlewared.plugins.device_.devd_middlewared: loaded plugin middlewared.plugins.device_      middlewared: loaded plugin middlewared.plugins.device_.devicmiddlewared: loaded plugin middlewared.plugins.zfs_.disks_frmiddlewared: loaded plugin middlewared.plugins.zfs_.events  middlewared: loaded plugin middlewared.plugins.zfs_.zfs_evenmiddlewared: loaded plugin middlewared.plugins.zfs_         middlewared: loaded plugin middlewared.plugins.nfs_         middlewared: loaded plugin middlewared.plugins.nfs_.v4_freebmiddlewared: loaded plugin middlewared.plugins.nfs_.sec     middlewared: loaded plugin middlewared.plugins.interface.capmiddlewared: loaded plugin middlewared.plugins.interface.brimiddlewared: loaded plugin middlewared.plugins.interface.vlamiddlewared: loaded plugin middlewared.plugins.interface.netmiddlewared: loaded plugin middlewared.plugins.interface.conmiddlewared: loaded plugin middlewared.plugins.interface.typmiddlewared: loaded plugin middlewared.plugins.interface.dhcmiddlewared: loaded plugin middlewared.plugins.interface    middlewared: loaded plugin middlewared.plugins.interface.lagmiddlewared: loaded plugin middlewared.plugins.interface.lagmiddlewared: loaded plugin middlewared.plugins.interface.conmiddlewared: loaded plugin middlewared.plugins.iscsi_.globalmiddlewared: loaded plugin middlewared.plugins.iscsi_       middlewared: loaded plugin middlewared.plugins.iscsi_.iscsi_middlewared: loaded plugin middlewared_truenas.plugins.fcpormiddlewared: loaded plugin middlewared_truenas.plugins.truenmiddlewared: loaded plugin middlewared_truenas.plugins.enclomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.sensomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.failomiddlewared: loaded plugin middlewared_truenas.plugins.enclomiddlewared: loaded plugin middlewared_truenas.plugins.enclomiddlewared: loaded plugin middlewared_truenas.plugins.enclomiddlewared: resolving plugins schemas                      middlewared: registering services                           middlewared: setting up plugins (datastore) [1/54]          middlewared: setting up plugins (auth) [2/54]               middlewared: setting up plugins (service) [3/54]            middlewared: setting up plugins (pwenc) [4/54]              middlewared: setting up plugins (boot) [5/54]               middlewared: setting up plugins (system) [6/54]             middlewared: setting up plugins (alert) [7/54]              middlewared: setting up plugins (account) [8/54]            middlewared: setting up plugins (trueview) [9/54]           middlewared: setting up plugins (nfs) [10/54]               middlewared: setting up plugins (ftp) [11/54]               middlewared: setting up plugins (replication) [12/54]       middlewared: setting up plugins (ups) [13/54]               middlewared: setting up plugins (cache) [14/54]             middlewared: setting up plugins (rsync) [15/54]             middlewared: setting up plugins (vpn) [16/54]               middlewared: setting up plugins (cloud_sync) [17/54]        middlewared: setting up plugins (crypto) [18/54]            middlewared: setting up plugins (disk) [19/54]              middlewared: setting up plugins (mdns) [20/54]              middlewared: setting up plugins (ec2) [21/54]               middlewared: setting up plugins (activedirectory) [22/54]   middlewared: setting up plugins (filesystem) [23/54]        middlewared: setting up plugins (usage) [24/54]             middlewared: setting up plugins (snapshot) [25/54]          middlewared: setting up plugins (jail_freebsd) [26/54]      middlewared: setting up plugins (afp) [27/54]               middlewared: setting up plugins (sysdataset) [28/54]        middlewared: setting up plugins (ipmi) [29/54]              middlewared: setting up plugins (directoryservices) [30/54] middlewared: setting up plugins (iscsi) [31/54]             middlewared: setting up plugins (network) [32/54]           middlewared: setting up plugins (zettarepl) [33/54]         middlewared: setting up plugins (etc) [34/54]               middlewared: setting up plugins (unscheduled_reboot_alert) [middlewared: setting up plugins (pool) [36/54]              middlewared: setting up plugins (smb) [37/54]               middlewared: setting up plugins (webdav) [38/54]            middlewared: setting up plugins (vm) [39/54]                middlewared: setting up plugins (kmip) [40/54]              middlewared: setting up plugins (disk_) [41/54]             middlewared: setting up plugins (disk_) [42/54]             middlewared: setting up plugins (disk_) [43/54]             middlewared: setting up plugins (disk_) [44/54]             middlewared: setting up plugins (disk_) [45/54]             middlewared: setting up plugins (reporting) [46/54]         middlewared: setting up plugins (truecommand) [47/54]       
warning: KLD '/boot/kernel-debug/vmm.ko' is newer than the linker.hints file
<118><118>middlewared: setting up plugins (device_) [48/54]           middlewared: setting up plugins (zfs_) [49/54]              middlewared: setting up plugins (zfs_) [50/54]              middlewared: setting up plugins (enclosure) [51/54]         middlewared: setting up plugins (failover) [52/54]          middlewared: setting up plugins (failover_) [53/54]         middlewared: setting up plugins (failover_) [54/54]         middlewared: loading completed                              
<118>Loading early kernel modules:
warning: KLD '/boot/kernel-debug/opensolaris.ko' is newer than the linker.hints file
warning: KLD '/boot/kernel-debug/sdt.ko' is newer than the linker.hints file
warning: KLD '/boot/kernel-debug/systrace.ko' is newer than the linker.hints file
warning: KLD '/boot/kernel-debug/systrace_freebsd32.ko' is newer than the linker.hints file
warning: KLD '/boot/kernel-debug/profile.ko' is newer than the linker.hints file
<118>Syncing disks...
lock order reversal:
 1st 0xfffff80270062898 zio->io_lock (zio->io_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/zio.c:3866
 2nd 0xfffffe00fd97b8e0 vq->vq_lock (vq->vq_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/vdev_queue.c:950
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82e5ba25 at vdev_queue_change_io_priority+0xe5
#3 0xffffffff82f035b8 at zio_change_priority+0x68
#4 0xffffffff82f036e1 at zio_change_priority+0x191
#5 0xffffffff82d6a389 at arc_read+0x469
#6 0xffffffff82d8bdf0 at dbuf_read_impl+0xd30
#7 0xffffffff82d81ab2 at dbuf_read_+0x622
#8 0xffffffff82d9812e at dmu_buf_set_init+0x34e
#9 0xffffffff82d97c7d at dmu_issue+0x10d
#10 0xffffffff82d9a889 at dmu_read_uio_dbuf+0x79
#11 0xffffffff82d43ec9 at zfs_freebsd_read+0x6a9
#12 0xffffffff811ca4ce at VOP_READ_APV+0xce
#13 0xffffffff80b5ef05 at vn_read+0x1f5
#14 0xffffffff80b5ec63 at vn_io_fault_doio+0x43
#15 0xffffffff80b5c9ac at vn_io_fault1+0x16c
#16 0xffffffff80b5acad at vn_io_fault+0x19d
#17 0xffffffff80ae36e5 at dofileread+0x95
<118>Alarm clock
<118>Starting file system checks:
<118>Setting hostuuid: 4c4c4544-0058-5010-8059-b6c04f303832.
<118>Setting hostid: 0x007cc5ec.
<118>Mounting local filesystems:.
lock order reversal:
 1st 0xfffff810b7111150 zcw->zcw_lock (zcw->zcw_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/zil.c:2517
 2nd 0xfffff80102540400 zilog->zl_lock (zilog->zl_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/zil.c:565
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82efcca3 at zil_alloc_lwb+0x193
#3 0xffffffff82efd76d at zil_lwb_write_issue+0x3fd
#4 0xffffffff82efaa89 at zil_commit_impl+0x1a89
#5 0xffffffff82d454f3 at zfs_freebsd_fsync+0xd3
#6 0xffffffff811cb084 at VOP_FSYNC_APV+0xd4
#7 0xffffffff80b589be at kern_fsync+0x17e
#8 0xffffffff80fea34e at amd64_syscall+0x2be
#9 0xffffffff80fc02ee at fast_syscall_common+0xf8
<118>Beginning pools import
<118>Importing pool1
<118>vdev.c:129:vdev_dbgmsg(): disk vdev '/dev/gptid/5647ff21-068b-11eb-9505-60a44c622426': best uberblock found for spa $import. txg 165283
<118>spa_misc.c:411:spa_load_note(): spa_load($import, config untrusted): using uberblock with txg=165283
lock order reversal:
 1st 0xfffff8010254cbb8 os->os_userused_lock (os->os_userused_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1825
 2nd 0xfffff81044807000 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dnode.c:275
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a80687 at _sx_slock_int+0x67
#2 0xffffffff82dba358 at dnode_verify+0x98
#3 0xffffffff82dbcd94 at dnode_hold_impl+0x164
#4 0xffffffff82d98e28 at dmu_buf_hold_noread+0x28
#5 0xffffffff82d9902c at dmu_buf_hold+0x1c
#6 0xffffffff82ed48de at zap_lockdir+0x2e
#7 0xffffffff82ed607c at zap_lookup_norm+0x3c
#8 0xffffffff82ed6031 at zap_lookup+0x11
#9 0xffffffff82ed0001 at zap_increment+0x41
#10 0xffffffff82da27f4 at userquota_updates_task+0x524
#11 0xffffffff82d2036f at taskq_run+0x1f
#12 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#13 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#14 0xffffffff80a360a0 at fork_exit+0x80
#15 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff8010254cbb8 os->os_userused_lock (os->os_userused_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1825
 2nd 0xfffff81044807118 dn->dn_mtx (dn->dn_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dnode.c:1600
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82dbe437 at dnode_rele+0x27
#3 0xffffffff82d98eab at dmu_buf_hold_noread+0xab
#4 0xffffffff82d9902c at dmu_buf_hold+0x1c
#5 0xffffffff82ed48de at zap_lockdir+0x2e
#6 0xffffffff82ed607c at zap_lookup_norm+0x3c
#7 0xffffffff82ed6031 at zap_lookup+0x11
#8 0xffffffff82ed0001 at zap_increment+0x41
#9 0xffffffff82da27f4 at userquota_updates_task+0x524
#10 0xffffffff82d2036f at taskq_run+0x1f
#11 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#12 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#13 0xffffffff80a360a0 at fork_exit+0x80
#14 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff8010254cbb8 os->os_userused_lock (os->os_userused_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1825
 2nd 0xfffff81044e4fca0 db->db_mtx (db->db_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:2113
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d81584 at dbuf_read_+0xf4
#3 0xffffffff82d99059 at dmu_buf_hold+0x49
#4 0xffffffff82ed48de at zap_lockdir+0x2e
#5 0xffffffff82ed607c at zap_lookup_norm+0x3c
#6 0xffffffff82ed6031 at zap_lookup+0x11
#7 0xffffffff82ed0001 at zap_increment+0x41
#8 0xffffffff82da27f4 at userquota_updates_task+0x524
#9 0xffffffff82d2036f at taskq_run+0x1f
#10 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#11 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#12 0xffffffff80a360a0 at fork_exit+0x80
#13 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff8010254cbb8 os->os_userused_lock (os->os_userused_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1825
 2nd 0xffffffff830e8a78 buf_hash_table.ht_locks[i].ht_lock (buf_hash_table.ht_locks[i].ht_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/arc.c:1005
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d6bfda at buf_hash_find+0xba
#3 0xffffffff82d6a0d3 at arc_read+0x1b3
#4 0xffffffff82d8bdf0 at dbuf_read_impl+0xd30
#5 0xffffffff82d81ab2 at dbuf_read_+0x622
#6 0xffffffff82d99059 at dmu_buf_hold+0x49
#7 0xffffffff82ed48de at zap_lockdir+0x2e
#8 0xffffffff82ed607c at zap_lookup_norm+0x3c
#9 0xffffffff82ed6031 at zap_lookup+0x11
#10 0xffffffff82ed0001 at zap_increment+0x41
#11 0xffffffff82da27f4 at userquota_updates_task+0x524
#12 0xffffffff82d2036f at taskq_run+0x1f
#13 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#14 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#15 0xffffffff80a360a0 at fork_exit+0x80
#16 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff8010254cbb8 os->os_userused_lock (os->os_userused_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1825
 2nd 0xfffff802d5f874f0 zap->zap_rwlock (zap->zap_rwlock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/zap_micro.c:421
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82ed4af9 at zap_lockdir_impl+0x159
#3 0xffffffff82ed496e at zap_lockdir+0xbe
#4 0xffffffff82ed607c at zap_lookup_norm+0x3c
#5 0xffffffff82ed6031 at zap_lookup+0x11
#6 0xffffffff82ed0001 at zap_increment+0x41
#7 0xffffffff82da27f4 at userquota_updates_task+0x524
#8 0xffffffff82d2036f at taskq_run+0x1f
#9 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#10 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#11 0xffffffff80a360a0 at fork_exit+0x80
#12 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff8010254cbb8 os->os_userused_lock (os->os_userused_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1848
 2nd 0xfffff8104480d318 dn->dn_dbufs_mtx (dn->dn_dbufs_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:4166
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d854a8 at dbuf_destroy+0x278
#3 0xffffffff82ed6101 at zap_lookup_norm+0xc1
#4 0xffffffff82ed6031 at zap_lookup+0x11
#5 0xffffffff82ed0001 at zap_increment+0x41
#6 0xffffffff82da29e4 at userquota_updates_task+0x714
#7 0xffffffff82d2036f at taskq_run+0x1f
#8 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#9 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#10 0xffffffff80a360a0 at fork_exit+0x80
#11 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffff8010254cbb8 os->os_userused_lock (os->os_userused_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu_objset.c:1848
 2nd 0xffffffff83136a40 h->hash_mutexes[i] (h->hash_mutexes[i]) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:508
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d8558b at dbuf_destroy+0x35b
#3 0xffffffff82ed6101 at zap_lookup_norm+0xc1
#4 0xffffffff82ed6031 at zap_lookup+0x11
#5 0xffffffff82ed0001 at zap_increment+0x41
#6 0xffffffff82da29e4 at userquota_updates_task+0x714
#7 0xffffffff82d2036f at taskq_run+0x1f
#8 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#9 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#10 0xffffffff80a360a0 at fork_exit+0x80
#11 0xffffffff80fc09fe at fork_trampoline+0xe
lock order reversal:
 1st 0xfffffe01210959b8 vd->vdev_dtl_lock (vd->vdev_dtl_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/vdev.c:2888
 2nd 0xfffff802d51b0810 dn->dn_struct_rwlock (dn->dn_struct_rwlock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dmu.c:1609
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a80687 at _sx_slock_int+0x67
#2 0xffffffff82d998b9 at dmu_prefetch+0x149
#3 0xffffffff82e3be16 at space_map_iterate+0xd6
#4 0xffffffff82e3cb33 at space_map_load_length+0x83
#5 0xffffffff82e4757f at vdev_dtl_load+0xdf
#6 0xffffffff82e495be at vdev_load+0x1fe
#7 0xffffffff82e49400 at vdev_load+0x40
#8 0xffffffff82e49400 at vdev_load+0x40
#9 0xffffffff82e2c975 at spa_load_impl+0x1475
#10 0xffffffff82e23bc6 at spa_load+0x56
#11 0xffffffff82e2384e at spa_tryimport+0x17e
#12 0xffffffff82eedc7d at zfs_ioc_pool_tryimport+0x3d
#13 0xffffffff82ee60a9 at zfsdev_ioctl_common+0x489
#14 0xffffffff82d272a3 at zfsdev_ioctl+0x143
#15 0xffffffff80908485 at devfs_ioctl+0xb5
#16 0xffffffff811ca944 at VOP_IOCTL_APV+0xd4
#17 0xffffffff80b5b03d at vn_ioctl+0x13d
<118>metaslab.c:3837:metaslab_flush(): flushing: txg 2062, spa boot-pool, vdev_id 0, ms_id 35, unflushed_allocs 0, unflushed_frees 52736, appended 248 bytes
lock order reversal:
 1st 0xfffffe01210959b8 vd->vdev_dtl_lock (vd->vdev_dtl_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/vdev.c:988
 2nd 0xfffff81044e4f408 db->db_mtx (db->db_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:4989
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d88c57 at dmu_buf_rele+0x27
#3 0xffffffff82e3dcff at space_map_close+0x1f
#4 0xffffffff82e43a29 at vdev_free+0x3f9
#5 0xffffffff82e43839 at vdev_free+0x209
#6 0xffffffff82e43839 at vdev_free+0x209
#7 0xffffffff82e2246a at spa_unload+0x3ea
#8 0xffffffff82e23a2b at spa_tryimport+0x35b
#9 0xffffffff82eedc7d at zfs_ioc_pool_tryimport+0x3d
#10 0xffffffff82ee60a9 at zfsdev_ioctl_common+0x489
#11 0xffffffff82d272a3 at zfsdev_ioctl+0x143
#12 0xffffffff80908485 at devfs_ioctl+0xb5
#13 0xffffffff811ca944 at VOP_IOCTL_APV+0xd4
#14 0xffffffff80b5b03d at vn_ioctl+0x13d
#15 0xffffffff80908afe at devfs_ioctl_f+0x1e
#16 0xffffffff80ae4385 at kern_ioctl+0x2d5
#17 0xffffffff80ae4007 at sys_ioctl+0x127
lock order reversal:
 1st 0xfffffe01210959b8 vd->vdev_dtl_lock (vd->vdev_dtl_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/vdev.c:988
 2nd 0xfffff8108d5bd118 dn->dn_mtx (dn->dn_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dnode.c:1600
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82dbe437 at dnode_rele+0x27
#3 0xffffffff82e3dcff at space_map_close+0x1f
#4 0xffffffff82e43a29 at vdev_free+0x3f9
#5 0xffffffff82e43839 at vdev_free+0x209
#6 0xffffffff82e43839 at vdev_free+0x209
#7 0xffffffff82e2246a at spa_unload+0x3ea
#8 0xffffffff82e23a2b at spa_tryimport+0x35b
#9 0xffffffff82eedc7d at zfs_ioc_pool_tryimport+0x3d
#10 0xffffffff82ee60a9 at zfsdev_ioctl_common+0x489
#11 0xffffffff82d272a3 at zfsdev_ioctl+0x143
#12 0xffffffff80908485 at devfs_ioctl+0xb5
#13 0xffffffff811ca944 at VOP_IOCTL_APV+0xd4
#14 0xffffffff80b5b03d at vn_ioctl+0x13d
#15 0xffffffff80908afe at devfs_ioctl_f+0x1e
#16 0xffffffff80ae4385 at kern_ioctl+0x2d5
#17 0xffffffff80ae4007 at sys_ioctl+0x127
<118>spa.c:8186:spa_async_request(): spa=$import async request task=2048
<118>spa_misc.c:411:spa_load_note(): spa_load($import, config trusted): LOADED
<118>spa_misc.c:411:spa_load_note(): spa_load($import, config trusted): UNLOADING
<118>spa.c:5992:spa_import(): spa_import: importing pool1
<118>spa_misc.c:411:spa_load_note(): spa_load(pool1, config trusted): LOADING
<118>vdev.c:129:vdev_dbgmsg(): disk vdev '/dev/gptid/5647ff21-068b-11eb-9505-60a44c622426': best uberblock found for spa pool1. txg 165283
<118>spa_misc.c:411:spa_load_note(): spa_load(pool1, config untrusted): using uberblock with txg=165283
lock order reversal:
 1st 0xfffff8106a42c940 mls->mls_lock (mls->mls_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/multilist.c:288
 2nd 0xfffff810d622b2c8 dr->dt.di.dr_mtx (dr->dt.di.dr_mtx) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/dbuf.c:5436
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d89fd3 at dbuf_sync_indirect+0x2c3
#3 0xffffffff82d89c60 at dbuf_sync_list+0x50
#4 0xffffffff82dc1ab7 at dnode_sync+0xf17
#5 0xffffffff82da1e8f at sync_dnodes_task+0x5f
#6 0xffffffff82d2036f at taskq_run+0x1f
#7 0xffffffff80ad1e88 at taskqueue_run_locked+0x168
#8 0xffffffff80ad2e04 at taskqueue_thread_loop+0x94
#9 0xffffffff80a360a0 at fork_exit+0x80
#10 0xffffffff80fc09fe at fork_trampoline+0xe
<118>spa_misc.c:411:spa_load_note(): spa_load(pool1, config trusted): read 470 log space maps (470 total blocks - blksz = 131072 bytes) in 2762 ms
<118>mmp.c:249:mmp_thread_start(): MMP thread started pool 'pool1' gethrtime 70167640207
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 3, ms_id 6, smp_length 45424, unflushed_allocs 1183744, unflushed_frees 1826816, freed 0, defer 0 + 0, unloaded time 70167 ms, loading_time 2 ms, ms_max_size 17131798528, m
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 0, ms_id 38, smp_length 497040, unflushed_allocs 1802240, unflushed_frees 1646592, freed 0, defer 0 + 0, unloaded time 70170 ms, loading_time 46 ms, ms_max_size 13411880960
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 1, ms_id 19, smp_length 112440, unflushed_allocs 3035136, unflushed_frees 2195456, freed 0, defer 0 + 0, unloaded time 70217 ms, loading_time 43 ms, ms_max_size 8731254784,
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 3, ms_id 28, smp_length 154064, unflushed_allocs 1671168, unflushed_frees 1519616, freed 0, defer 0 + 0, unloaded time 70260 ms, loading_time 35 ms, ms_max_size 17148346368
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 0, ms_id 46, smp_length 337184, unflushed_allocs 1265664, unflushed_frees 1810432, freed 0, defer 0 + 0, unloaded time 70296 ms, loading_time 30 ms, ms_max_size 8832774144,
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 1, ms_id 22, smp_length 125888, unflushed_allocs 2183168, unflushed_frees 2609152, freed 0, defer 0 + 0, unloaded time 70326 ms, loading_time 6 ms, ms_max_size 8615682048,
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 3, ms_id 49, smp_length 500720, unflushed_allocs 1531904, unflushed_frees 1150976, freed 0, defer 0 + 0, unloaded time 70333 ms, loading_time 38 ms, ms_max_size 8598908928,
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 0, ms_id 44, smp_length 417248, unflushed_allocs 45056, unflushed_frees 102400, freed 0, defer 0 + 0, unloaded time 70372 ms, loading_time 32 ms, ms_max_size 8593326080, ma
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165284, spa pool1, vdev_id 1, ms_id 23, smp_length 464648, unflushed_allocs 1507328, unflushed_frees 1589248, freed 0, defer 0 + 0, unloaded time 70404 ms, loading_time 29 ms, ms_max_size 13399523328
<118>metaslab.c:3837:metaslab_flush(): flushing: txg 165284, spa pool1, vdev_id 2, ms_id 17, unflushed_allocs 0, unflushed_frees 327680, appended 232 bytes
<118>spa.c:8186:spa_async_request(): spa=pool1 async request task=1
<118>spa.c:8186:spa_async_request(): spa=pool1 async request task=2048
<118>spa_misc.c:411:spa_load_note(): spa_load(pool1, config trusted): LOADED
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 3, ms_id 50, smp_length 465416, unflushed_allocs 2453504, unflushed_frees 1699840, freed 0, defer 0 + 0, unloaded time 70633 ms, loading_time 48 ms, ms_max_size 8767004672,
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 0, ms_id 47, smp_length 488344, unflushed_allocs 745472, unflushed_frees 1183744, freed 0, defer 0 + 0, unloaded time 70682 ms, loading_time 67 ms, ms_max_size 8592084992,
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 1, ms_id 25, smp_length 232672, unflushed_allocs 1716224, unflushed_frees 1638400, freed 0, defer 0 + 0, unloaded time 70750 ms, loading_time 10 ms, ms_max_size 15985876992
<118>spa_history.c:309:spa_history_log_sync(): txg 165285 open pool version 5000; software version unknown; uts  12.2-PRERELEASE 1201524 amd64
<118>metaslab.c:3837:metaslab_flush(): flushing: txg 165285, spa pool1, vdev_id 2, ms_id 45, unflushed_allocs 0, unflushed_frees 0, appended 0 bytes
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 2, ms_id 6, smp_length 48176, unflushed_allocs 1630208, unflushed_frees 1671168, freed 0, defer 90112 + 0, unloaded time 70765 ms, loading_time 2 ms, ms_max_size 1709893632
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 2, ms_id 14, smp_length 176696, unflushed_allocs 2076672, unflushed_frees 3260416, freed 0, defer 36864 + 0, unloaded time 70767 ms, loading_time 41 ms, ms_max_size 9945686
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 3, ms_id 65, smp_length 19032, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 1 ms, ms_max_size 13338267648, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 3, ms_id 61, smp_length 88, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 20 ms, ms_max_size 17179635712, max size error
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 2, ms_id 30, smp_length 200, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 22 ms, ms_max_size 17179734016, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 3, ms_id 58, smp_length 664, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 23 ms, ms_max_size 17179656192, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 1, ms_id 41, smp_length 88, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 25 ms, ms_max_size 17179754496, max size error
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 3, ms_id 74, smp_length 96728, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 25 ms, ms_max_size 8643403776, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 3, ms_id 66, smp_length 32968, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 26 ms, ms_max_size 8801771520, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 0, ms_id 62, smp_length 31272, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 27 ms, ms_max_size 17179836416, max size er
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 0, ms_id 49, smp_length 1336, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 28 ms, ms_max_size 17179168768, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 0, ms_id 56, smp_length 35352, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 31 ms, ms_max_size 16850128896, max size er
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 0, ms_id 65, smp_length 14616, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 33 ms, ms_max_size 11637420032, max size er
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 0, ms_id 63, smp_length 22224, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 36 ms, ms_max_size 11456593920, max size er
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 0, ms_id 52, smp_length 288, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 47 ms, ms_max_size 17179648000, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 1, ms_id 37, smp_length 304, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 49 ms, ms_max_size 17179582464, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165285, spa pool1, vdev_id 1, ms_id 38, smp_length 232, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 54 ms, ms_max_size 17179144192, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 3, ms_id 67, smp_length 11424, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 59 ms, ms_max_size 12242612224, max size er
<118>metaslab.c:3837:metaslab_flush(): flushing: txg 165286, spa pool1, vdev_id 0, ms_id 8, unflushed_allocs 0, unflushed_frees 0, appended 0 bytes
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 1, ms_id 27, smp_length 36744, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 61 ms, ms_max_size 8639000576, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 22, smp_length 78336, unflushed_allocs 0, unflushed_frees 159744, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 61 ms, ms_max_size 16615800832, max si
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 1, ms_id 26, smp_length 26344, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 64 ms, ms_max_size 16750600192, max size er
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 0, ms_id 55, smp_length 216, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 65 ms, ms_max_size 17179693056, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 3, ms_id 68, smp_length 5216, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 65 ms, ms_max_size 17179561984, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 36, smp_length 48824, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 71 ms, ms_max_size 8620032000, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 3, ms_id 55, smp_length 195008, unflushed_allocs 0, unflushed_frees 466944, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 73 ms, ms_max_size 15797440512, max s
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 1, ms_id 33, smp_length 64440, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 73 ms, ms_max_size 8610332672, max size err
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 19, smp_length 259024, unflushed_allocs 36864, unflushed_frees 139264, freed 0, defer 110592 + 8192, unloaded time 71145 ms, loading_time 81 ms, ms_max_size 862084
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 1, ms_id 29, smp_length 164424, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 85 ms, ms_max_size 17179316224, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 31, smp_length 112, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 89 ms, ms_max_size 17179246592, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 38, smp_length 106432, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71146 ms, loading_time 108 ms, ms_max_size 8750186496, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 0, ms_id 58, smp_length 374520, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 110 ms, ms_max_size 8625106944, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 25, smp_length 174152, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 110 ms, ms_max_size 17179811840, max size
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 3, ms_id 72, smp_length 381296, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 116 ms, ms_max_size 8591790080, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 1, ms_id 44, smp_length 47400, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 117 ms, ms_max_size 8594636800, max size er
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 20, smp_length 479360, unflushed_allocs 1945600, unflushed_frees 2871296, freed 0, defer 0 + 8192, unloaded time 71145 ms, loading_time 119 ms, ms_max_size 1438708
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 1, ms_id 32, smp_length 264688, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 122 ms, ms_max_size 8595103744, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 0, ms_id 50, smp_length 522112, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 127 ms, ms_max_size 8591220736, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 3, ms_id 53, smp_length 433408, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 129 ms, ms_max_size 14035664896, max size
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 32, smp_length 413424, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 132 ms, ms_max_size 8599212032, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 1, ms_id 45, smp_length 443112, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71146 ms, loading_time 142 ms, ms_max_size 8827351040, max size e
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 0, ms_id 67, smp_length 48, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 144 ms, ms_max_size 17179734016, max size erro
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165286, spa pool1, vdev_id 2, ms_id 27, smp_length 293448, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71145 ms, loading_time 151 ms, ms_max_size 8598237184, max size e
<118>spa.c:8186:spa_async_request(): spa=pool1 async request task=32
<118>metaslab.c:3837:metaslab_flush(): flushing: txg 165287, spa pool1, vdev_id 1, ms_id 108, unflushed_allocs 0, unflushed_frees 0, appended 0 bytes
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165287, spa pool1, vdev_id 2, ms_id 39, smp_length 222880, unflushed_allocs 0, unflushed_frees 4096, freed 0, defer 0 + 0, unloaded time 71419 ms, loading_time 37 ms, ms_max_size 8617422848, max size
<118>spa_history.c:309:spa_history_log_sync(): txg 165288 import pool version 5000; software version unknown; uts  12.2-PRERELEASE 1201524 amd64
<118>metaslab.c:3837:metaslab_flush(): flushing: txg 165288, spa pool1, vdev_id 1, ms_id 109, unflushed_allocs 0, unflushed_frees 0, appended 0 bytes
<118>metaslab.c:2421:metaslab_load_impl(): metaslab_load: txg 165288, spa pool1, vdev_id 2, ms_id 41, smp_length 26192, unflushed_allocs 0, unflushed_frees 0, freed 0, defer 0 + 0, unloaded time 71582 ms, loading_time 48 ms, ms_max_size 8592224256, max size err
<118>dsl_scan.c:3688:dsl_scan_sync(): doing scan sync txg 165289; ddt bm=0/0/0/0
<118>dsl_scan.c:2648:dsl_scan_ddt(): scanned 0 ddt entries with class_max = 1; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 48 (pool1/$ORIGIN@$ORIGIN) with min=0 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 45 (pool1/$ORIGIN) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 54 (pool1) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 69 (pool1/.system/syslog-1ae6384c4fbd402fb437ba266f4c6876) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 176 (pool1/.system/samba4) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 260 (pool1/.system) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 271 (pool1/.system/rrd-1ae6384c4fbd402fb437ba266f4c6876) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 279 (pool1/.system/services) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 388 (pool1/.system/webui) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 515 (pool1/.system/cores) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 643 (pool1/.system/configs-1ae6384c4fbd402fb437ba266f4c6876) with min=1 max=165083; suspending=0
<118>dsl_scan.c:2433:dsl_scan_visitds(): scanned dataset 668 (pool1/proxmox2) with min=1 max=165083; suspending=1
<118>dsl_scan.c:3730:dsl_scan_sync(): scan visited 234126 blocks in 6451ms (13 os's, 229520 holes, 0 < mintxg, 0 in ddt, 1746 > maxtxg)
<118>metaslab.c:3837:metaslab_flush(): flushing: txg 165289, spa pool1, vdev_id 1, ms_id 110, unflushed_allocs 0, unflushed_frees 0, appended 0 bytes
<118>dsl_scan.c:3696:dsl_scan_sync(): doing scan sync txg 165290; bm=668/8/0/0
lock order reversal:
 1st 0xfffff810b7eb47e8 tmpfs (tmpfs) @ /truenas-releng/freenas/_BE/os/sys/kern/vfs_mount.c:930
 2nd 0xfffff810b7eb4608 zfs (zfs) @ /truenas-releng/freenas/_BE/os/sys/kern/vfs_mount.c:941
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a4aa3d at lockmgr_lock_fast_path+0x18d
#2 0xffffffff811cc5d8 at VOP_LOCK1_APV+0xb8
#3 0xffffffff80b5c2d5 at _vn_lock+0x65
#4 0xffffffff80b41965 at vfs_domount+0xda5
#5 0xffffffff80b405f7 at vfs_donmount+0x7d7
#6 0xffffffff80b3fdf1 at sys_nmount+0x71
#7 0xffffffff80fea34e at amd64_syscall+0x2be
#8 0xffffffff80fc02ee at fast_syscall_common+0xf8
lock order reversal:
 1st 0xfffff801025404b8 zilog->zl_issuer_lock (zilog->zl_issuer_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/zfs/zil.c:2453
 2nd 0xfffffe0115fc7d30 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/os/freebsd/zfs/zfs_znode.c:947
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d4bac6 at zfs_zget+0x56
#3 0xffffffff82d3b8e3 at zfs_get_data+0xa3
#4 0xffffffff82efa130 at zil_commit_impl+0x1130
#5 0xffffffff82d454f3 at zfs_freebsd_fsync+0xd3
#6 0xffffffff811cb084 at VOP_FSYNC_APV+0xd4
#7 0xffffffff82ee0eeb at zfs_file_fsync+0x7b
#8 0xffffffff82e31cfd at spa_write_cachefile+0x29d
#9 0xffffffff82eeda28 at zfs_ioc_pool_set_props+0x128
#10 0xffffffff82ee60a9 at zfsdev_ioctl_common+0x489
#11 0xffffffff82d272a3 at zfsdev_ioctl+0x143
#12 0xffffffff80908485 at devfs_ioctl+0xb5
#13 0xffffffff811ca944 at VOP_IOCTL_APV+0xd4
#14 0xffffffff80b5b03d at vn_ioctl+0x13d
#15 0xffffffff80908afe at devfs_ioctl_f+0x1e
#16 0xffffffff80ae4385 at kern_ioctl+0x2d5
#17 0xffffffff80ae4007 at sys_ioctl+0x127
lock order reversal:
 1st 0xfffffe0115fc7520 zfsvfs->z_teardown_inactive_lock (zfsvfs->z_teardown_inactive_lock) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/os/freebsd/zfs/zfs_vnops.c:5860
 2nd 0xfffffe0115fc7890 zfsvfs->z_hold_mtx[i] (zfsvfs->z_hold_mtx[i]) @ /wrkdirs/usr/ports/sysutils/openzfs-kmod/work/ZoF-735d1ffe9/module/os/freebsd/zfs/zfs_znode.c:1234
stack backtrace:
#0 0xffffffff80adf361 at witness_debugger+0x71
#1 0xffffffff80a7f5c7 at _sx_xlock+0x67
#2 0xffffffff82d4c9de at zfs_zinactive+0x4e
#3 0xffffffff82d464cb at zfs_freebsd_reclaim+0x5b
#4 0xffffffff811cc414 at VOP_RECLAIM_APV+0xd4
#5 0xffffffff80b4bb49 at vgonel+0x249
#6 0xffffffff80b4c0cc at vrecycle+0x5c
#7 0xffffffff82d463a1 at zfs_freebsd_inactive+0xb1
#8 0xffffffff811cc244 at VOP_INACTIVE_APV+0xd4
#9 0xffffffff80b4a838 at vinactive+0xe8
#10 0xffffffff80b4ae72 at vputx+0x322
#11 0xffffffff80b5573c at kern_unlinkat+0x20c
#12 0xffffffff80fea34e at amd64_syscall+0x2be
#13 0xffffffff80fc02ee at fast_syscall_common+0xf8
GEOM_MIRROR: Device mirror/swap0 launched (2/2).
GEOM_MIRROR: Device mirror/swap1 launched (2/2).
GEOM_MIRROR: Device mirror/swap2 launched (2/2).
GEOM_MIRROR: Device mirror/swap3 launched (2/2).
GEOM_ELI: Device mirror/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device mirror/swap1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device mirror/swap2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
GEOM_ELI: Device mirror/swap3.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: hardware
<118>Pools import completed
<118>Loading kernel modules:
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI>
warning: KLD '/boot/kernel-debug/t4_tom.ko' is newer than the linker.hints file
warning: KLD '/boot/kernel-debug/toecore.ko' is newer than the linker.hints file
<118>Setting hostname: truenas.forsythshirt.net.
<118>Setting up harvesting: [UMA],[FS_ATIME],SWI,INTERRUPT,NET_NG,NET_ETHER,NET_TUN,MOUSE,KEYBOARD,ATTACH,CACHED
<118>Feeding entropy: .
<118>Starting interfaces...
<6>bge2: link state changed to DOWN
<118>Generating configuration for interface_sync checkpoint
<118>ELF ldconfig path: /lib /usr/lib /usr/lib/compat /usr/local/lib /usr/local/lib/compat/pkg /usr/local/lib/compat/pkg /usr/local/lib/e2fsprogs /usr/local/lib/gcc9 /usr/local/lib/perl5/5.30/mach/CORE /usr/local/lib/samba4
<118>32-bit compatibility ldconfig path:
<118>add host 127.0.0.1: gateway lo0 fib 0: route already in table
<118>add net default: gateway 10.1.1.227 fib 0: route already in table
<118>add host ::1: gateway lo0 fib 0: route already in table
<118>add net fe80::: gateway ::1
<118>add net ff02::: gateway ::1
<118>add net ::ffff:0.0.0.0: gateway ::1
<118>add net ::0.0.0.0: gateway ::1
<118>Starting syslog_ng.
<118>Creating and/or trimming log files.
<118>Starting rpcbind.
<118>Updating CPU Microcode...
CPU: Intel(R) Xeon(R) CPU E5-2667 0 @ 2.90GHz (2900.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206d7  Family=0x6  Model=0x2d  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
<118>Done.
<118>Starting watchdogd.
<118>Starting ctld.
<118>Starting nfsuserd.
<118>Starting mountd.
<118>Starting nfsd.
nfsd: can't register svc name
<118>Starting statd.
<118>Starting lockd.
<118>Updating motd:.
<118>Mounting late filesystems:.
Security policy loaded: MAC/ntpd (mac_ntpd)
<118>Starting ntpd.
<118>Oct 15 01:53:41 truenas 1 2020-10-15T01:53:41.276657-04:00 truenas.forsythshirt.net ntpd 2513 - - restrict: ignoring line 10, address/host '0.freebsd.pool.ntp.org' unusable.
<118>Oct 15 01:53:41 truenas 1 2020-10-15T01:53:41.277086-04:00 truenas.forsythshirt.net ntpd 2513 - - restrict: ignoring line 11, address/host '1.freebsd.pool.ntp.org' unusable.
<118>Oct 15 01:53:41 truenas 1 2020-10-15T01:53:41.277478-04:00 truenas.forsythshirt.net ntpd 2513 - - restrict: ignoring line 12, address/host '2.freebsd.pool.ntp.org' unusable.
<118>Updating /var/run/os-release done.
<6>bge2: link state changed to UP
<118>Starting smartd_daemon.
<118>Starting rrdcached.
<118>Performing sanity check on openssh configuration.
<118>Starting openssh.
<118>Performing sanity check on nginx configuration:
<118>nginx: the configuration file /usr/local/etc/nginx/nginx.conf syntax is ok
<118>nginx: configuration file /usr/local/etc/nginx/nginx.conf test is successful
<118>Starting nginx.
<118>Starting dbus.
<118>Starting collectd_daemon.
<118>Syncing multipaths...
<118>Configuring vt: blanktime.
<118>Starting cron.
<118>net.inet.carp.allow: 0 -> 1
<118>
<118>Thu Oct 15 01:53:44 EDT 2020
nfsd: can't register svc name
nfsd: can't register svc name
<6>ix1: link state changed to UP
panic: VERIFY(err == 0) failed

cpuid = 23
time = 1602741592
KDB: stack backtrace:
db_trace_self_wrapper() at db_trace_self_wrapper+0x2b/frame 0xfffffe0135f35840
vpanic() at vpanic+0x17b/frame 0xfffffe0135f35890
spl_panic() at spl_panic+0x3a/frame 0xfffffe0135f358f0
dmu_buf_set_init() at dmu_buf_set_init+0x3c0/frame 0xfffffe0135f35980
dmu_issue() at dmu_issue+0x10d/frame 0xfffffe0135f359c0
dmu_read_uio_dbuf() at dmu_read_uio_dbuf+0x79/frame 0xfffffe0135f35b30
zfs_freebsd_read() at zfs_freebsd_read+0x6a9/frame 0xfffffe0135f35be0
VOP_READ_APV() at VOP_READ_APV+0xce/frame 0xfffffe0135f35c10
nfsvno_read() at nfsvno_read+0x36d/frame 0xfffffe0135f35ce0
nfsrvd_read() at nfsrvd_read+0x579/frame 0xfffffe0135f35f40
nfsrvd_dorpc() at nfsrvd_dorpc+0x6e8/frame 0xfffffe0135f36110
nfssvc_program() at nfssvc_program+0x596/frame 0xfffffe0135f362d0
svc_run_internal() at svc_run_internal+0x99f/frame 0xfffffe0135f36410
svc_run() at svc_run+0x253/frame 0xfffffe0135f36470
nfsrvd_nfsd() at nfsrvd_nfsd+0x33c/frame 0xfffffe0135f365d0
nfssvc_nfsd() at nfssvc_nfsd+0x553/frame 0xfffffe0135f36aa0
sys_nfssvc() at sys_nfssvc+0xca/frame 0xfffffe0135f36ac0
amd64_syscall() at amd64_syscall+0x2be/frame 0xfffffe0135f36bf0
fast_syscall_common() at fast_syscall_common+0xf8/frame 0xfffffe0135f36bf0
--- syscall (155, FreeBSD ELF64, sys_nfssvc), rip = 0x8002e1aaa, rsp = 0x7fffffffe558, rbp = 0x7fffffffe7f0 ---
KDB: enter: panic
