run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config

Joined
Dec 29, 2014
Messages
1,135
I just started getting this message during boot on both my FreeNAS systems (FreeNAS-11.1-U5). Both systems are Cisco UCS C240 M3S. Primary has dual Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz with 128GB RAM, and the secondary had dual Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz with 64GB of RAM. LSI 9271 in JBOD mode for internal drives (24 2.5" drive bays), LSI 9207-8e, Chelsio T520-CR NIC (storage facing), and onboard Intel 350's (user facing). Both systems boot from an SD card, and the system database is on a mirrored set of 300GB SAS drives. Dmesg and pciconf info below or attached.

Code:
Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
   The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.1-STABLE #0 r321665+e0c4ca60dfc(freenas/11.1-stable): Wed May 30 14:18:20 EDT 2018
	root@nemesis.tn.ixsystems.com:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn)
CPU: Intel(R) Xeon(R) CPU E5-2660 v2 @ 2.20GHz (2200.05-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x306e4  Family=0x6  Model=0x3e  Stepping=4
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7fbee3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features=0x281<FSGSBASE,SMEP,ERMS>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr
  TSC: P-state invariant, performance statistics
real memory  = 139586437120 (133120 MB)
avail memory = 133477036032 (127293 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <Cisco0 CiscoUCS>
FreeBSD/SMP: Multiprocessor System Detected: 40 CPUs
FreeBSD/SMP: 2 package(s) x 10 core(s) x 2 hardware threads
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
SMP: AP CPU #30 Launched!
SMP: AP CPU #31 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #24 Launched!
SMP: AP CPU #25 Launched!
SMP: AP CPU #20 Launched!
SMP: AP CPU #38 Launched!
SMP: AP CPU #39 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #32 Launched!
SMP: AP CPU #22 Launched!
SMP: AP CPU #23 Launched!
SMP: AP CPU #18 Launched!
SMP: AP CPU #34 Launched!
SMP: AP CPU #35 Launched!
SMP: AP CPU #17 Launched!
SMP: AP CPU #28 Launched!
SMP: AP CPU #29 Launched!
SMP: AP CPU #19 Launched!
SMP: AP CPU #16 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #37 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #10 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #26 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #36 Launched!
SMP: AP CPU #27 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #1 Launched!
SMP: AP CPU #33 Launched!
SMP: AP CPU #21 Launched!
Timecounter "TSC-low" frequency 1100023158 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
nexus0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <Cisco0 CiscoUCS> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> numa-domain 0 on acpi0
cpu1: <ACPI CPU> numa-domain 0 on acpi0
cpu2: <ACPI CPU> numa-domain 0 on acpi0
cpu3: <ACPI CPU> numa-domain 0 on acpi0
cpu4: <ACPI CPU> numa-domain 0 on acpi0
cpu5: <ACPI CPU> numa-domain 0 on acpi0
cpu6: <ACPI CPU> numa-domain 0 on acpi0
cpu7: <ACPI CPU> numa-domain 0 on acpi0
cpu8: <ACPI CPU> numa-domain 0 on acpi0
cpu9: <ACPI CPU> numa-domain 0 on acpi0
cpu10: <ACPI CPU> numa-domain 0 on acpi0
cpu11: <ACPI CPU> numa-domain 0 on acpi0
cpu12: <ACPI CPU> numa-domain 0 on acpi0
cpu13: <ACPI CPU> numa-domain 0 on acpi0
cpu14: <ACPI CPU> numa-domain 0 on acpi0
cpu15: <ACPI CPU> numa-domain 0 on acpi0
cpu16: <ACPI CPU> numa-domain 0 on acpi0
cpu17: <ACPI CPU> numa-domain 0 on acpi0
cpu18: <ACPI CPU> numa-domain 0 on acpi0
cpu19: <ACPI CPU> numa-domain 0 on acpi0
cpu20: <ACPI CPU> numa-domain 1 on acpi0
cpu21: <ACPI CPU> numa-domain 1 on acpi0
cpu22: <ACPI CPU> numa-domain 1 on acpi0
cpu23: <ACPI CPU> numa-domain 1 on acpi0
cpu24: <ACPI CPU> numa-domain 1 on acpi0
cpu25: <ACPI CPU> numa-domain 1 on acpi0
cpu26: <ACPI CPU> numa-domain 1 on acpi0
cpu27: <ACPI CPU> numa-domain 1 on acpi0
cpu28: <ACPI CPU> numa-domain 1 on acpi0
cpu29: <ACPI CPU> numa-domain 1 on acpi0
cpu30: <ACPI CPU> numa-domain 1 on acpi0
cpu31: <ACPI CPU> numa-domain 1 on acpi0
cpu32: <ACPI CPU> numa-domain 1 on acpi0
cpu33: <ACPI CPU> numa-domain 1 on acpi0
cpu34: <ACPI CPU> numa-domain 1 on acpi0
cpu35: <ACPI CPU> numa-domain 1 on acpi0
cpu36: <ACPI CPU> numa-domain 1 on acpi0
cpu37: <ACPI CPU> numa-domain 1 on acpi0
cpu38: <ACPI CPU> numa-domain 1 on acpi0
cpu39: <ACPI CPU> numa-domain 1 on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Event timer "HPET4" frequency 14318180 Hz quality 340
Event timer "HPET5" frequency 14318180 Hz quality 340
Event timer "HPET6" frequency 14318180 Hz quality 340
Event timer "HPET7" frequency 14318180 Hz quality 340
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> numa-domain 0 on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 26 at device 1.0 numa-domain 0 on pci0
pci1: <ACPI PCI bus> numa-domain 0 on pcib1
pcib2: <ACPI PCI-PCI bridge> irq 26 at device 1.1 numa-domain 0 on pci0
pci2: <ACPI PCI bus> numa-domain 0 on pcib2
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x8060-0x807f mem 0xdec00000-0xdecfffff,0xdef0c000-0xdef0ffff irq 27 at device 0.0 numa-domain 0 on pci2
igb0: Using MSIX interrupts with 9 vectors
igb0: Ethernet address: e4:c7:22:84:8f:30
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: Bound queue 4 to cpu 4
igb0: Bound queue 5 to cpu 5
igb0: Bound queue 6 to cpu 6
igb0: Bound queue 7 to cpu 7
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x8040-0x805f mem 0xdeb00000-0xdebfffff,0xdef08000-0xdef0bfff irq 30 at device 0.1 numa-domain 0 on pci2
igb1: Using MSIX interrupts with 9 vectors
igb1: Ethernet address: e4:c7:22:84:8f:31
igb1: Bound queue 0 to cpu 8
igb1: Bound queue 1 to cpu 9
igb1: Bound queue 2 to cpu 10
igb1: Bound queue 3 to cpu 11
igb1: Bound queue 4 to cpu 12
igb1: Bound queue 5 to cpu 13
igb1: Bound queue 6 to cpu 14
igb1: Bound queue 7 to cpu 15
igb2: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x8020-0x803f mem 0xdea00000-0xdeafffff,0xdef04000-0xdef07fff irq 28 at device 0.2 numa-domain 0 on pci2
igb2: Using MSIX interrupts with 9 vectors
igb2: Ethernet address: e4:c7:22:84:8f:32
igb2: Bound queue 0 to cpu 16
igb2: Bound queue 1 to cpu 17
igb2: Bound queue 2 to cpu 18
igb2: Bound queue 3 to cpu 19
igb2: Bound queue 4 to cpu 20
igb2: Bound queue 5 to cpu 21
igb2: Bound queue 6 to cpu 22
igb2: Bound queue 7 to cpu 23
igb3: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x8000-0x801f mem 0xde900000-0xde9fffff,0xdef00000-0xdef03fff irq 29 at device 0.3 numa-domain 0 on pci2
igb3: Using MSIX interrupts with 9 vectors
igb3: Ethernet address: e4:c7:22:84:8f:33
igb3: Bound queue 0 to cpu 24
igb3: Bound queue 1 to cpu 25
igb3: Bound queue 2 to cpu 26
igb3: Bound queue 3 to cpu 27
igb3: Bound queue 4 to cpu 28
igb3: Bound queue 5 to cpu 29
igb3: Bound queue 6 to cpu 30
igb3: Bound queue 7 to cpu 31
pcib3: <ACPI PCI-PCI bridge> irq 32 at device 2.0 numa-domain 0 on pci0
pci3: <ACPI PCI bus> numa-domain 0 on pcib3
mps0: <Avago Technologies (LSI) SAS2308> port 0x7000-0x70ff mem 0xdf140000-0xdf14ffff,0xdf100000-0xdf13ffff irq 32 at device 0.0 numa-domain 0 on pci3
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
pcib4: <ACPI PCI-PCI bridge> irq 32 at device 2.2 numa-domain 0 on pci0
pci4: <ACPI PCI bus> numa-domain 0 on pcib4
pcib5: <ACPI PCI-PCI bridge> irq 42 at device 3.0 numa-domain 0 on pci0
pci5: <ACPI PCI bus> numa-domain 0 on pcib5
nvme0: <Generic NVMe Device> mem 0xdf210000-0xdf213fff irq 40 at device 0.0 numa-domain 0 on pci5
pcib6: <ACPI PCI-PCI bridge> irq 16 at device 17.0 numa-domain 0 on pci0
pci6: <ACPI PCI bus> numa-domain 0 on pcib6
ehci0: <Intel Patsburg USB 2.0 controller> mem 0xdf301000-0xdf3013ff irq 22 at device 26.0 numa-domain 0 on pci0
usbus0: EHCI version 1.0
usbus0 numa-domain 0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
pcib7: <ACPI PCI-PCI bridge> irq 17 at device 28.0 numa-domain 0 on pci0
pci7: <ACPI PCI bus> numa-domain 0 on pcib7
pcib8: <ACPI PCI-PCI bridge> irq 19 at device 28.7 numa-domain 0 on pci0
pci8: <ACPI PCI bus> numa-domain 0 on pcib8
vgapci0: <VGA-compatible display> mem 0xdd000000-0xddffffff,0xde810000-0xde813fff,0xde000000-0xde7fffff irq 19 at device 0.0 numa-domain 0 on pci8
vgapci0: Boot video device
ehci1: <Intel Patsburg USB 2.0 controller> mem 0xdf300000-0xdf3003ff irq 23 at device 29.0 numa-domain 0 on pci0
usbus1: EHCI version 1.0
usbus1 numa-domain 0 on ehci1
usbus1: 480Mbps High Speed USB v2.0
pcib9: <ACPI PCI-PCI bridge> at device 30.0 numa-domain 0 on pci0
pci9: <ACPI PCI bus> numa-domain 0 on pcib9
isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci0
isa0: <ISA bus> numa-domain 0 on isab0
pcib10: <ACPI Host-PCI bridge> numa-domain 0 on acpi0
pci10: <ACPI PCI bus> numa-domain 0 on pcib10
pci10: <dasp, performance counters> at device 14.1 (no driver attached)
pci10: <dasp, performance counters> at device 19.1 (no driver attached)
pci10: <dasp, performance counters> at device 19.5 (no driver attached)
pcib11: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pcib11: _OSC returned error 0x10
pci11: <ACPI PCI bus> numa-domain 1 on pcib11
pcib12: <ACPI PCI-PCI bridge> irq 50 at device 1.0 numa-domain 1 on pci11
pci12: <ACPI PCI bus> numa-domain 1 on pcib12
pcib13: <ACPI PCI-PCI bridge> irq 56 at device 2.0 numa-domain 1 on pci11
pci13: <ACPI PCI bus> numa-domain 1 on pcib13
AVAGO MegaRAID SAS FreeBSD mrsas driver version: 06.712.04.00-fbsd
mrsas0: <AVAGO Thunderbolt SAS Controller> port 0xf000-0xf0ff mem 0xfbc60000-0xfbc63fff,0xfbc00000-0xfbc3ffff irq 56 at device 0.0 numa-domain 1 on pci13
mrsas0: Using MSI-X with 16 number of vectors
mrsas0: FW supports <16> MSIX vector,Online CPU 40 Current MSIX <16>
mrsas0: MSI-x interrupts setup success
pcib14: <ACPI PCI-PCI bridge> irq 64 at device 3.0 numa-domain 1 on pci11
pci14: <ACPI PCI bus> numa-domain 1 on pcib14
t5nex0: <Chelsio T520-CR> mem 0xfb300000-0xfb37ffff,0xfa000000-0xfaffffff,0xfbb04000-0xfbb05fff irq 64 at device 0.4 numa-domain 1 on pci14
cxl0: <port 0> numa-domain 1 on t5nex0
cxl0: Ethernet address: 00:07:43:2d:14:70
cxl0: 16 txq, 8 rxq (NIC)
cxl1: <port 1> numa-domain 1 on t5nex0
cxl1: Ethernet address: 00:07:43:2d:14:78
cxl1: 16 txq, 8 rxq (NIC)
t5nex0: PCIe gen3 x8, 2 ports, 18 MSI-X interrupts, 51 eq, 17 iq
pci14: <mass storage, SCSI> at device 0.5 (no driver attached)
pci14: <serial bus, Fibre Channel> at device 0.6 (no driver attached)
pcib15: <ACPI Host-PCI bridge> numa-domain 1 on acpi0
pci15: <ACPI PCI bus> numa-domain 1 on pcib15
pci15: <dasp, performance counters> at device 14.1 (no driver attached)
pci15: <dasp, performance counters> at device 19.1 (no driver attached)
pci15: <dasp, performance counters> at device 19.5 (no driver attached)
acpi_button0: <Power Button> on acpi0
acpi_acad0: <AC Adapter> on acpi0
acpi_syscontainer0: <System Container> on acpi0
acpi_syscontainer1: <System Container> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
ichwd0: <Intel Patsburg watchdog timer> on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
ichwd0: <Intel Patsburg watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0
ichwd0: ICH WDT present but disabled in BIOS or hardware
device_attach: ichwd0 attach returned 6
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc87ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
coretemp0: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu0
est0: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu0
coretemp1: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu1
est1: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu1
coretemp2: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu2
est2: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu2
coretemp3: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu3
est3: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu3
coretemp4: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu4
est4: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu4
coretemp5: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu5
est5: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu5
coretemp6: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu6
est6: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu6
coretemp7: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu7
est7: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu7
coretemp8: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu8
est8: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu8
coretemp9: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu9
est9: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu9
coretemp10: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu10
est10: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu10
coretemp11: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu11
est11: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu11
coretemp12: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu12
est12: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu12
coretemp13: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu13
est13: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu13
coretemp14: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu14
est14: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu14
coretemp15: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu15
est15: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu15
coretemp16: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu16
est16: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu16
coretemp17: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu17
est17: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu17
coretemp18: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu18
est18: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu18
coretemp19: <CPU On-Die Thermal Sensors> numa-domain 0 on cpu19
est19: <Enhanced SpeedStep Frequency Control> numa-domain 0 on cpu19
coretemp20: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu20
est20: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu20
coretemp21: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu21
est21: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu21
coretemp22: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu22
est22: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu22
coretemp23: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu23
est23: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu23
coretemp24: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu24
est24: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu24
coretemp25: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu25
est25: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu25
coretemp26: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu26
est26: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu26
coretemp27: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu27
est27: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu27
coretemp28: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu28
est28: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu28
coretemp29: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu29
est29: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu29
coretemp30: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu30
est30: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu30
coretemp31: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu31
est31: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu31
coretemp32: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu32
est32: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu32
coretemp33: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu33
est33: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu33
coretemp34: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu34
est34: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu34
coretemp35: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu35
est35: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu35
coretemp36: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu36
est36: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu36
coretemp37: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu37
est37: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu37
coretemp38: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu38
est38: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu38
coretemp39: <CPU On-Die Thermal Sensors> numa-domain 1 on cpu39
est39: <Enhanced SpeedStep Frequency Control> numa-domain 1 on cpu39
random: unblocking device.
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen0.1: <Intel EHCI root HUB> at usbus0
ugen1.1: <Intel EHCI root HUB> at usbus1
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
nvd0: <INTEL SSDPED1D280GA> NVMe namespace
nvd0: 267090MB (547002288 512 byte sectors)
mrsas0: Disestablish mrsas intr hook
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
uhub2 numa-domain 0 on uhub0
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087 product 0x0024> at usbus1
uhub3 numa-domain 0 on uhub1
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
uhub2: 6 ports with 6 removable, self powered
uhub3: 8 ports with 8 removable, self powered
ugen0.3: <Avocent Cisco USB Composite Device-0> at usbus0
ukbd0 numa-domain 0 on uhub2
ukbd0: <Keyboard> on usbus0
kbd0 at ukbd0
ugen1.3: <Avocent Avocent USBIAC> at usbus1
ukbd1 numa-domain 0 on uhub3
ukbd1: <EP1 Interrupt> on usbus1
kbd2 at ukbd1
ugen1.4: <Cypress West Bridge> at usbus1
umass0 numa-domain 0 on uhub3
umass0: <Cypress West Bridge, class 0/0, rev 2.00/0.02, addr 4> on usbus1
umass0:  SCSI over Bulk-Only; quirks = 0x4100
umass0:4:0: Attached to scbus4
ugen1.5: <vendor 0x0424 product 0x2512> at usbus1
uhub4 numa-domain 0 on uhub3
uhub4: <vendor 0x0424 product 0x2512, class 9/0, rev 2.00/b.b3, addr 5> on usbus1
uhub4: MTT enabled
uhub4: 2 ports with 2 removable, self powered
run_interrupt_driven_hooks: still waiting after 60 seconds for xpt_config
run_interrupt_driven_hooks: still waiting after 120 seconds for xpt_config
mrsas0: Initiating Target RESET because of SCSI IO timeout!
mrsas0: Task management NOT SUPPORTED for CAM target:26
mrsas0: target reset FAIL!!
mrsas0: Initiaiting OCR because of TM FAILURE!
da1 at mrsas0 bus 1 scbus2 target 28 lun 0
da2 at mrsas0 bus 1 scbus2 target 29 lun 0
da0 at mrsas0 bus 1 scbus2 target 27 lun 0
da4 at mrsas0 bus 1 scbus2 target 31 lun 0
da5 at mrsas0 bus 1 scbus2 target 32 lun 0
da3 at mrsas0 bus 1 scbus2 target 30 lun 0
da1: <ATA ST91000640NS CC03> Fixed Direct Access SPC-4 SCSI device
da3: <ATA ST91000640NS CC03> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number 9XG8SX92
da3: 150.000MB/s transfers
da3: 953869MB (1953525168 512 byte sectors)
da19 at umass-sim0 bus 0 scbus4 target 0 lun 0
da19: <HV Hypervisor_0 1.01> Removable Direct Access SCSI device
da19: Serial Number 012345678901
da19: 40.000MB/s transfers
da19: 29696MB (60817408 512 byte sectors)
da19: quirks=0x2<NO_6_BYTE>
da15 at mrsas0 bus 1 scbus2 target 51 lun 0
da15: <ATA ST91000640NS BK03> Fixed Direct Access SPC-4 SCSI device
da15: Serial Number 9XG1A4XV
da15: 150.000MB/s transfers
da15: 953869MB (1953525168 512 byte sectors)
da4: <ATA ST91000640NS CC03> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number 9XG7T4CV
da4: 150.000MB/s transfersda1: Serial Number 9XG7TGL6
da1: 150.000MB/s transfers
da1: 953869MB (1953525168 512 byte sectors)



I am not normally watching the systems boot, but I think this started at the time when I added the 9207-8e to both systems. The secondary has an HP D2700 external enclosure attached (multipath) to the 9207, but the primary doesn't have anything attached there at the moment. The onboard SATA controller is disabled. I looked at the other forum threads on this issue, but I didn't see anything that matched my symptoms. Any ideas?
 

Attachments

  • pci.txt
    28.5 KB · Views: 473
Joined
Dec 29, 2014
Messages
1,135
I realize I failed to mention that the controller BIOS was disabled on the 9207, but enabled on the 9271. In the secondary system, disabling the BIOS on the 9271 resolved the problem. On the primary the problem remains after disabling the 9271 BIOS. I wonder if the fact that the 9207 has nothing attached to it is contributing to the problem in the primary. Everything does still work correctly. It just causes a 2-3 minute delay in the boot process.
 
Joined
Dec 29, 2014
Messages
1,135
Joined
Dec 29, 2014
Messages
1,135
I removed the LSI 9271 RAID cards, and replaced them with LSI 9207-8i controllers. Now all seem to be happy.
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
I have nearly the same setup, Cisco C240-M3S but I am using a NetApp shelf.

The big problem I am having is that it was working very well for a few weeks, and now I cant get past this screen
1577128354704.png


The server crashed at one point, and now it wont come back up.

Any ideas?
 
Joined
Dec 29, 2014
Messages
1,135
I have nearly the same setup, Cisco C240-M3S but I am using a NetApp shelf.
It is only a guess, but that sure looks like you are having issues communicating with the external shelf. What version of FreeNAS, and what exact hardware are you using (specifically to talk to the external shelf)?
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
Thanks for the quick reply!

I have:
  1. NetApp 111-00341+F1 4-port 6Gbps SAS Disk/Tape QSFP PCIe HBA Card PM8003 Sierra
  2. NetApp DS4246 Disk Array Shelf W/ 24x SAS Trays 2x IOM6 SAS Expansion Array
  3. HGST HUH728080AL4200
Thanks!
 
Joined
Dec 29, 2014
Messages
1,135
NetApp 111-00341+F1 4-port 6Gbps SAS Disk/Tape QSFP PCIe HBA Card PM8003 Sierra
I am not at all familiar with this card, so I can't offer anything constructive there. It does like rather like an LSI 9207-8e. What kind of hardware does FreeNAS identify that card as? Here is what mine looks like inside of code tags.
Code:
mps0: <Avago Technologies (LSI) SAS2308> port 0x7000-0x70ff mem 0xdf140000-0xdf14ffff,0xdf100000-0xdf13ffff irq 34 at device 0.0 numa-domain 0 on pci4
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>
mps1: <Avago Technologies (LSI) SAS2308> port 0xf000-0xf0ff mem 0xfbe40000-0xfbe4ffff,0xfbe00000-0xfbe3ffff irq 56 at device 0.0 numa-domain 1 on pci13
mps1: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps1: IOCCapabilities: 5a85c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,MSIXIndex,HostDisc>

Based on your description, my gut still goes towards a hardware failure. HBA, cables, or external array/array IO controller.
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
I wish I knew, I can’t get anything past that screen. Is there a safe mode boot I can use to get around it?
 
Joined
Dec 29, 2014
Messages
1,135
I would try taking the HBA out of the chassis and see if it boots. I am basing that on the assumption that you are booting from some media other than the drives in the external array. It will probably whine if it expects to see a ZFS pool that it can't mount. You could also try the Cisco diagnostic DVD. Not to carp, but it sure sounds like a hardware fault... If possible, remove components until it stops crashing. [Disclaimer]You are responsible for your own files and equipment. Use your own judgement and discretion! :) [/Disclaimer]
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
Apologies for the delay, was getting on site. I have a 9240-8i HBA in there. Unplugging the shelf and will see what comes of it
 
Joined
Dec 29, 2014
Messages
1,135
I have a 9240-8i HBA in there.
That is a RAID card, not an HBA. I suspect that is why you are getting the "run_interrupt_hooks" message, but that message is mostly cosmetic. Your larger issue is still likely hardware. I'll be interested to see what you discover. If the error go away when the external shelf is unplugged, I would unplug, reseat, and reconnect things to see if anything changes.
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
So unplugging that let it boot. One cable (for whatever reason) when re-inserted, only showed 2 drives and the other didn't show any drives. Not sure whats is the deal here.

Apologies for my terminology, I am newer to the storage space.
 
Joined
Dec 29, 2014
Messages
1,135
So unplugging that let it boot. One cable (for whatever reason) when re-inserted, only showed 2 drives and the other didn't show any drives. Not sure whats is the deal here.
Could be some issues with multi-path or possibly one of the IO modules in the external shelf.
Apologies for my terminology, I am newer to the storage space.
No worries. I just wanted to make the point that FreeNAS works great with HBA cards (ones the just present the drives and nothing more), and not so well with RAID cards that can obscure some of the drive properties. What happens if you power everything down, re-seat the connections, and then try to boot it?
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
So rebooted the device, detatched from the shelf. Let it boot and get happy. Then I attached the shelf to the NetApp. At first it only saw the two drives, then after some time it found the others.

`zpool list` shows the full array finally. I remember now that one of the drives never showed up for some reason. Will see if I can find that drive and if removal helps anything.
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
Ok, getting some logs here:

Code:
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): READ(6). CDB: 08 00 00 04 1c 00
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): CAM status: SCSI Status Error
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): SCSI status: Check Condition
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): SCSI sense: HARDWARE FAILURE asc:40,a0 ((null))
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0):
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Field Replaceable Unit: 0
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Command Specific Info: 0
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0):
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Descriptor 0x80: f1 19
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Descriptor 0x81: ff ff ff ff 00 0f
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Retrying command (per sense data)
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): READ(6). CDB: 08 00 00 44 1c 00
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): CAM status: SCSI Status Error
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): SCSI status: Check Condition
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): SCSI sense: HARDWARE FAILURE asc:3e,3 (Logical unit failed self-test)
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0):
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Field Replaceable Unit: 0
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Command Specific Info: 0
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0):
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Descriptor 0x80: f7 5d
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Descriptor 0x81: ff ff ff ff 00 0f
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): Retrying command (per sense data)
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): READ(10). CDB: 28 00 74 70 24 c4 00 00 1c 00
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): CAM status: SCSI Status Error
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): SCSI status: Check Condition
Dec 23 16:08:02 san1 (da9:pmspcbsd1:0:9:0): SCSI sense: HARDWARE FAILURE asc:3e,3 (Logical unit failed self-test)
 
Joined
Dec 29, 2014
Messages
1,135
Your errors are from the pms driver - "pms - PMC-Sierra PM8001/8081/8088/8089/8074/8076/8077 SAS/SATA HBA Controller". They all reference da9, so, that would seem to be the particularly annoyed component. You can get that info (hopefully) with a camcontrol inquiry da9. On my system, that looks like this.
Code:
root@freenas2:/nonexistent # camcontrol inquiry da9
pass9: <ATA ST91000640NS CC02> Fixed Direct Access SPC-4 SCSI device
pass9: Serial Number             9XG2TFBP
pass9: 600.000MB/s transfers, Command Queueing Enabled

It may require powering things off and pulling drives to check, but the serial number should help you identify the failing drive.
 

mactoserver

Dabbler
Joined
Dec 23, 2019
Messages
12
So I was thinking I had a bad drive in there, but in the process of debugging it, I pulled all the drives out of the shelf, but left the cables connected to the NetApp but that locks it up too. Grrr, this is getting annoying
 
Joined
Dec 29, 2014
Messages
1,135
That would point to the "pms" card, the cables, or something in the external shelf/IO modules.
 
Top