Hello All,
Long time lurker of the forums, but first time poster (I think?).
I have a new build I'm working on at the moment, and am hoping the community can help me understand whether my problem is related to the hardware, or if it's something else.
We've got an HP DL360 Gen9 as the server hardware. Dual Proc, 192G of RAM, and 2 500GB SAS disks in RAID 1 attached to a P440ar RAID controller for the OS. FreeNAS 11.1-U5 is installed on this server.
We then have 5 HP DAS devices. One D3700 2.5" 25-bay DAS, followed by 4 D3600 3.5" 12-bay DAS devices. These interconnect to the DL360 Gen9 via a PCI H241 HBA.
We see all of the drives in HP's storage view of iLO, we don't see any issues with cabling whatsoever, two of the drives are showing SMART issues in iLO and will be replaced during the week.
When we go to create our first storage pool, we notice that only drives from the first three daisy chained DAS arrays are visible. dmesg doesn't seem to even detect any drives past the first 3 DAS arrays.
Output of camcontrol devlist:
Output of dmesg:
Interestingly, only ONE of the disks that iLO lists as pending failure seems to be showing up in FreeNAS, however FreeNAS shows that 2 sequential devices are apparently somehow failing?
Are we missing something here? I'm aware of the fact that the H241 may not be ideal, and if the answer is buy a 9300-8e and be done with it, then fine. What I'm hoping for is some diagnostic direction to take to ensure that before we spend more money on extra hardware, we determine whether this is a daisy chain limitation imposed by the D3600/D3700 arrays, or whether this is down to the fact that FreeNAS/FreeBSD treats the H241 card like a RAID controller, even though it seems to know enough to present the disks as direct-attached SCSI. We've run 3-DAS setups like this for quite a while now, without issue.
Firmware on the DAS's have been updated to latest available, as well as on the H241 card.
Thanks for any help - much appreciated.
Long time lurker of the forums, but first time poster (I think?).
I have a new build I'm working on at the moment, and am hoping the community can help me understand whether my problem is related to the hardware, or if it's something else.
We've got an HP DL360 Gen9 as the server hardware. Dual Proc, 192G of RAM, and 2 500GB SAS disks in RAID 1 attached to a P440ar RAID controller for the OS. FreeNAS 11.1-U5 is installed on this server.
We then have 5 HP DAS devices. One D3700 2.5" 25-bay DAS, followed by 4 D3600 3.5" 12-bay DAS devices. These interconnect to the DL360 Gen9 via a PCI H241 HBA.
We see all of the drives in HP's storage view of iLO, we don't see any issues with cabling whatsoever, two of the drives are showing SMART issues in iLO and will be replaced during the week.
When we go to create our first storage pool, we notice that only drives from the first three daisy chained DAS arrays are visible. dmesg doesn't seem to even detect any drives past the first 3 DAS arrays.
Output of camcontrol devlist:
Code:
root@nas02:~ # camcontrol devlist <HP RAID 1(1+0) OK> at scbus0 target 0 lun 0 (pass0,da0) <HP MO0800JDVEV HPD2> at scbus3 target 16 lun 0 (pass1,da1) <HP MO0800JDVEV HPD2> at scbus3 target 17 lun 0 (pass2,da2) <HP MO0800JEFPB HPD3> at scbus3 target 18 lun 0 (pass3,da3) <HP MO0800JEFPB HPD3> at scbus3 target 19 lun 0 (pass4,da4) <HP MO0800JEFPB HPD3> at scbus3 target 20 lun 0 (pass5,da5) <HP MO0800JEFPB HPD3> at scbus3 target 21 lun 0 (pass6,da6) <HP MO0800JEFPB HPD3> at scbus3 target 22 lun 0 (pass7,da7) <HP MO0800JEFPB HPD3> at scbus3 target 23 lun 0 (pass8,da8) <HP MB6000JEFND HPD9> at scbus3 target 41 lun 0 (pass9,da9) <HP MB6000JEFND HPD9> at scbus3 target 42 lun 0 (pass10,da10) <HP MB6000JEFND HPD9> at scbus3 target 43 lun 0 (pass11,da11) <HP MB6000JEFND HPD9> at scbus3 target 44 lun 0 (pass12,da12) <HP MB6000JEFND HPD9> at scbus3 target 45 lun 0 (pass13,da13) <HP MB6000JEFND HPD9> at scbus3 target 46 lun 0 (pass14,da14) <HP MB6000JEFND HPD9> at scbus3 target 47 lun 0 (pass15,da15) <HP MB6000JEFND HPD9> at scbus3 target 48 lun 0 (pass16,da16) <HP MB6000JEFND HPD9> at scbus3 target 49 lun 0 (pass17,da17) <HP MB6000JEFND HPD9> at scbus3 target 53 lun 0 (pass18,da18) <HP MB6000JEFND HPD9> at scbus3 target 54 lun 0 (pass19,da19) <HP MB6000JEFND HPD9> at scbus3 target 55 lun 0 (pass20,da20) <HP MB6000JEFND HPD9> at scbus3 target 56 lun 0 (pass21,da21) <HP MB6000JEFND HPD9> at scbus3 target 57 lun 0 (pass22,da22) <HP MB6000JEFND HPD9> at scbus3 target 58 lun 0 (pass23,da23) <HP MB6000JEFND HPD9> at scbus3 target 59 lun 0 (pass24,da24) <HP MB6000JEFND HPD9> at scbus3 target 60 lun 0 (pass25,da25) <HP MB6000JEFND HPD9> at scbus3 target 61 lun 0 (pass26,da26)
Output of dmesg:
Code:
root@nas02:~ # dmesg Copyright (c) 1992-2017 The FreeBSD Project. Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994 The Regents of the University of California. All rights reserved. FreeBSD is a registered trademark of The FreeBSD Foundation. FreeBSD 11.1-STABLE #0 r321665+e0c4ca60dfc(freenas/11.1-stable): Wed May 30 14:18:20 EDT 2018 root@nemesis.tn.ixsystems.com:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64 FreeBSD clang version 5.0.0 (tags/RELEASE_500/final 312559) (based on LLVM 5.0.0svn) VT(efifb): resolution 1024x768 CPU: Intel(R) Xeon(R) CPU E5-2678 v3 @ 2.50GHz (2497.17-MHz K8-class CPU) Origin="GenuineIntel" Id=0x306f2 Family=0x6 Model=0x3f Stepping=2 Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE> Features2=0x7ffefbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND> AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM> AMD Features2=0x21<LAHF,ABM> Structured Extended Features=0x37ab<FSGSBASE,TSCADJ,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,PQM,NFPUSG> XSAVE Features=0x1<XSAVEOPT> VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID,VID,PostIntr TSC: P-state invariant, performance statistics real memory = 208305913856 (198656 MB) avail memory = 200331149312 (191050 MB) Event timer "LAPIC" quality 600 ACPI APIC Table: <HP ProLiant> FreeBSD/SMP: Multiprocessor System Detected: 48 CPUs FreeBSD/SMP: 2 package(s) x 12 core(s) x 2 hardware threads WARNING: VIMAGE (virtualized network stack) is a highly experimental feature. ioapic0 <Version 2.0> irqs 0-23 on motherboard ioapic1 <Version 2.0> irqs 24-47 on motherboard ioapic2 <Version 2.0> irqs 48-71 on motherboard SMP: AP CPU #1 Launched! SMP: AP CPU #16 Launched! SMP: AP CPU #40 Launched! SMP: AP CPU #24 Launched! SMP: AP CPU #11 Launched! SMP: AP CPU #4 Launched! SMP: AP CPU #9 Launched! SMP: AP CPU #17 Launched! SMP: AP CPU #28 Launched! SMP: AP CPU #14 Launched! SMP: AP CPU #5 Launched! SMP: AP CPU #8 Launched! SMP: AP CPU #7 Launched! SMP: AP CPU #46 Launched! SMP: AP CPU #41 Launched! SMP: AP CPU #10 Launched! SMP: AP CPU #27 Launched! SMP: AP CPU #18 Launched! SMP: AP CPU #19 Launched! SMP: AP CPU #36 Launched! SMP: AP CPU #15 Launched! SMP: AP CPU #25 Launched! SMP: AP CPU #3 Launched! SMP: AP CPU #26 Launched! SMP: AP CPU #6 Launched! SMP: AP CPU #2 Launched! SMP: AP CPU #34 Launched! SMP: AP CPU #35 Launched! SMP: AP CPU #45 Launched! SMP: AP CPU #43 Launched! SMP: AP CPU #23 Launched! SMP: AP CPU #42 Launched! SMP: AP CPU #13 Launched! SMP: AP CPU #37 Launched! SMP: AP CPU #20 Launched! SMP: AP CPU #29 Launched! SMP: AP CPU #44 Launched! SMP: AP CPU #21 Launched! SMP: AP CPU #38 Launched! SMP: AP CPU #47 Launched! SMP: AP CPU #22 Launched! SMP: AP CPU #33 Launched! SMP: AP CPU #12 Launched! SMP: AP CPU #30 Launched! SMP: AP CPU #31 Launched! SMP: AP CPU #39 Launched! SMP: AP CPU #32 Launched! Timecounter "TSC-low" frequency 1248582512 Hz quality 1000 random: entropy device external interface kbd1 at kbdmux0 module_register_init: MOD_LOAD (vesa, 0xffffffff80fc74c0, 0) error 19 random: registering fast source Intel Secure Key RNG random: fast provider: "Intel Secure Key RNG" nexus0 cryptosoft0: <software crypto> on motherboard aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard padlock0: No ACE support. acpi0: <HP ProLiant> on motherboard acpi0: Power Button (fixed) atrtc0: <AT realtime clock> port 0x70-0x71,0x74-0x77 irq 8 on acpi0 atrtc0: registered as a time-of-day clock, resolution 1.000000s Event timer "RTC" frequency 32768 Hz quality 0 attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0 Timecounter "i8254" frequency 1193182 Hz quality 0 Event timer "i8254" frequency 1193182 Hz quality 100 hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0 Timecounter "HPET" frequency 14318180 Hz quality 950 Event timer "HPET" frequency 14318180 Hz quality 350 Event timer "HPET1" frequency 14318180 Hz quality 340 Event timer "HPET2" frequency 14318180 Hz quality 340 Event timer "HPET3" frequency 14318180 Hz quality 340 Event timer "HPET4" frequency 14318180 Hz quality 340 Event timer "HPET5" frequency 14318180 Hz quality 340 Event timer "HPET6" frequency 14318180 Hz quality 340 Event timer "HPET7" frequency 14318180 Hz quality 340 Timecounter "ACPI-safe" frequency 3579545 Hz quality 850 acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0 pcib0: <ACPI Host-PCI bridge> on acpi0 pci0: <ACPI PCI bus> on pcib0 pci0: <dasp, performance counters> at device 11.1 (no driver attached) pci0: <dasp, performance counters> at device 11.2 (no driver attached) pci0: <dasp, performance counters> at device 16.1 (no driver attached) pci0: <dasp, performance counters> at device 16.6 (no driver attached) pci0: <dasp, performance counters> at device 18.1 (no driver attached) pci0: <dasp, performance counters> at device 18.5 (no driver attached) pcib1: <ACPI Host-PCI bridge> on acpi0 pci1: <ACPI PCI bus> on pcib1 pci1: <dasp, performance counters> at device 11.1 (no driver attached) pci1: <dasp, performance counters> at device 11.2 (no driver attached) pci1: <dasp, performance counters> at device 16.1 (no driver attached) pci1: <dasp, performance counters> at device 16.6 (no driver attached) pci1: <dasp, performance counters> at device 18.1 (no driver attached) pci1: <dasp, performance counters> at device 18.5 (no driver attached) acpi_syscontainer0: <System Container> on acpi0 acpi_syscontainer1: <System Container> on acpi0 pcib2: <ACPI Host-PCI bridge> port 0xcf8-0xcff numa-domain 0 on acpi0 pcib2: _OSC returned error 0x10 pci2: <ACPI PCI bus> numa-domain 0 on pcib2 pcib3: <ACPI PCI-PCI bridge> at device 1.0 numa-domain 0 on pci2 pci3: <ACPI PCI bus> numa-domain 0 on pcib3 ciss0: <HP Smart Array P440ar> port 0x5000-0x50ff mem 0x93300000-0x933fffff,0x93400000-0x934003ff at device 0.0 numa-domain 0 on pci3 ciss0: PERFORMANT Transport pcib4: <ACPI PCI-PCI bridge> at device 1.1 numa-domain 0 on pci2 pci4: <ACPI PCI bus> numa-domain 0 on pcib4 pcib5: <ACPI PCI-PCI bridge> at device 2.0 numa-domain 0 on pci2 pci5: <ACPI PCI bus> numa-domain 0 on pcib5 ciss1: <HP Smart Array H241> port 0x4000-0x40ff mem 0x93100000-0x931fffff,0x93200000-0x932003ff at device 0.0 numa-domain 0 on pci5 ciss1: PERFORMANT Transport pcib6: <ACPI PCI-PCI bridge> at device 2.1 numa-domain 0 on pci2 pci6: <ACPI PCI bus> numa-domain 0 on pcib6 pcib7: <ACPI PCI-PCI bridge> at device 2.2 numa-domain 0 on pci2 pci7: <ACPI PCI bus> numa-domain 0 on pcib7 ix0: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x3020-0x303f mem 0x92f00000-0x92ffffff,0x93004000-0x93007fff at device 0.0 numa-domain 0 on pci7 ix0: Using MSI-X interrupts with 9 vectors ix0: Ethernet address: 38:ea:a7:8e:06:d4 ix0: PCI Express Bus: Speed 5.0GT/s Width x8 ix1: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x3000-0x301f mem 0x92e00000-0x92efffff,0x93000000-0x93003fff at device 0.1 numa-domain 0 on pci7 ix1: Using MSI-X interrupts with 9 vectors ix1: Ethernet address: 38:ea:a7:8e:06:d5 ix1: PCI Express Bus: Speed 5.0GT/s Width x8 pcib8: <ACPI PCI-PCI bridge> at device 2.3 numa-domain 0 on pci2 pci8: <ACPI PCI bus> numa-domain 0 on pcib8 pcib9: <ACPI PCI-PCI bridge> at device 3.0 numa-domain 0 on pci2 pci9: <ACPI PCI bus> numa-domain 0 on pcib9 ix2: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x2020-0x203f mem 0x92c80000-0x92cfffff,0x92d04000-0x92d07fff at device 0.0 numa-domain 0 on pci9 ix2: Using MSI-X interrupts with 9 vectors ix2: Ethernet address: 00:1b:21:bc:76:1c ix2: PCI Express Bus: Speed 5.0GT/s Width x8 ix3: <Intel(R) PRO/10GbE PCI-Express Network Driver, Version - 3.2.12-k> port 0x2000-0x201f mem 0x92c00000-0x92c7ffff,0x92d00000-0x92d03fff at device 0.1 numa-domain 0 on pci9 ix3: Using MSI-X interrupts with 9 vectors ix3: Ethernet address: 00:1b:21:bc:76:1d ix3: PCI Express Bus: Speed 5.0GT/s Width x8 pcib10: <ACPI PCI-PCI bridge> at device 3.1 numa-domain 0 on pci2 pci10: <ACPI PCI bus> numa-domain 0 on pcib10 pcib11: <ACPI PCI-PCI bridge> at device 3.2 numa-domain 0 on pci2 pci11: <ACPI PCI bus> numa-domain 0 on pcib11 pcib12: <ACPI PCI-PCI bridge> at device 3.3 numa-domain 0 on pci2 pci12: <ACPI PCI bus> numa-domain 0 on pcib12 pci2: <unknown> at device 17.0 (no driver attached) xhci0: <Intel Wellsburg USB 3.0 controller> mem 0x39ffff00000-0x39ffff0ffff at device 20.0 numa-domain 0 on pci2 xhci0: 32 bytes context size, 64-bit DMA usbus0 numa-domain 0 on xhci0 usbus0: 5.0Gbps Super Speed USB v3.0 ehci0: <Intel Wellsburg USB 2.0 controller> mem 0x93501000-0x935013ff at device 26.0 numa-domain 0 on pci2 usbus1: EHCI version 1.0 usbus1 numa-domain 0 on ehci0 usbus1: 480Mbps High Speed USB v2.0 pcib13: <ACPI PCI-PCI bridge> at device 28.0 numa-domain 0 on pci2 pci13: <ACPI PCI bus> numa-domain 0 on pcib13 pcib14: <ACPI PCI-PCI bridge> at device 28.2 numa-domain 0 on pci2 pci14: <ACPI PCI bus> numa-domain 0 on pcib14 vgapci0: <VGA-compatible display> mem 0x91000000-0x91ffffff,0x92a88000-0x92a8bfff,0x92000000-0x927fffff at device 0.1 numa-domain 0 on pci14 vgapci0: Boot video device uhci0: <HP iLO Standard Virtual USB controller> port 0x1300-0x131f at device 0.4 numa-domain 0 on pci14 usbus2 numa-domain 0 on uhci0 usbus2: 12Mbps Full Speed USB v1.0 pcib15: <ACPI PCI-PCI bridge> at device 28.4 numa-domain 0 on pci2 pci15: <ACPI PCI bus> numa-domain 0 on pcib15 bge0: <HP Ethernet 1Gb 4-port 331i Adapter, ASIC rev. 0x5719001> mem 0x92b90000-0x92b9ffff,0x92ba0000-0x92baffff,0x92bb0000-0x92bbffff at device 0.0 numa-domain 0 on pci15 bge0: APE FW version: NCSI v1.4.18.0 bge0: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E miibus0: <MII bus> numa-domain 0 on bge0 brgphy0: <BCM5719C 1000BASE-T media interface> PHY 1 on miibus0 brgphy0: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge0: Using defaults for TSO: 65518/35/2048 bge0: Ethernet address: 14:02:ec:34:7c:a8 bge1: <HP Ethernet 1Gb 4-port 331i Adapter, ASIC rev. 0x5719001> mem 0x92b60000-0x92b6ffff,0x92b70000-0x92b7ffff,0x92b80000-0x92b8ffff at device 0.1 numa-domain 0 on pci15 bge1: APE FW version: NCSI v1.4.18.0 bge1: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E miibus1: <MII bus> numa-domain 0 on bge1 brgphy1: <BCM5719C 1000BASE-T media interface> PHY 2 on miibus1 brgphy1: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge1: Using defaults for TSO: 65518/35/2048 bge1: Ethernet address: 14:02:ec:34:7c:a9 bge2: <HP Ethernet 1Gb 4-port 331i Adapter, ASIC rev. 0x5719001> mem 0x92b30000-0x92b3ffff,0x92b40000-0x92b4ffff,0x92b50000-0x92b5ffff at device 0.2 numa-domain 0 on pci15 bge2: APE FW version: NCSI v1.4.18.0 bge2: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E miibus2: <MII bus> numa-domain 0 on bge2 brgphy2: <BCM5719C 1000BASE-T media interface> PHY 3 on miibus2 brgphy2: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge2: Using defaults for TSO: 65518/35/2048 bge2: Ethernet address: 14:02:ec:34:7c:aa bge3: <HP Ethernet 1Gb 4-port 331i Adapter, ASIC rev. 0x5719001> mem 0x92b00000-0x92b0ffff,0x92b10000-0x92b1ffff,0x92b20000-0x92b2ffff at device 0.3 numa-domain 0 on pci15 bge3: APE FW version: NCSI v1.4.18.0 bge3: CHIP ID 0x05719001; ASIC REV 0x5719; CHIP REV 0x57190; PCI-E miibus3: <MII bus> numa-domain 0 on bge3 brgphy3: <BCM5719C 1000BASE-T media interface> PHY 4 on miibus3 brgphy3: 10baseT, 10baseT-FDX, 100baseTX, 100baseTX-FDX, 1000baseT, 1000baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, auto, auto-flow bge3: Using defaults for TSO: 65518/35/2048 bge3: Ethernet address: 14:02:ec:34:7c:ab pcib16: <ACPI PCI-PCI bridge> at device 28.6 numa-domain 0 on pci2 pci16: <ACPI PCI bus> numa-domain 0 on pcib16 pcib17: <ACPI PCI-PCI bridge> at device 28.7 numa-domain 0 on pci2 pci17: <ACPI PCI bus> numa-domain 0 on pcib17 ehci1: <Intel Wellsburg USB 2.0 controller> mem 0x93500000-0x935003ff at device 29.0 numa-domain 0 on pci2 usbus3: EHCI version 1.0 usbus3 numa-domain 0 on ehci1 usbus3: 480Mbps High Speed USB v2.0 isab0: <PCI-ISA bridge> at device 31.0 numa-domain 0 on pci2 isa0: <ISA bus> numa-domain 0 on isab0 pcib18: <ACPI Host-PCI bridge> numa-domain 1 on acpi0 pcib18: _OSC returned error 0x10 pci18: <ACPI PCI bus> numa-domain 1 on pcib18 pcib19: <ACPI PCI-PCI bridge> at device 0.0 numa-domain 1 on pci18 pci19: <ACPI PCI bus> numa-domain 1 on pcib19 pcib20: <ACPI PCI-PCI bridge> at device 1.0 numa-domain 1 on pci18 pci20: <ACPI PCI bus> numa-domain 1 on pcib20 pcib21: <ACPI PCI-PCI bridge> at device 1.1 numa-domain 1 on pci18 pci21: <ACPI PCI bus> numa-domain 1 on pcib21 pcib22: <ACPI PCI-PCI bridge> at device 2.0 numa-domain 1 on pci18 pci22: <ACPI PCI bus> numa-domain 1 on pcib22 pcib23: <ACPI PCI-PCI bridge> at device 2.1 numa-domain 1 on pci18 pci23: <ACPI PCI bus> numa-domain 1 on pcib23 pcib24: <ACPI PCI-PCI bridge> at device 2.2 numa-domain 1 on pci18 pci24: <ACPI PCI bus> numa-domain 1 on pcib24 pcib25: <ACPI PCI-PCI bridge> at device 2.3 numa-domain 1 on pci18 pci25: <ACPI PCI bus> numa-domain 1 on pcib25 pcib26: <ACPI PCI-PCI bridge> at device 3.0 numa-domain 1 on pci18 pci26: <ACPI PCI bus> numa-domain 1 on pcib26 pcib27: <ACPI PCI-PCI bridge> at device 3.1 numa-domain 1 on pci18 pci27: <ACPI PCI bus> numa-domain 1 on pcib27 pcib28: <ACPI PCI-PCI bridge> at device 3.2 numa-domain 1 on pci18 pci28: <ACPI PCI bus> numa-domain 1 on pcib28 pcib29: <ACPI PCI-PCI bridge> at device 3.3 numa-domain 1 on pci18 pci29: <ACPI PCI bus> numa-domain 1 on pcib29 uart0: <Non-standard ns8250 class UART with FIFOs> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0 uart1: <Non-standard ns8250 class UART with FIFOs> port 0x2f8-0x2ff irq 3 on acpi0 ichwd0: <Intel Wellsburg watchdog timer> on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 ichwd0: <Intel Wellsburg watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0 atkbd0: <AT Keyboard> irq 1 on atkbdc0 kbd0 at atkbd0 atkbd0: [GIANT-LOCKED] ZFS filesystem version: 5 ZFS storage pool version: features support (5000) Timecounters tick every 1.000 msec freenas_sysctl: adding account. freenas_sysctl: adding directoryservice. freenas_sysctl: adding middlewared. freenas_sysctl: adding network. freenas_sysctl: adding services. ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled ugen2.1: <HP UHCI root HUB> at usbus2 ugen0.1: <0x8086 XHCI root HUB> at usbus0 ugen3.1: <Intel EHCI root HUB> at usbus3 ugen1.1: <Intel EHCI root HUB> at usbus1 uhub0: <HP UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2 uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3 uhub2: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0 uhub3: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1 uhub0: 2 ports with 2 removable, self powered uhub1: 2 ports with 2 removable, self powered uhub3: 2 ports with 2 removable, self powered uhub2: 21 ports with 21 removable, self powered ugen2.2: <BMC Virtual Keyboard> at usbus2 ukbd0 numa-domain 0 on uhub0 ukbd0: <Virtual Keyboard > on usbus2 kbd2 at ukbd0 ugen0.2: <vendor 0x0424 product 0x2660> at usbus0 uhub4 numa-domain 0 on uhub2 uhub4: <vendor 0x0424 product 0x2660, class 9/0, rev 2.00/8.01, addr 1> on usbus0 ugen3.2: <vendor 0x8087 product 0x8002> at usbus3 uhub5 numa-domain 0 on uhub1 uhub5: <vendor 0x8087 product 0x8002, class 9/0, rev 2.00/0.05, addr 2> on usbus3 ugen1.2: <vendor 0x8087 product 0x800a> at usbus1 uhub6 numa-domain 0 on uhub3 uhub6: <vendor 0x8087 product 0x800a, class 9/0, rev 2.00/0.05, addr 2> on usbus1 uhub4: 2 ports with 1 removable, self powered uhub5: 8 ports with 8 removable, self powered uhub6: 6 ports with 6 removable, self powered da0 at ciss0 bus 0 scbus0 target 0 lun 0 da1 at ciss1 bus 32 scbus3 target 16 lun 0 da2 at ciss1 bus 32 scbus3 target 17 lun 0 da3 at ciss1 bus 32 scbus3 target 18 lun 0 da4 at ciss1 bus 32 scbus3 target 19 lun 0 da5 at ciss1 bus 32 scbus3 target 20 lun 0 da6 at ciss1 bus 32 scbus3 target 21 lun 0 da7 at ciss1 bus 32 scbus3 target 22 lun 0 da8 at ciss1 bus 32 scbus3 target 23 lun 0 da0: <HP RAID 1(1+0) OK> Fixed Direct Access SPC-3 SCSI device da0: Serial Number PDNLH0BRH9OAXK da0: 135.168MB/s transfersda8: da5: da7: da2: da3: da4: <HP MO0800JEFPB HPD3> Fixed Direct Access SPC-4 SCSI device da1: <HP MO0800JEFPB HPD3> Fixed Direct Access SPC-4 SCSI device da6: <HP MO0800JDVEV HPD2> Fixed Direct Access SPC-4 SCSI device <HP MO0800JEFPB HPD3> Fixed Direct Access SPC-4 SCSI device da7: Serial Number 0RY26W5A <HP MO0800JDVEV HPD2> Fixed Direct Access SPC-4 SCSI device <HP MO0800JEFPB HPD3> Fixed Direct Access SPC-4 SCSI device da8: Serial Number 0RY26BSA da2: Serial Number 2MVGZL3A da1: Serial Number 2MVJNY3A da8: 135.168MB/s transfersda0: Command Queueing enabled da1: 135.168MB/s transfers <HP MO0800JEFPB HPD3> Fixed Direct Access SPC-4 SCSI device da4: Serial Number 0RY26V6A <HP MO0800JEFPB HPD3> Fixed Direct Access SPC-4 SCSI device da7: 135.168MB/s transfersda5: Serial Number 0RY26ZUA da3: Serial Number 0RY22Z0A da6: Serial Number 0RY26V5A da5: 135.168MB/s transfersda3: 135.168MB/s transfersda2: 135.168MB/s transfersda4: 135.168MB/s transfers da1: Command Queueing enabled da8: Command Queueing enabled da0: 476908MB (976707632 512 byte sectors) da3: Command Queueing enabled da6: 135.168MB/s transfersda2: Command Queueing enabled da3: 763097MB (1562824368 512 byte sectors) da4: Command Queueing enabled da8: 763097MB (1562824368 512 byte sectors) da2: 763097MB (1562824368 512 byte sectors) da19 at ciss1 bus 32 scbus3 target 54 lun 0 da4: 763097MB (1562824368 512 byte sectors) da18 at ciss1 bus 32 scbus3 target 53 lun 0 da24 at ciss1 bus 32 scbus3 target 59 lun 0 da19: da6: Command Queueing enabled da24: da18: da1: 763097MB (1562824368 512 byte sectors) da7: Command Queueing enabled <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da14 at ciss1 bus 32 scbus3 target 46 lun 0 <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da7: 763097MB (1562824368 512 byte sectors) da6: 763097MB (1562824368 512 byte sectors) da5: Command Queueing enabled da14: da18: Serial Number Z4D2K3SS da24: Serial Number Z4D2K450 da19: Serial Number Z4D2KPSV <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da19: 135.168MB/s transfersda24: 135.168MB/s transfersda26 at ciss1 bus 32 scbus3 target 61 lun 0 da16 at ciss1 bus 32 scbus3 target 48 lun 0 da14: Serial Number Z4D2K4A4 da5: 763097MB (1562824368 512 byte sectors) da17 at ciss1 bus 32 scbus3 target 49 lun 0 da16: da14: 135.168MB/s transfersda26: da17: da21 at ciss1 bus 32 scbus3 target 56 lun 0 <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da10 at ciss1 bus 32 scbus3 target 42 lun 0 da18: 135.168MB/s transfersda19: Command Queueing enabled da24: Command Queueing enabled <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da26: Serial Number Z4D2L9PS da21: da16: Serial Number Z4D2L9QT da14: Command Queueing enabled da10: da16: 135.168MB/s transfers<HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da24: 5723166MB (11721045168 512 byte sectors) da26: 135.168MB/s transfersda18: Command Queueing enabled da9 at ciss1 bus 32 scbus3 target 41 lun 0 da21: Serial Number Z4D2K46X <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da17: Serial Number Z4D2KFC0 da16: Command Queueing enabled da14: 5723166MB (11721045168 512 byte sectors) da19: 5723166MB (11721045168 512 byte sectors) da17: 135.168MB/s transfersda16: 5723166MB (11721045168 512 byte sectors) da9: da13 at ciss1 bus 32 scbus3 target 45 lun 0 da18: 5723166MB (11721045168 512 byte sectors) da22 at ciss1 bus 32 scbus3 target 57 lun 0 da13: da21: 135.168MB/s transfersda26: Command Queueing enabled <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da12 at ciss1 bus 32 scbus3 target 44 lun 0 da10: Serial Number Z4D2GJDV <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da26: 5723166MB (11721045168 512 byte sectors) da9: Serial Number Z4D2KYAZ da21: Command Queueing enabled da11 at ciss1 bus 32 scbus3 target 43 lun 0 da10: 135.168MB/s transfersda12: da9: 135.168MB/s transfersda21: 5723166MB (11721045168 512 byte sectors) da17: Command Queueing enabled da11: da13: Serial Number Z4D2K4BD da20 at ciss1 bus 32 scbus3 target 55 lun 0 da22: da13: 135.168MB/s transfersda15 at ciss1 bus 32 scbus3 target 47 lun 0 da20: da10: Command Queueing enabled <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da17: 5723166MB (11721045168 512 byte sectors) <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da15: da11: Serial Number Z4D2K3ZJ da13: Command Queueing enabled da22: Serial Number Z4D2KQ9A da12: Serial Number Z4D2KPSX da20: Serial Number Z4D2K43E da9: Command Queueing enabled da13: 5723166MB (11721045168 512 byte sectors) da22: 135.168MB/s transfersda20: 135.168MB/s transfers<HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da23 at ciss1 bus 32 scbus3 target 58 lun 0 da10: 5723166MB (11721045168 512 byte sectors) da15: Serial Number Z4D2KQ4V da9: 5723166MB (11721045168 512 byte sectors) da23: da12: 135.168MB/s transfersda20: Command Queueing enabled da22: Command Queueing enabled <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da20: 5723166MB (11721045168 512 byte sectors) da15: 135.168MB/s transfersda11: 135.168MB/s transfersda22: 5723166MB (11721045168 512 byte sectors) da12: Command Queueing enabled da23: Serial Number Z4D2K45B da25 at ciss1 bus 32 scbus3 target 60 lun 0 da12: 5723166MB (11721045168 512 byte sectors) da11: Command Queueing enabled da25: da23: 135.168MB/s transfersda11: 5723166MB (11721045168 512 byte sectors) da15: Command Queueing enabled <HP MB6000JEFND HPD9> Fixed Direct Access SPC-4 SCSI device da25: Serial Number Z4D2K49W random: unblocking device. da15: 5723166MB (11721045168 512 byte sectors) da23: Command Queueing enabled da25: 135.168MB/s transfersda23: 5723166MB (11721045168 512 byte sectors) da25: Command Queueing enabled da25: 5723166MB (11721045168 512 byte sectors) Trying to mount root from zfs:freenas-boot/ROOT/11.1-U5 []... ipmi0: <IPMI System Interface> port 0xca2-0xca3 on acpi0 ipmi0: KCS mode found at io 0xca2 on acpi ipmi0: IPMI device rev. 2, firmware rev. 2.55, version 2.0 ipmi0: Number of channels 1 ipmi0: Attached watchdog ichwd0: <Intel Wellsburg watchdog timer> at port 0x430-0x437,0x460-0x47f on isa0 ichwd0: ICH WDT present but disabled in BIOS or hardware device_attach: ichwd0 attach returned 6 GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7) hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> bge0: link state changed to DOWN lagg0: link state changed to DOWN lagg1: link state changed to DOWN ix1: link state changed to UP lagg1: link state changed to UP ix0: link state changed to UP lagg0: link state changed to UP ix2: link state changed to UP ix3: link state changed to UP bge0: link state changed to UP ums0 numa-domain 0 on uhub0 ums0: <Virtual Mouse> on usbus2 ix3: link state changed to DOWN ix3: link state changed to UP ix3: link state changed to DOWN ix2: link state changed to DOWN ix2: link state changed to UP ix2: link state changed to DOWN ix2: link state changed to UP ix3: link state changed to UP ugen2.2: <BMC Virtual Keyboard> at usbus2 (disconnected) ukbd0: at uhub0, port 1, addr 2 (disconnected) ukbd0: detached ums0: at uhub0, port 1, addr 2 (disconnected) ums0: detached
Interestingly, only ONE of the disks that iLO lists as pending failure seems to be showing up in FreeNAS, however FreeNAS shows that 2 sequential devices are apparently somehow failing?
Are we missing something here? I'm aware of the fact that the H241 may not be ideal, and if the answer is buy a 9300-8e and be done with it, then fine. What I'm hoping for is some diagnostic direction to take to ensure that before we spend more money on extra hardware, we determine whether this is a daisy chain limitation imposed by the D3600/D3700 arrays, or whether this is down to the fact that FreeNAS/FreeBSD treats the H241 card like a RAID controller, even though it seems to know enough to present the disks as direct-attached SCSI. We've run 3-DAS setups like this for quite a while now, without issue.
Firmware on the DAS's have been updated to latest available, as well as on the H241 card.
Thanks for any help - much appreciated.
