Disks not recognized

[mgab]

Hello, thanks in advance for reading and helping if possible.

I started building a FreeNAS server for our lab users last week using the following hardware found in the lab:
- SuperMicro X11SCA-F-O Motherboard
- Ballistix Sport LT 32GB DDR4 2400Mhz RAM (2x16GB)
- 3 x SYBA SI-PEX40071 (8 x SATAIII multiplier PCIe... I'm pretty sure this is where the no no is, since it is stated in the Hardware Reqs documentation not to use multipliers)
- Intel Core i3 8100 Coffee Lake
- Norco RPC-4224 4U Rackmount server case 24 SAS/SATA bays
- 24 x 2TB 7200RPM Seagate SAS HDDs

Now part of the reason I am building this is because we scavenged a whole bunch of perfectly good HDDs and the rest was lying around basically. I know that the SATA multipliers are not to be used with FreeNAS, but from the output of dmesg I can still see the Marvell controllers and everything. Is it completely impossible to get this exact setup running with the full 24 disks (using FreeNAS or not), or will I have to get something like the following + 6 SFF to SFF cables:
https://www.newegg.ca/supermicro-aoc-sas2lp-mv8-sata-sas/p/N82E16816101792?Description=SFF SFF&cm_re=SFF_SFF-_-16-101-792-_-Product

I will still attach the output of dmesg here:

Code:

Copyright (c) 1992-2018 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.2-STABLE #0 r325575+95cc58ca2a0(HEAD): Fri May 10 15:57:35 EDT 2019
    root@mp20.tn.ixsystems.com:/freenas-releng/freenas/_BE/objs/freenas-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 6.0.0 (tags/RELEASE_600/final 326565) (based on LLVM 6.0.0)
VT(vga): resolution 640x480
CPU: Intel(R) Core(TM) i3-8100 CPU @ 3.60GHz (3600.29-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x906eb  Family=0x6  Model=0x9e  Stepping=11
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbbf<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,EST,TM2,SSSE3,SDBG,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x121<LAHF,ABM,Prefetch>
  Structured Extended Features=0x29c67af<FSGSBASE,TSCADJ,SGX,BMI1,AVX2,SMEP,BMI2,ERMS,INVPCID,NFPUSG,MPX,RDSEED,ADX,SMAP,CLFLUSHOPT,PROCTRACE>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,SSBD>
  XSAVE Features=0xf<XSAVEOPT,XSAVEC,XINUSE,XSAVES>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33130684416 (31595 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: < >
FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s)
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-119 on motherboard
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
Timecounter "TSC-low" frequency 1800143426 Hz quality 1000
random: entropy device external interface
random: registering fast source Intel Secure Key RNG
random: fast provider: "Intel Secure Key RNG"
kbd1 at kbdmux0
nexus0
vtvga0: <VT VGA driver> on motherboard
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS,AES-GCM,AES-ICM> on motherboard
padlock0: No ACE support.
acpi0: <SUPERM SUPERM> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 24000000 Hz quality 950
Event timer "HPET" frequency 24000000 Hz quality 550
Event timer "HPET1" frequency 24000000 Hz quality 440
Event timer "HPET2" frequency 24000000 Hz quality 440
Event timer "HPET3" frequency 24000000 Hz quality 440
Event timer "HPET4" frequency 24000000 Hz quality 440
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x1808-0x180b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
ahci0: <Marvell 88SE9235 AHCI SATA controller> port 0x7050-0x7057,0x7040-0x7043,0x7030-0x7037,0x7020-0x7023,0x7000-0x701f mem 0xa2540000-0xa25407ff irq 16 at device 0.0 on pci1
ahci0: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
ahci1: <Marvell 88SE9235 AHCI SATA controller> port 0x6050-0x6057,0x6040-0x6043,0x6030-0x6037,0x6020-0x6023,0x6000-0x601f mem 0xa2440000-0xa24407ff irq 17 at device 0.0 on pci2
ahci1: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
ahcich4: <AHCI channel> at channel 0 on ahci1
ahcich5: <AHCI channel> at channel 1 on ahci1
ahcich6: <AHCI channel> at channel 2 on ahci1
ahcich7: <AHCI channel> at channel 3 on ahci1
xhci0: <XHCI (generic) USB 3.0 controller> mem 0xa2620000-0xa262ffff irq 16 at device 20.0 on pci0
xhci0: 32 bytes context size, 64-bit DMA
usbus0 on xhci0
usbus0: 5.0Gbps Super Speed USB v3.0
pci0: <memory, RAM> at device 20.2 (no driver attached)
pci0: <serial bus> at device 21.0 (no driver attached)
pci0: <serial bus> at device 21.1 (no driver attached)
pci0: <simple comms> at device 22.0 (no driver attached)
ahci2: <AHCI SATA controller> port 0x8050-0x8057,0x8040-0x8043,0x8020-0x803f mem 0xa2634000-0xa2635fff,0xa263b000-0xa263b0ff,0xa263a000-0xa263a7ff irq 16 at device 23.0 on pci0
ahci2: AHCI v1.31 with 8 6Gbps ports, Port Multiplier not supported
ahcich8: <AHCI channel> at channel 0 on ahci2
ahcich9: <AHCI channel> at channel 1 on ahci2
ahcich10: <AHCI channel> at channel 2 on ahci2
ahcich11: <AHCI channel> at channel 3 on ahci2
ahcich12: <AHCI channel> at channel 4 on ahci2
ahcich13: <AHCI channel> at channel 5 on ahci2
ahcich14: <AHCI channel> at channel 6 on ahci2
ahcich15: <AHCI channel> at channel 7 on ahci2
pcib3: <ACPI PCI-PCI bridge> at device 27.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 27.4 on pci0
pci4: <ACPI PCI bus> on pcib4
ahci3: <Marvell 88SE9235 AHCI SATA controller> port 0x5050-0x5057,0x5040-0x5043,0x5030-0x5037,0x5020-0x5023,0x5000-0x501f mem 0xa2340000-0xa23407ff irq 16 at device 0.0 on pci4
ahci3: AHCI v1.00 with 4 6Gbps ports, Port Multiplier supported with FBS
ahcich16: <AHCI channel> at channel 0 on ahci3
ahcich17: <AHCI channel> at channel 1 on ahci3
ahcich18: <AHCI channel> at channel 2 on ahci3
ahcich19: <AHCI channel> at channel 3 on ahci3
pcib5: <ACPI PCI-PCI bridge> at device 28.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 28.5 on pci0
pci6: <ACPI PCI bus> on pcib6
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0x4000-0x401f mem 0xa2200000-0xa227ffff,0xa2280000-0xa2283fff irq 17 at device 0.0 on pci6
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: ac:1f:6b:ad:83:0f
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib7: <ACPI PCI-PCI bridge> at device 28.6 on pci0
pci7: <ACPI PCI bus> on pcib7
pcib8: <ACPI PCI-PCI bridge> irq 18 at device 0.0 on pci7
pci8: <ACPI PCI bus> on pcib8
vgapci0: <VGA-compatible display> port 0x3000-0x307f mem 0xa1000000-0xa1ffffff,0xa2000000-0xa201ffff irq 18 at device 0.0 on pci8
vgapci0: Boot video device
pcib9: <ACPI PCI-PCI bridge> at device 28.7 on pci0
pci9: <ACPI PCI bus> on pcib9
pcib10: <ACPI PCI-PCI bridge> at device 0.0 on pci9
pci10: <ACPI PCI bus> on pcib10
pci0: <simple comms> at device 30.0 (no driver attached)
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
pci0: <multimedia, HDA> at device 31.3 (no driver attached)
pci0: <serial bus> at device 31.5 (no driver attached)
em0: <Intel(R) PRO/1000 Network Connection 7.6.1-k> mem 0xa2600000-0xa261ffff irq 16 at device 31.6 on pci0
em0: Using an MSI interrupt
em0: Ethernet address: ac:1f:6b:ad:83:0e
acpi_button0: <Sleep Button> on acpi0
acpi_button1: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
uart2: <16550 or compatible> port 0x248-0x24f irq 10 on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
acpi_syscontainer0: <System Container> on acpi0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff on isa0
atrtc0: <AT realtime clock> at port 0x70 irq 8 on isa0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ipmi0: IPMI device rev. 1, firmware rev. 1.12, version 2.0
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ugen0.1: <0x8086 XHCI root HUB> at usbus0
uhub0: <0x8086 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus0
uhub0: 26 ports with 26 removable, self powered
ugen0.2: <vendor 0x0557 product 0x7000> at usbus0
uhub1 on uhub0
uhub1: <vendor 0x0557 product 0x7000, class 9/0, rev 2.00/0.00, addr 1> on usbus0
uhub1: 4 ports with 3 removable, self powered
ugen0.3: <vendor 0x0557 product 0x2419> at usbus0
ukbd0 on uhub1
ukbd0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0
kbd2 at ukbd0
ahcich7: AHCI reset: device not ready after 31000ms (tfd = 00000080)
ahcich3: pmp0 at ahcich7 bus 0 scbus7 target 15 lun 0
AHCI reset: device not ready after 31000ms (tfd = 00000080)
pmp0: <Port Multiplier 97051b4b a00e> ATA device
pmp0: 600.000MB/s transfers (SATA 3.x, NONE, PIO 8192bytes)
pmp0: 5 fan-out ports
pmp1 at ahcich3 bus 0 scbus3 target 15 lun 0
pmp1: <Port Multiplier 97051b4b a00e> ATA device
pmp1: 600.000MB/s transfers (SATA 3.x, NONE, PIO 8192bytes)
pmp1: 5 fan-out ports
ahcich19: AHCI reset: device not ready after 31000ms (tfd = 00000080)
pmp2 at ahcich19 bus 0 scbus19 target 15 lun 0
pmp2: <Port Multiplier 97051b4b a00e> ATA device
pmp2: 600.000MB/s transfers (SATA 3.x, NONE, PIO 8192bytes)
pmp2: 5 fan-out ports
ada0 at ahcich8 bus 0 scbus8 target 0 lun 0
ada0: <KINGSTON SA400S37120G 03090004> ACS-3 ATA SATA 3.x device
ada0: Serial Number 50026B72822CB245
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors)
ada1 at ahcich9 bus 0 scbus9 target 0 lun 0
ada1: <KINGSTON SA400S37120G 03090004> ACS-3 ATA SATA 3.x device
ada1: Serial Number 50026B72822CD2F4
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 512bytes)
ada1: Command Queueing enabled
ada1: 114473MB (234441648 512 byte sectors)
random: unblocking device.
Trying to mount root from zfs:freenas-boot/ROOT/default []...
pmc: Unknown Intel CPU.
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI>
lagg0: link state changed to DOWN
ums0 on uhub1
ums0: <vendor 0x0557 product 0x2419, class 0/0, rev 1.10/1.00, addr 2> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
em0: link state changed to UP
lagg0: link state changed to UP
igb0: link state changed to UP

 

[tfran1990]

E bay.
get one of the many listed lsi HBA's(lsi 9211 hp220 dellM1015) with an IBM 46M0997 to get 24 drives
Move the drive around to a different cable and connector, if the drive is not bad then it more then likely is the SYBA SI-PEX4007,sometime they work other times they work for a short amount of time.

 

[Chris Moore]

SATAIII multiplier PCIe... I'm pretty sure this is where the no no is, since it is stated in the Hardware Reqs documentation not to use multipliers

SATA port multipliers are known to not work reliably, and I am sure there must have been some mention of that, somewhere.

- Norco RPC-4224 4U Rackmount server case 24 SAS/SATA bays

It is a real shame you bought one of those. It is not the value option it appears to be.

Something like this would have been much better:
https://www.ebay.com/itm/Supermicro-4U-SAS2-Storage-2x-E5-2620-16GB-LSI-9265-8I-Rails/383001589533
Just change out the hardware RAID controller to a SAS HBA like this:
https://www.ebay.com/itm/HP-H220-6G...0-IT-Mode-for-ZFS-FreeNAS-unRAID/162862201664
and start plugging in your drives.
If I recall correctly, that server can be upgraded to 1TB of RAM using Registered ECC memory and you can upgrade the processors if you need more horsepower. It is a real workhorse.

 

[Chris Moore]

- 24 x 2TB 7200RPM Seagate SAS HDDs

Here is a problem... What is the exact model number of those drives? You say they are SAS drives, but no model number to verify...

A SAS HBA can control a SATA hard drive but,

- 3 x SYBA SI-PEX40071 (8 x SATAIII multiplier PCIe... I'm pretty sure this is where the no no is, since it is stated in the Hardware Reqs documentation not to use multipliers)

These are SATA controllers and a SATA controller can not run a SAS drive.

 

[mgab]

Here is a problem... What is the exact model number of those drives? You say they are SAS drives, but no model number to verify...

A SAS HBA can control a SATA hard drive but,

These are SATA controllers and a SATA controller can not run a SAS drive.

Thanks for the quick answers @Chris Moore, this makes much sense. I am currently not at the lab, I will check about the drives to make 100% sure when I get back there. If they are in fact SAS drives, and knowing that the Norco case has 6 x SFF-8087, would I simply be able to get an IBM 46M0997 like the one described by @tfran1990 and simply plug all 6 connectors there?

 

[Chris Moore]

knowing that the Norco case has 6 x SFF-8087, would I simply be able to get an IBM 46M0997 like the one described

https://www.ebay.com/itm/IBM-ServeR...on-Adapter-46M0997-Firmware-634A/163321588238
This is a great device. I have two of them myself. The thing to understand is that they are a SAS Expander that needs to be connected to a SAS Controller. So you would need one expander and one controller like this:
https://www.ebay.com/itm/Dell-H310-...e-Low-Profile-ZFS-FreeNAS-unRAID/162903881137
or this:
https://www.ebay.com/itm/Lenovo-924...A-P20-IT-Mode-ZFS-FreeNAS-unRAID/163600383999

The data path would be from the controller to the expander then to the drives. The expander module works much like a network switch for adding more connectivity to a SAS controller.

Here is a video that discusses a method of using that SAS expander with a SAS controller to connect up to 24 drives:
https://www.youtube.com/watch?v=qccpopxc_Uo

 

[Chris Moore]

PS. You might need one or two of these:
https://www.ebay.com/itm/miniSAS-IP...ch-33-cm-host-to-backplane-cable/163288762214

 

[mgab]

https://www.ebay.com/itm/IBM-ServeR...on-Adapter-46M0997-Firmware-634A/163321588238
This is a great device. I have two of them myself. The thing to understand is that they are a SAS Expander that needs to be connected to a SAS Controller. So you would need one expander and one controller like this:
https://www.ebay.com/itm/Dell-H310-...e-Low-Profile-ZFS-FreeNAS-unRAID/162903881137
or this:
https://www.ebay.com/itm/Lenovo-924...A-P20-IT-Mode-ZFS-FreeNAS-unRAID/163600383999

The data path would be from the controller to the expander then to the drives. The expander module works much like a network switch for adding more connectivity to a SAS controller.

Here is a video that discusses a method of using that SAS expander with a SAS controller to connect up to 24 drives:
https://www.youtube.com/watch?v=qccpopxc_Uo

This is really cool, and very helpful, although I think it is worth it to note that the Norco case has 6 x SFF-8087 connectors that regroup 4 drives each. So I am wondering, I could probably plug them in the IBM expander and in turn plug into the controller?

 

[Chris Moore]

, I could probably plug them in the IBM expander and in turn plug into the controller?

Did you watch the video I linked you to? It fully explains the process.

 

[mgab]

Did you watch the video I linked you to? It fully explains the process.

Yup, thank you very much, this is most probably exactly what I will do.
Cheers!

 

[mgab]

Did you watch the video I linked you to? It fully explains the process.

So I finished this build using the Dell H310 controller (had to change the bracket for full profile, had a couple of spares in the lab), the IBM expander, connected as intended, did cable management, booted everything up and voilà! System boots up like a charm, all drives are detected and part of a pool now, RW speeds are not bad at all, configured LACP link aggregation, all shares are up and I am happy :)

Thanks for the suggestions guys, they were right on !



Cheers,
Mike