Unable to see my 25 disks, only shows 14

[alisonsilveira]

Hi Everyone,

Sorry if I missed something, this is my first post here.
First of all, see below my configuration.

HP ProLiant DL 385p Gen8
2x AMD Opteron 6380
128 GB RAM
HP Smart Array P420i - Disabled
LSI SAS9340-8i in IT mode
2x SSD 240GB Seagate XA240ME10003 - Will be used for logs
25x HDD 1TB Seagate ST1000VT001 2.5" - on a 25-SFF drive bays

I've installed TrueNAS Core on this server, but when I check the disks are just showing 14 (sometimes 15) disks and if I run "mprutil" for example I can see all disks. I also tried to install UNRAID just to check if I can see all disks and I was able to see all 25 disks.
Can someone help me if I'm missing something or some steps that I should have been done?

Thank you,

 

[jgreco]

Sometimes disks that already contain a partition table, such as from a previously installed OS, do not show up correctly (or at all). This would especially make sense if it had previously been used with a Linux OS.

You can try erasing the partition table on a "problem" hard disk to test this theory.

From the CLI, "camcontrol devlist" will give you the list of target devices.

Code:

<VMware Virtual disk 1.0>          at scbus2 target 2 lun 0 (pass3,da2)
<LSI SAS2X36 0e0b>                 at scbus3 target 20 lun 0 (pass4,ses0)
<ATA WDC WD140EDFZ-11 0A81>        at scbus3 target 96 lun 0 (pass5,da3)
<ATA WDC WD140EDFZ-11 0A81>        at scbus3 target 97 lun 0 (pass6,da4)

In this example, da3 and da4 are the real hard drives, accessed as /dev/da3 and /dev/da4

Compare the camcontrol devlist list to what the GUI is displaying, and I suspect you'll find all your disks on the camcontrol list. See if you can identify one that isn't in the GUI, and then from the command line, do

% sysctl -w kern.geom.debugflags=16 (possibly not necessary)
% dd if=/dev/zero of=/dev/da3 bs=512 count=65536 (definitely excessive but helpful for disk health testing)

which will write zeroes to the first 65536 sectors of da3.

If the drive then shows up after a reboot, I'm guessing partitioning crud is making FreeBSD unhappy, which can and does happen. You can either repeat the dd for each drive to wipe all their partitions, or you can run a tool like DBAN in zero-only mode to do it all at once.

 

[HoneyBadger]

Welcome. A quick side note/question about performance intentions.

2x SSD 240GB Seagate XA240ME10003 - Will be used for logs

Is this "logs" as in SLOG devices, or "logs" just being other data?

I'm not 100% on whether they have power-loss-prevention for in-flight data, or if it's similar to Crucial where "existing data is protected" - if it's the latter, you'll likely have poor SLOG performance.

What's the desired use case? Based on the "lots of small spindles" I'm expecting something involving databases/small files/block storage.

 

[Alecmascot]

Not a direct bearing on your issue, but from what I can find the ST1000VT001 is a shingled drive and is not suitable for ZFS.

 

[HoneyBadger]

Not a direct bearing on your issue, but from what I can find the ST1000VT001 is a shingled drive and is not suitable for ZFS.

I can't find a datasheet that lists that model specifically (will keep digging) but if it is SMR (128MB cache on that model suggests it might be) then this is spot on and the drives will mostly likely need to be changed.

Earlier "Seagate Video" drives weren't SMR but they all had model numbers ending in VT000.

Edit: Badger claws have unearthed a datasheet, see attached. SMR is confirmed - these drives will not be a good fit for a ZFS solution.

 

[alisonsilveira]

Thanks guys for your help.
"camcontrol devlist" doesn't show all disks as "mprutil show devices" shows. I tried to use "egrep 'da[0-9]|cd[0-9]' /var/run/dmesg.boot" see the return below:

Code:

ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfccd0000-0xfccd0fff irq 22 at device 18.1 on pci0
ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
ada0: <XA240ME10003 SF42147L> ACS-4 ATA SATA 3.x device
ada0: Serial Number HLH02CT9
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 228936MB (468862128 512 byte sectors)
ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
ada1: <XA240ME10003 SF42147L> ACS-4 ATA SATA 3.x device
ada1: Serial Number HLH02CN4
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 228936MB (468862128 512 byte sectors)
ada2 at ahcich2 bus 0 scbus3 target 0 lun 0
ada2: <KINGSTON SA400S37120G S3700100> ACS-3 ATA SATA 3.x device
ada2: Serial Number 50026B7783679ED7
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ada2: Command Queueing enabled
ada2: 114473MB (234441648 512 byte sectors)
ses0: da0 in Array Device Slot 1, SAS Slot: 1+ phys at slot 14
ses0: da1 in Array Device Slot 2, SAS Slot: 1+ phys at slot 15
ses0: da2,pass4 in Array Device Slot 3, SAS Slot: 1+ phys at slot 16
ses0: da3,pass5 in Array Device Slot 4, SAS Slot: 1+ phys at slot 17
ses0: da4,pass6 in Array Device Slot 5, SAS Slot: 1+ phys at slot 18
ses1: da11,pass13 in Array Device Slot 12, SAS Slot: 1+ phys at slot 12
ses0: da5,pass7 in Array Device Slot 6, SAS Slot: 1+ phys at slot 19
ses1: da13,pass17 in Array Device Slot 13, SAS Slot: 1+ phys at slot 13
ses0: da6,pass8 in Array Device Slot 7, SAS Slot: 1+ phys at slot 20
ses0: da7,pass9 in Array Device Slot 8, SAS Slot: 1+ phys at slot 21
ses0: da8,pass10 in Array Device Slot 9, SAS Slot: 1+ phys at slot 22
ses0: da9,pass11 in Array Device Slot 10, SAS Slot: 1+ phys at slot 23
ses0: da10,pass12 in Array Device Slot 11, SAS Slot: 1+ phys at slot 24
ses0: da12,pass16 in Array Device Slot 12, SAS Slot: 1+ phys at slot 25
da6 at mpr0 bus 0 scbus2 target 6 lun 0
da6: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da6: Serial Number WL1M7TM5
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 953869MB (1953525168 512 byte sectors)
da11 at mpr0 bus 0 scbus2 target 11 lun 0
da11: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da11: Serial Number WL1MPAL6
da11: 600.000MB/s transfers
da11: Command Queueing enabled
da11: 953869MB (1953525168 512 byte sectors)
da9 at mpr0 bus 0 scbus2 target 9 lun 0
da9: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da9: Serial Number WL1MNXH6
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 953869MB (1953525168 512 byte sectors)
da1 at mpr0 bus 0 scbus2 target 1 lun 0
da1: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number WL1MSAZX
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 953869MB (1953525168 512 byte sectors)
da7 at mpr0 bus 0 scbus2 target 7 lun 0
da7: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da7: Serial Number WL1MSCK8
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 953869MB (1953525168 512 byte sectors)
da4 at mpr0 bus 0 scbus2 target 4 lun 0
da4: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da4: Serial Number WL1MDENT
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 953869MB (1953525168 512 byte sectors)
da3 at mpr0 bus 0 scbus2 target 3 lun 0
da3: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number WL1MSBYD
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 953869MB (1953525168 512 byte sectors)
da2 at mpr0 bus 0 scbus2 target 2 lun 0
da2: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number WL1MSEA5
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 953869MB (1953525168 512 byte sectors)
da5 at mpr0 bus 0 scbus2 target 5 lun 0
da5: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number WL1HH2FA
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 953869MB (1953525168 512 byte sectors)
da10 at mpr0 bus 0 scbus2 target 10 lun 0
da10: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da10: Serial Number WL1M7X6D
da10: 600.000MB/s transfers
da10: Command Queueing enabled
da10: 953869MB (1953525168 512 byte sectors)
da8 at mpr0 bus 0 scbus2 target 8 lun 0
da8: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da8: Serial Number WL1HEBYN
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 953869MB (1953525168 512 byte sectors)
da0 at mpr0 bus 0 scbus2 target 0 lun 0
da0: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number WL1MPN4J
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 953869MB (1953525168 512 byte sectors)
da12 at mpr0 bus 0 scbus2 target 102 lun 0
da12: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da12: Serial Number WL1E2ESD
da12: 600.000MB/s transfers
da12: Command Queueing enabled
da12: 953869MB (1953525168 512 byte sectors)
da13 at mpr0 bus 0 scbus2 target 121 lun 0
da13: <ATA ST1000VT001-1RE1 SDC2> Fixed Direct Access SPC-4 SCSI device
da13: Serial Number WL1MDCQB

 

[jgreco]

I know you say "9340 in IT mode", but can you please share specifics of that here? A failure for all devices to enumerate is common if you are not running the correct firmware on a HBA.

 

[alisonsilveira]

I know you say "9340 in IT mode", but can you please share specifics of that here? A failure for all devices to enumerate is common if you are not running the correct firmware on a HBA.

Hi, here is more information about my controller

Code:

        Adapter Selected is a Avago SAS: SAS3008(C0)

        Controller Number              : 0
        Controller                     : SAS3008(C0)
        PCI Address                    : 00:08:00:00
        SAS Address                    : 500605b-0-0c33-9dc0
        NVDATA Version (Default)       : 0e.01.00.07
        NVDATA Version (Persistent)    : 0e.01.00.07
        Firmware Product ID            : 0x2221 (IT)
        Firmware Version               : 16.00.10.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9300-8i
        BIOS Version                   : N/A
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : SAS9300-8i
        Board Assembly                 : N/A
        Board Tracer Number            : N/A