Added Drive, Others Became Unavailable

CragMonkey

Cadet
Joined
Jan 11, 2021
Messages
4
I had an error indicating that a drive had failed, so I slipped a spare drive into an open bay without touching any of the other drives. Then when I checked the pool status, I see that now three drives show "unavailable". Removing the new drive had no effect, and neither did rebooting the system.

Even tried powering down, then reseating every drive, all 5 RAID cards, and both ends of all 9 cables connecting the RAID cards to the hot swap backplane.

Nothing I try will let the system recognize that the drives are in place.

Really not sure where to go from here...open to suggestions!


This is a pre-owned Coraid SRX6300 4U server running FreeNAS-11.3-U5 with 12x 4TB drives in RAIDZ2 with two vdevs + 1x 250GB drive for the OS.



zpool status -x
Code:
  pool: Tank
 state: DEGRADED
status: One or more devices could not be opened.  Sufficient replicas exist for
        the pool to continue functioning in a degraded state.
action: Attach the missing device and online it using 'zpool online'.
   see: http://illumos.org/msg/ZFS-8000-2Q
  scan: resilvered 10.1M in 0 days 00:00:02 with 0 errors on Mon Jan 11 07:38:10 2021
config:

        NAME                                            STATE     READ WRITE CKSUM
        Tank                                            DEGRADED     0     0 0
          raidz2-0                                      DEGRADED     0     0 0
            gptid/2373af68-1138-11eb-ab9e-002590e14676  ONLINE       0     0 0
            gptid/28477b8a-1138-11eb-ab9e-002590e14676  ONLINE       0     0 0
            gptid/2d4d3061-1138-11eb-ab9e-002590e14676  ONLINE       0     0 0
            4706385122737712328                         UNAVAIL      0     0 0  was /dev/gptid/3019e88f-1138-11eb-ab9e-002590e14676
            2147381399695210159                         UNAVAIL      0     0 0  was /dev/gptid/2ee93bb6-1138-11eb-ab9e-002590e14676
            gptid/34674fa1-1138-11eb-ab9e-002590e14676  ONLINE       0     0 0
          raidz2-1                                      DEGRADED     0     0 0
            gptid/6aae91d9-157a-11eb-81e7-002590e14676  ONLINE       0     0 0
            18054547547586041078                        UNAVAIL      0     0 0  was /dev/gptid/6f8b3dd0-157a-11eb-81e7-002590e14676
            gptid/7134c080-157a-11eb-81e7-002590e14676  ONLINE       0     0 0
            gptid/7769a695-157a-11eb-81e7-002590e14676  ONLINE       0     019
            gptid/79cc8eeb-157a-11eb-81e7-002590e14676  ONLINE       0     0 0
            gptid/7b706784-157a-11eb-81e7-002590e14676  ONLINE       0     0 0

errors: No known data errors



camcontrol devlist

Code:

<ATA ST4000NC000-1FR1 CE03>        at scbus0 target 2 lun 0 (pass0,da0)
<ATA ST4000NC000-1FR1 CE03>        at scbus0 target 3 lun 0 (pass1,da1)
<ATA ST4000NC000-1FR1 CE03>        at scbus0 target 4 lun 0 (pass2,da2)
<ATA ST4000NC000-1FR1 CE03>        at scbus0 target 5 lun 0 (pass3,da3)
<ATA ST4000NC000-1FR1 CE03>        at scbus1 target 2 lun 0 (pass4,da4)
<ATA ST4000NC000-1FR1 CE03>        at scbus1 target 3 lun 0 (pass5,da5)
<ATA ST4000NC000-1FR1 CE03>        at scbus1 target 4 lun 0 (pass6,da6)
<ATA ST4000NC000-1FR1 CE03>        at scbus1 target 5 lun 0 (pass7,da7)
<ATA ST4000NC000-1FR1 CE03>        at scbus1 target 6 lun 0 (pass8,da8)
<ATA ST4000NC000-1FR1 CE03>        at scbus1 target 7 lun 0 (pass9,da9)
<ATA HDT722525DLA380 A96A>         at scbus2 target 2 lun 0 (pass10,da10)
<AHCI SGPIO Enclosure 2.00 0001>   at scbus10 target 0 lun 0 (pass11,ses0)





glabel status

Code:

Name  Status  Components
gpt/Microsoft%20reserved%20partition     N/A  da0p1
gptid/2195cd15-9a96-47f5-8641-5a9f2682f11c     N/A  da0p1
gptid/7769a695-157a-11eb-81e7-002590e14676     N/A  da1p2
gptid/34674fa1-1138-11eb-ab9e-002590e14676     N/A  da2p2
gptid/28477b8a-1138-11eb-ab9e-002590e14676     N/A  da3p2
gptid/2373af68-1138-11eb-ab9e-002590e14676     N/A  da4p2
gptid/6aae91d9-157a-11eb-81e7-002590e14676     N/A  da5p2
gptid/79cc8eeb-157a-11eb-81e7-002590e14676     N/A  da6p2
gptid/7134c080-157a-11eb-81e7-002590e14676     N/A  da7p2
gptid/2d4d3061-1138-11eb-ab9e-002590e14676     N/A  da8p2
gptid/7b706784-157a-11eb-81e7-002590e14676     N/A  da9p2
gptid/4b7323d2-08a2-11eb-8c72-002590e14676     N/A  da10p1
gptid/4df674ec-08a2-11eb-8c72-002590e14676     N/A  da10p3
gptid/73444428-157a-11eb-81e7-002590e14676     N/A  da1p1


dmidecode
Code:
# dmidecode 3.2
Scanning /dev/mem for entry point.
SMBIOS 2.7 present.
63 structures occupying 4120 bytes.
Table at 0x000E9820.

Handle 0x0000, DMI type 0, 24 bytes
BIOS Information
        Vendor: American Megatrends Inc.
        Version: 3.3
        Release Date: 07/13/2018
        Address: 0xF0000
        Runtime Size: 64 kB
        ROM Size: 8192 kB
        Characteristics:
                PCI is supported
                BIOS is upgradeable
                BIOS shadowing is allowed
                Boot from CD is supported
                Selectable boot is supported
                BIOS ROM is socketed
                EDD is supported
                5.25"/1.2 MB floppy services are supported (int 13h)
                3.5"/720 kB floppy services are supported (int 13h)
                3.5"/2.88 MB floppy services are supported (int 13h)
                Print screen service is supported (int 5h)
                8042 keyboard services are supported (int 9h)
                Serial services are supported (int 14h)
                Printer services are supported (int 17h)
                ACPI is supported
                USB legacy is supported
                BIOS boot specification is supported
                Function key-initiated network boot is supported
                Targeted content distribution is supported
                UEFI is supported
        BIOS Revision: 1.0

Handle 0x0001, DMI type 1, 27 bytes
System Information
        Manufacturer: Supermicro
        Product Name: X9DRH-7TF/7F/iTF/iF
        Version: 1234567890
        Serial Number: 1234567890
        UUID: 00000000-0000-0000-0000-002590e14676
        Wake-up Type: Power Switch
        SKU Number: 1234567890
        Family: To be filled by O.E.M.

Handle 0x0002, DMI type 2, 15 bytes
Base Board Information
        Manufacturer: Supermicro
        Product Name: X9DRH-7TF/7F/iTF/iF
        Version: 1234567890
        Serial Number: 1234567890
        Asset Tag: To be filled by O.E.M.
        Features:
                Board is a hosting board
                Board is replaceable
        Location In Chassis: To be filled by O.E.M.
        Chassis Handle: 0x0003
        Type: Motherboard
        Contained Object Handles: 0

Handle 0x0003, DMI type 3, 22 bytes
Chassis Information
        Manufacturer: Supermicro
        Type: Main Server Chassis
        Lock: Not Present
        Version: 1234567890
        Serial Number: 1234567890
        Asset Tag: To Be Filled By O.E.M.
        Boot-up State: Safe
        Power Supply State: Safe
        Thermal State: Safe
        Security Status: None
        OEM Information: 0x00000000
        Height: Unspecified
        Number Of Power Cords: 1
        Contained Elements: 0
        SKU Number: To be filled by O.E.M.

Handle 0x0004, DMI type 4, 42 bytes
Processor Information
        Socket Designation: SOCKET 0
        Type: Central Processor
        Family: Xeon
        Manufacturer: Intel
        ID: D7 06 02 00 FF FB EB BF
        Signature: Type 0, Family 6, Model 45, Stepping 7
        Flags:
                FPU (Floating-point unit on-chip)
                VME (Virtual mode extension)
                DE (Debugging extension)
                PSE (Page size extension)
                TSC (Time stamp counter)
                MSR (Model specific registers)
                PAE (Physical address extension)
                MCE (Machine check exception)
                CX8 (CMPXCHG8 instruction supported)
                APIC (On-chip APIC hardware supported)
                SEP (Fast system call)
                MTRR (Memory type range registers)
                PGE (Page global enable)
                MCA (Machine check architecture)
                CMOV (Conditional move instruction supported)
                PAT (Page attribute table)
                PSE-36 (36-bit page size extension)
                CLFSH (CLFLUSH instruction supported)
                DS (Debug store)
                ACPI (ACPI supported)
                MMX (MMX technology supported)
                FXSR (FXSAVE and FXSTOR instructions supported)
                SSE (Streaming SIMD extensions)
                SSE2 (Streaming SIMD extensions 2)
                SS (Self-snoop)
                HTT (Multi-threading)
                TM (Thermal monitor supported)
                PBE (Pending break enabled)
        Version: Intel(R) Xeon(R) CPU E5-2620 0 @ 2.00GHz
        Voltage: 0.0 V
        External Clock: 100 MHz
        Max Speed: 4000 MHz
        Current Speed: 2000 MHz
        Status: Populated, Enabled
        Upgrade: Socket LGA2011
        L1 Cache Handle: 0x0005
        L2 Cache Handle: 0x0006
        L3 Cache Handle: 0x0007
        Serial Number: Not Specified
        Asset Tag: Not Specified
        Part Number: Not Specified
        Core Count: 6
        Core Enabled: 6
        Thread Count: 12
        Characteristics:
                64-bit capable
                Multi-Core
                Hardware Thread
                Execute Protection
                Enhanced Virtualization
                Power/Performance Control

Note: The full output of dmidecode is 30k characters long, if there is another specific section that is relevant please let me know and I'll add it here.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Exactly what raid card are you using and it's it flashed to IT mode or are you actually using raid?
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Really not sure where to go from here...open to suggestions!
 

CragMonkey

Cadet
Joined
Jan 11, 2021
Messages
4
Exactly what raid card are you using and it's it flashed to IT mode or are you actually using raid?

They are all Supermicro AOC-S2308L-L8i 8-Port SAS/SATA PCIe RAID Cards.

FreeNAS recognizes each disk individually, so I assume that means they're in IT mode...
 
Top