SAS tray missing disks

[hrrmcelroy]

Today I added an external SAS shelf to my existing setup to allow for more drives than my existing setup offered. After doing so when I go to Storage -> Disks I only see 3 of my 24 disks. I also can only see one of the new blank drives under Storage -> Pools -> Create New Pool. Prior to adding my SAS shelf via an external miniSAS connector I could see all 12 of my drives that were in the main case. All 12 of these drives were assigned to various pools and are working like a charm including showing up in the Storage -> Disks window. Below is my setup before and after the addition of the SAS shelf. Now I'm already aware my setup isn't optimal or recommended so please help with my issue and don't point out that my hardware isn't recommended. My boss is cheap so I make do with what I've got. :)

Before SAS Shelf

• HP Proliant DL380 G6
• 96B of RAM
• 2x 3.07GHz quad core CPUs
• 1x PCI-E Ethernet card with 4x 1Gig ports setup in a LCAP LAG
• 1x SSD drive on a PCI-E expansion card (boot drive)
• 4x 8TB Samsung SSD drives in a RAIDZ2 ZFS pool
• 8x 300GB SAS drives in a RAIDZ1 ZFS pool (slush area for temp files)
• Perc 410i RAID controller (has an external minSAS connector)
  • Each drive is configured in a single disk RAID 0 so it can be handed to TrueNAS (YIKES!!!! I know, but it is what it is.)

This setup has worked like a charm for months so while not optimal or recommended it does work.

After SAS Shelf

All of my other hardware remained the same, but I added the following device as my SAS shelf. NetApp DS4246 with 2x IOM6 SAS SATA Expansion connected via the miniSAS port on the Perc 410i controller. Now the Perc 410i can see all the disks and they've all been configured in the same single disk RAID 0 setup as the other drives. When I boot into TrueNAS I only see my boot drive, one of the new drives, and an external USB box that is irrelevant to this discussion. Thankfully even though my existing 12 drives no longer show up in the Storage -> Disks window all of my existing pools are operational and work just fine. Oh and the SAS shelf has a LUN of 24 currently, but I've tried 03 and nothing changes. The "camcontrol devlist" shows all the drives. Output below.

Code:

[~]# camcontrol devlist
<HP RAID 0 OK>                     at scbus0 target 0 lun 0 (pass0,da0)
<HP RAID 0 OK>                     at scbus0 target 1 lun 0 (pass1,da1)
<HP RAID 0 OK>                     at scbus0 target 2 lun 0 (pass2,da2)
<HP RAID 0 OK>                     at scbus0 target 3 lun 0 (pass3,da3)
<HP RAID 0 OK>                     at scbus0 target 4 lun 0 (pass4,da4)
<HP RAID 0 OK>                     at scbus0 target 5 lun 0 (pass5,da5)
<HP RAID 0 OK>                     at scbus0 target 6 lun 0 (pass6,da6)
<HP RAID 0 OK>                     at scbus0 target 7 lun 0 (pass7,da7)
<HP RAID 0 OK>                     at scbus0 target 8 lun 0 (pass8,da8)
<HP RAID 0 OK>                     at scbus0 target 9 lun 0 (pass9,da9)
<HP RAID 0 OK>                     at scbus0 target 10 lun 0 (pass10,da10)
<HP RAID 0 OK>                     at scbus0 target 11 lun 0 (pass11,da11)
<HP RAID 0 OK>                     at scbus0 target 12 lun 0 (pass12,da12)
<HP RAID 0 OK>                     at scbus0 target 13 lun 0 (pass13,da13)
<HP RAID 0 OK>                     at scbus0 target 14 lun 0 (pass14,da14)
<HP RAID 0 OK>                     at scbus0 target 15 lun 0 (pass15,da15)
<HP RAID 0 OK>                     at scbus0 target 16 lun 0 (pass16,da16)
<HP RAID 0 OK>                     at scbus0 target 17 lun 0 (pass17,da17)
<HP RAID 0 OK>                     at scbus0 target 18 lun 0 (pass18,da18)
<HP RAID 0 OK>                     at scbus0 target 19 lun 0 (pass19,da19)
<HP RAID 0 OK>                     at scbus0 target 20 lun 0 (pass20,da20)
<HP RAID 0 OK>                     at scbus0 target 21 lun 0 (pass21,da21)
<HP RAID 0 OK>                     at scbus0 target 22 lun 0 (pass22,da22)
<HP RAID 0 OK>                     at scbus0 target 23 lun 0 (pass23,da23)
<hp DVD A  DS8A8SH KHE4>           at scbus3 target 0 lun 0 (cd0,pass24)
<KINGSTON SUV400S37120G 0C3FD6SD>  at scbus4 target 0 lun 0 (pass25,ada0)
<Mobius H /W RAID10 0103>          at scbus7 target 0 lun 0 (da24,pass26)

Here is the output of "geom disk list"

Code:

[~]# geom disk list
Geom name: ada0
Providers:
1. Name: ada0
   Mediasize: 120034123776 (112G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r1w1e2
   descr: KINGSTON SUV400S37120G
   lunid: 0550380440010000
   ident: 50026B77680016CA
   rotationrate: 0
   fwsectors: 63
   fwheads: 16

Geom name: da0
Providers:
1. Name: da0
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001ce9a104969a06f8e7912c
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da1
Providers:
1. Name: da1
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c49857ef1aebfc604f4e8
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da2
Providers:
1. Name: da2
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c0dc8a56f321e33fb95eb
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da3
Providers:
1. Name: da3
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001cf0c20c3d8ea903c67ae0
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da4
Providers:
1. Name: da4
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c7ae365683dadf9c09cca
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da5
Providers:
1. Name: da5
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c004fd893c9c972cc88e1
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da6
Providers:
1. Name: da6
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c3d291b30a6737d6ad338
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da7
Providers:
1. Name: da7
   Mediasize: 299966445568 (279G)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c8a141bd369eec685f8a1
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da8
Providers:
1. Name: da8
   Mediasize: 8001529667584 (7.3T)
   Sectorsize: 512
   Mode: r1w1e3
   descr: HP RAID 0
   lunid: 600508b1001c9f5c3fa7308fdba03f07
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da9
Providers:
1. Name: da9
   Mediasize: 8001529667584 (7.3T)
   Sectorsize: 512
   Mode: r1w1e3
   descr: HP RAID 0
   lunid: 600508b1001c2c148ae905550fa5bce9
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da10
Providers:
1. Name: da10
   Mediasize: 8001529667584 (7.3T)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c49c67e58600edde81c41
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da11
Providers:
1. Name: da11
   Mediasize: 8001529667584 (7.3T)
   Sectorsize: 512
   Mode: r2w2e5
   descr: HP RAID 0
   lunid: 600508b1001c038c271bf4f930a5e72d
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da12
Providers:
1. Name: da12
   Mediasize: 4000753475584 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c0b5b461287b0672423fe
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da13
Providers:
1. Name: da13
   Mediasize: 4000753475584 (3.6T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001cbd6967bf965a05d274bd
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da14
Providers:
1. Name: da14
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001cdce56335bb8ea47298d9
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da15
Providers:
1. Name: da15
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c1ad213a9648e4700a5f5
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da16
Providers:
1. Name: da16
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c988a42d81d445a267d3c
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da17
Providers:
1. Name: da17
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c50fb58d7136a1d50ae18
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da18
Providers:
1. Name: da18
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c45da4e44923be4adf5f4
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da19
Providers:
1. Name: da19
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c1f30b784931c4e767a79
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da20
Providers:
1. Name: da20
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001cd9e1fc87d79129bc318f
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da21
Providers:
1. Name: da21
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c8a9bb71d7a630550c6a4
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da22
Providers:
1. Name: da22
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001cd9bbfc2b1f58827e1ba3
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: da23
Providers:
1. Name: da23
   Mediasize: 3000559427584 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   descr: HP RAID 0
   lunid: 600508b1001c434e64d4ca520c788eb6
   ident: 50014380238AF680
   rotationrate: unknown
   fwsectors: 32
   fwheads: 255

Geom name: cd0
Providers:
1. Name: cd0
   Mediasize: 0 (0B)
   Sectorsize: 2048
   Mode: r0w0e0
   descr: hp DVD A  DS8A8SH
   ident: (null)
   rotationrate: unknown
   fwsectors: 0
   fwheads: 0

Geom name: da24
Providers:
1. Name: da24
   Mediasize: 12002218999808 (11T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r2w2e5
   descr: Mobius H /W RAID10
   lunname: JMICRON JMS567 DISK00   0123456789ABCDEF
   lunid: 3001234567891234
   ident: 0123456789ABCDEF
   rotationrate: unknown
   fwsectors: 63
   fwheads: 255

Anyone have any thoughts? Does my LUN for the SAS shelf need to be 0 since it is on Perc 410i which is LUN 0? I'm kind of in uncharted territory here.

 

[sybreeder]

Start with throwing out that p410 and buy any HBA. You can get it dirt cheap.. 9211-8i, IBM M1015 crossflashed to IT mode... anything...

 

[hrrmcelroy]

You did what I specifically asked repliers not to and that was say "get new hardware". I'm fully aware of the flaws in my setup and have a long term plan to address that, but none of that is relevant to my current situation. Also one does not simply "get a new card" when you have 8TB of data to move around. So help a brother out and tell me how to fix my issue so I can in the future change my setup without having to suck down 8TB of data from Back Blaze. :)

Obvious answer of changing the card aside why would it work before adding the SAS shelf and then not afterwards? To restate the question is this a LUN ID issue? Having only theoretical knowledge of how enterprise grade SAS expanders work my issue could be a simple as setting the right LUN ID for the SAS shelf.

 

[HoneyBadger]

You did what I specifically asked repliers not to and that was say "get new hardware"

Generally speaking, most forum users will rightfully refuse to help you load a gun you are currently pointing at your foot. While the initial reply might be dismissive it's not wrong, and the P410i in particular is not a very good disk controller. P420 is where they introduced HBA mode which elevated it to "not complete trash" but it still uses the poorly supported ciss driver which has far, far less engineering investment than the LSI drivers.

Furthermore, this is a business array from the sounds of it, which means that downtime is measured in lost dollars as well as lost hours. While your boss may be cheap, ask him what the cost of a failed array and restore from backup is. I wager it'll be more than the $100 or so it would take to get you onto the gold-standard of HBAs.

Thankfully there is a path forward; you'll need to acquire an LSI external HBA, at which point you could attach the NetApp disk shelf, set it to JBOD mode (it appears to be presenting itself as a single 11TB disk?) and then migrate your data there. Once that's done, you can get an LSI internal HBA, destroy the P410i-based pool, reconnect the drives to the LSI, and have a nice solid solution. The P410i can be part of a redundant boot environment or perhaps hold L2ARC devices but don't use it for pool vdevs.

And before you reply with "but I don't want to buy new hardware" know that because of hardware limitations on the P410 SmartArray cards you may have to in order to have the external devices populate correctly. It's likely that the external SAS port is numbered as ports 0-3 and the internal as 4-7 (because HP likes their backwards logic) and as such it's bumped the LUN numbers around. Or it's doing silly things with the drive S/N's.

 

[sybreeder]

You did what I specifically asked repliers not to and that was say "get new hardware". I'm fully aware of the flaws in my setup and have a long term plan to address that, but none of that is relevant to my current situation. Also one does not simply "get a new card" when you have 8TB of data to move around. So help a brother out and tell me how to fix my issue so I can in the future change my setup without having to suck down 8TB of data from Back Blaze. :)

Obvious answer of changing the card aside why would it work before adding the SAS shelf and then not afterwards? To restate the question is this a LUN ID issue? Having only theoretical knowledge of how enterprise grade SAS expanders work my issue could be a simple as setting the right LUN ID for the SAS shelf.

It's a production environment. If anything fails your boss will blame you that company lost data. In homelab i can test whatever i want. I use Virtualized TrueNAS, regular hard drives and so on. I'd never do this in production. We have Dell R730XD that i'm extending warranty till it ends then i'll order new one. I can't allow to alow to system fail And you shouldn't too. If your boss allows it its a shitty boss. What's your backup strategy? At home you can do whatever you want but if this is production remember. If anything will goes wrong. It's your fault.

 

[hrrmcelroy]

HoneyBadger I appreciate the explanation that went along with your advice to change hardware. More info to backup up the advice to change is all I was looking for. It gives me leverage with the boss.

Oh we've had the downtime discussion before, but since it hasn't happened yet it isn't real I guess. I do my best to make the cost of failure sink in. Sometimes it takes more work than others. :)

I did purchase an LSI external HBA and it is in route. The SAS shelf is presenting the disks directly to the RAID card now so I assume it will do the same with the LSI card. The 11TB drive you saw in my logs is an external USB device that is being replaced by the SAS shelf.

The S/N's are all showing the same now even after I unhooked and removed the logical arrays that were in the SAS shelf. Thankfully all of my pools are still working even though the disks don't show up in the Web UI. I'll have enough space on my SAS shelf to move everything to it, switch to an internal LSI card, and then move some stuff back. Just waiting on the card to arrive so I can get started.

Thinking ahead to the internal SAS card.... The P410i has 6 SAS connectors in use, 8 in total I think. 4 go to the 16 drive bays and 2 to the motherboard. I assume the 2 that go to the motherboard where the RAID cache upgrade card goes are not required for the drives? My educated guess is they are used as an in and out from the RAID upgrade cache card. Based on that information I need an internal card that has 4 SAS connectors?