SOLVED Dell R720xd cannot see disks in NetApp DS4246

[Daisuke]

I have a Dell R720xd with a PERC H710 Mini flashed to IT mode and TrueNAS Scale 22.02.3 running on it. I purchased a NetApp DS4246 with dual IOM6's and dual power supplies. I had a PERC H810 flashed to IT mode and used a SFF-8436 to SFF-8088 cable, but the SATA disks are not seen in TrueNAS Scale.

So far I tried 2 cables and 3 different cards, with no success:

• Dell PERC H810 flashed to IT mode
• Dell PERC H200E (LSI 9200-8E)
• HP H221 (LSI SAS9207-8e)

Right now I have installed the two flashed cards, PERC H710 mini (internal) and PERC H810 (external). The cards are detected properly by Scale (ADDR0 is the internal PERC H710, with all internal disks in use by a pool):

Code:

# lspci | grep LSI
02:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)
44:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

# sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved

    Adapter Selected is a LSI SAS: SAS2308_2(D1)

Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------

0  SAS2308_2(D1)   20.00.07.00    14.01.00.06    07.39.02.00     00:02:00:00
1  SAS2308_2(D1)   20.00.07.00    14.01.00.06    07.39.02.00     00:44:00:00

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

I use the same disk model, for both Dell R720xd and NetApp DS4246: HGST Ultrastar He8 Helium (HUH728080ALE601).

The NetApp ID is set to 00 and cable is connected into square port as per this diagram. BTW, disks are installed correctly, flush with the bottom of each caddie. I also installed one interposer for one disk, no dice. When I tested the connectivity, I had both devices shutdown, power first DS4246, wait until fans slow down then power R720xd.

Can anyone using these appliances post some guidance what to troubleshoot next? I'm running out of ideas, thank you for your help.

 

[diogen]

Did this NetApp box with these drives ever work with any other OS/appliance?

IIRC, NetApp requires drives with custom firmware...
Most of brand name NAS boxes do: MD3x00, EqualLogic, Compellent, etc.

As a side note: this requirement is waved for drives you install in the R720 box itself - any brand/models will do.
We were using consumer Samsung 1TB Pro SSD (3x720xd cluster, vCenter 6.5, vSAN, 12 drives each, full flash)...

 

[Wolfspyre]

o/
Howdy!
seems like there might be a couple issues...

IF I UNDERSTAND YOUR ISSUE CORRECTLY:

```
/ host (r720xd)\
| |
| |_______
[H200E] [H810]
\...... /
array (4246)
```

you've tried both these controllers, and neither could see the array.

Have you tried anything *OTHER* than truenas to see the disks in that array?


I had to use a (flashed) H330 to connect my external sleds to my r720's...
I can see them in all OSes...


I suspect a couple things:

1) you SHOULD be able to see the attached scsi devices/controllers/disks from the array in the scsi/sas bios...

if the array isn't presending scsi block devices to the controllers, truenas wouldn't ever get the option to interact with them

2) the netapp may not support sata disks... might wanna test with a known functional sas disk as a canary.


3) I'm not sure if the 810 is visible in an r720.... I was looking into this myself and found conflicting reports of functionality or lack thereof.

can you see ANYTHING if its attached to this controller?

 

[Daisuke]

Did this NetApp box with these drives ever work with any other OS/appliance?

The vendor tested the appliance prior sale (he offers 1 year warranty) and confirmed my HGST Ultrastar He8 Helium (HUH728080ALE601) drives are okay to use with DS4246 enclosure. He said interposers are not required for my drives, but I purchased 24 for $50, just in case.

The issue is, the enclosure does not shows at all, it should show regardless the disks installed or not. Is like the cable is disconnected, only the 2 SAS cards show in BIOS.

sg_scan -i should show a line like this:

Code:

SCSI1:0,67,0 claimed=0 pdt=dh NETAPP DS224IOM6 0173

With SATA disks and no interposer, only one IOM is actually connected to the disks.

I'm not sure if the 810 is visible in an r720

Right now I have one H710 mini (9207-8i slot 00) and one H810 (9207-8e slot 02) both flashed to IT mode, installed in my R720xd and detected in BIOS:

I don't see an external version for H330, only internal. I need SFF-8088 connectors. I was reading on STH forums and a good old 9200-8e card in IT mode will work great with the DS4246 enclosure. If anyone can confirm a working card model, it will be appreciated.

Some updates: The H200E was actually defective and HP H221 had old firmware badly flashed. When I tried to update it, it bricked the card (the vendor refunded me the money when I showed him the performed upgrade details).

I ordered an OEM LSI 9200-8e. I'm also not sure the cable is good, so I ordered one from Amazon, instead of dealing with eBay.

 

[Daisuke]

Turns out Debian cares about the firmware present into PERC H810 controller, which had 9207-8i firmware flashed into. As soon as I flashed the 9207-8e firmware, everything was functional.

There is something interesting related how the enclosure disks are seen by Debian. If I connect the cable into top “square” IOM6 port, all disks are seen, no matter where they are installed, with or without interposers installed. If I connect the cable into bottom “square” IOM6 port, only the disks with interposer installed are seen by Debian.

 

[HoneyBadger]

There is something interesting related how the enclosure disks are seen by Debian. If I connect the cable into top “square” IOM6 port, all disks are seen, no matter where they are installed, with or without interposers installed. If I connect the cable into bottom “square” IOM6 port, only the disks with interposer installed are seen by Debian.

The two IOM controllers each have one SAS path wired to them - without an interposer, the SATA devices have no second path. Adding the interposer will result in both paths being present.

(Also, I wish SuperMicro interposers were that cheap. 24 for $50 is a steal.)

 

[Daisuke]

I wish SuperMicro interposers were that cheap. 24 for $50 is a steal.

I got lucky finding a batch of 24 interposers for $50 (actually $45 with accepted offer) on eBay. I see them still available, others sell them for $7/unit.

without an interposer, the SATA devices have no second path

I did not know, thank you for the useful info.

 

[Daisuke]

@HoneyBadger quick question, can you daisy chain two DS4246 with just one SFF-8436 cable? I found this document:
https://library.netapp.com/ecm/ecm_get_file/ECMP1115547

I'm ordering a second DS4246, just checking if I need 2 cables, or just one.

 

[HoneyBadger]

@HoneyBadger quick question, can you daisy chain two DS4246 with just one SFF-8436 cable? I found this document:
https://library.netapp.com/ecm/ecm_get_file/ECMP1115547

I'm ordering a second DS4246, just checking if I need 2 cables, or just one.

The simplest setup is "just run a single cable and use the top controllers only." Top IOM "circle port" goes to new shelf's top IOM "square port" and you leave the bottom IOMs alone (or unrack them, if it doesn't make the fan controllers go crazy)

If you want to be able to leverage that second SAS data path for redundancy (or speed) you'd have to install interposers in all of the bays, cable both paths in the "circle to square" setup, and then set up multipath SAS in TrueNAS.

 

[Daisuke]

leverage that second SAS data path for redundancy (or speed)

From your perspective, is it worth installing the interposers? I will always purchase the same SATA disk model and use top ports for both DS4246 units but I’m interested in possible performance gains. I ask this to determine if I should order another set of 24 interposers for the new enclosure.

set up multipath SAS in TrueNAS

Not familiar with the process, can you please detail more?

 

[HoneyBadger]

Apologies for the delay.

From your perspective, is it worth installing the interposers? I will always purchase the same SATA disk model and use top ports for both DS4246 units but I’m interested in possible performance gains. I ask this to determine if I should order another set of 24 interposers for the new enclosure.

From my perspective, not unless you want to have your build tolerate an HBA failure. You're planning to feed SATA devices only, and adding the interposers won't change the performance characteristics of the drives.

Not familiar with the process, can you please detail more?

The FreeBSD gmultipath utility is used to create a /dev/multipath/diskX device - it scans for disk that have identical unique IDs (usually the drive serial number is used) that has two child devices of individual daX devices underneath it - but I believe the multipath system is hidden in the UI for 12.x and 13.x and I unfortunately don't have a multipath SAS system right now to see if I can validate that.

 

[Daisuke]

From my perspective, not unless you want to have your build tolerate an HBA failure.

I purchased another DS4246 from a local classified for a really good deal and it came with the interposers. I’m going to install the interposers for all SATA disks, as like you said it will allow me prevent HBA failures. On the other hand, this could also be detrimental since I will not know if I deal with an HBA failure. :)

Also, I’m using Scale, not Core.

 

[HoneyBadger]

I purchased another DS4246 from a local classified for a really good deal and it came with the interposers. I’m going to install the interposers for all SATA disks, as like you said it will allow me prevent HBA failures. On the other hand, this could also be detrimental since I will not know if I deal with an HBA failure. :)

I'm hoping that the TrueNAS UI will be extremely loud and angry if it detects a degraded multipath SAS device (ACTIVE/PASSIVE transitioning to FAILED/ACTIVE) as well as the IPMI/out-of-band management yelling about "failed device at PCIe bus X:Y:Z" - but either of those are better than finding out because of a "system down" alert.

Also, I’m using Scale, not Core.

In that case, it should be multipathd handling the heavy lifting rather than gmultipath, but the same general behaviour should apply in that it will detect identical disk UUIDs and create a multipath device.

 

[Daisuke]

it should be multipathd handling the heavy lifting rather than gmultipath

I was thinking there is some option I need to enable in TrueNAS UI.

That said, I'm really happy I went the DS4246 way. Originally I was looking at Dell PowerVault MD3060e enclosures but they were way more expensive and sounded like jet engines, DS4246 is extremely quiet when both power sources are used and dirty cheap.