BUILD Supermicro Chassis for X10SL7-F? Modest/Reliable Backup Server.

[JayG30]

Well, SGPIO isn't really software, it can drive LED indicators for the drive bays and on the backplane. Without it I don't believe they work (for instance drive failure indicator).

I've already fired up the server (no HDD plugged in yet) and all the IPMI stuff works fine. With the fans on the MB they are all controlled correctly (still interested in checking through the backplane at some point). PSU information available as well via the PMBus.

I have a few feeler questions out to various people that know more about what these reverse breakout cables can do. If I don't hear anything back I'm going to just purchase one and see for myself. If it doesn't work I will decide how important it is to me and if I rather send the X10SL7-F board back and go to a cheaper board without the LSI 2308 controller but with an extra PCI lane and just put a HBA in it. Because I know it will work with a SFF-8087 to SFF-8087 cable.

 

[Ericloewe]

The most important part, drive activity and presence, doesn't require SGPIO. Drive fault/"replace me!" is the one that does, but also requires software support, since FreeNAS is the RAID controller.

Of course, assuming Supermicro didn't cheap out on the backplane.

 

[crocuta]

Of course, assuming Supermicro didn't cheap out on the backplane.

Similar build here (SC836E26-R1200 w/SAS2-836EL2 6Gbps expander backplane and X10SL7-F) chiming in to add you are correct, the drive activity lights work without the need for an SGPIO connection using just a reverse breakout cable.

If you were going with ZFS on Linux then I would say get a different board with an M1015 as you could use something like zfswatcher to control the exclosure LEDs for more than just activity but even in that case I don't see that as a necessity as labeling the drive serial on the tray for identification works well enough and I would hope that seeing a failure light on the enclosure would be how you inform yourself of the status of your drives.

 

[MikeT]

I am using with great success a build with X10SL7-F 32 GB RAM and 4 x WD30EFRX 3 TB in a tower case with the RAID controller flashed in IT mode.
Second build is using the same motherboard X10SL7-F and will have at least 6 x WD30EFRX 3 TB and with the RAID controller flashed in IT mode.
For that a rack mountable case was ordered SC826BE16-R920LPB which has an expander backplane with two SFF 8087 connectors.
I've read almost all the posts regarding this configuration and I've seen that it should work.
What I did not understand is how do you connect the backplane to the motherboard ? I understand that I need the "Discrete SATA to SFF-8087 Mini SAS Reverse breakout cable". Do I need one or two ? How does it work ?

Thank you for building such a great community.

 

[JayG30]

So I have some updates on the LED light/SES-2/SPGIO stuff. As mentioned activity LED's work out of the box. I will admit that I'm not deep into understanding all of the enclosure management stuff, I never really bothered in the past or if it was that important I'd have a vendor solution. HOWEVER, I think what I have here is very useful and promising. I threw some spart 1TB disks I had laying around to do some testing.

Using the Norco reverse breakout cables (sold on newegg, everyone has seen them), no SGPIO connection or anything, I'm able to blink the red lights on the case and the backplane shows up in FreeNAS/BSD.

First, here is what FreeNAS sees...

Code:

[root@freenas] ~# dmesg | grep ses
ses0 at mps0 bus 0 scbus0 target 12 lun 0
ses0: <LSI CORP SAS2X28 0717> Fixed Enclosure Services SCSI-5 device
...

Code:

[root@freenas] ~# camcontrol devlist
<ATA ST31000524AS JC45>            at scbus0 target 8 lun 0 (da0,pass0)
<ATA ST31000524AS JC45>            at scbus0 target 9 lun 0 (da1,pass1)
<ATA ST31000524AS JC45>            at scbus0 target 10 lun 0 (da2,pass2)
<ATA ST31000524AS JC45>            at scbus0 target 11 lun 0 (da3,pass3)
<LSI CORP SAS2X28 0717>            at scbus0 target 12 lun 0 (ses0,pass4)
<SanDisk Ultra Fit 1.00>           at scbus8 target 0 lun 0 (da4,pass5)

So clearly the motherboard and FreeBSD are aware of the LSI backplane/expander, somehow. :)

So next I went digging into FreeBSD options for SES-2 Enclosure Management. I came across a port of sg3_utils, found HERE. VERY simple to install. Had to look into the commands, and they have a pretty nice manual HERE. The command we are looking to use is "sg_ses" and that info is HERE. So once I had that in place I ran some commands to see what I could make work. :)

To blink disk0 red LED and than to clear it.

Code:

sg_ses -x 0 -S ident /dev/ses0
sg_ses -x 0 -C ident /dev/ses0

This is shorthand. The part "-x 0" stands for '--dev-slot-num=0' and identifies the disk you want to perform operations on. The -S is shorthand for set and -C is for clear.

Some more outputs of sg3_utils commands

Code:

[root@freenas] ~# sg_ses --page=ed /dev/ses0
  LSI CORP  SAS2X28           0717
  Primary enclosure logical identifier (hex): 50030480019578ff
Element Descriptor In diagnostic page:
  generation code: 0x0
  element descriptor by type list
    Element type: Array device slot, subenclosure id: 0 [ti=0]
      Overall descriptor: Drive Slots
      Element 0 descriptor: Slot 01
      Element 1 descriptor: Slot 02
      Element 2 descriptor: Slot 03
      Element 3 descriptor: Slot 04
      Element 4 descriptor: Slot 05
      Element 5 descriptor: Slot 06
      Element 6 descriptor: Slot 07
      Element 7 descriptor: Slot 08
      Element 8 descriptor: Slot 09
      Element 9 descriptor: Slot 10
      Element 10 descriptor: Slot 11
      Element 11 descriptor: Slot 12
    Element type: Temperature sensor, subenclosure id: 0 [ti=1]
      Overall descriptor: Temperature Sensors
      Element 0 descriptor: Temperature
    Element type: Cooling, subenclosure id: 0 [ti=2]
      Overall descriptor: Fans
      Element 0 descriptor: Fan1
      Element 1 descriptor: Fan2
      Element 2 descriptor: Fan3
    Element type: Audible alarm, subenclosure id: 0 [ti=3]
      Overall descriptor: Buzzers
      Element 0 descriptor: Buzzer
    Element type: Voltage sensor, subenclosure id: 0 [ti=4]
      Overall descriptor: Voltage Sensors
      Element 0 descriptor: 5V
      Element 1 descriptor: 12V
    Element type: Current sensor, subenclosure id: 0 [ti=5]
      Overall descriptor: Current Sensors
      Element 0 descriptor: 5V Line Current Sensor
      Element 1 descriptor: 12V Line Current Sensor
    Element type: Power supply, subenclosure id: 0 [ti=6]
      Overall descriptor: Power Supplies
      Element 0 descriptor: Power Supply 1
      Element 1 descriptor: Power Supply 2
    Element type: Enclosure, subenclosure id: 0 [ti=7]
      Overall descriptor: Enclosure
      Element 0 descriptor: Enclosure
    Element type: SAS expander, subenclosure id: 0 [ti=8]
      Overall descriptor: SAS Expanders
      Element 0 descriptor: Primary Expander
      Element 1 descriptor: Secondary Expander
    Element type: SAS connector, subenclosure id: 0 [ti=9]
      Overall descriptor: SAS Connectors
      Element 0 descriptor: Upstream Connector (Primary)
      Element 1 descriptor: Downstream Connector (Primary)
      Element 2 descriptor: Upstream Connector (Secondary)
      Element 3 descriptor: Downstream Connector (Secondary)
      Element 4 descriptor: Drive Connector 00
      Element 5 descriptor: Drive Connector 01
      Element 6 descriptor: Drive Connector 02
      Element 7 descriptor: Drive Connector 03
      Element 8 descriptor: Drive Connector 04
      Element 9 descriptor: Drive Connector 05
      Element 10 descriptor: Drive Connector 06
      Element 11 descriptor: Drive Connector 07
      Element 12 descriptor: Drive Connector 08
      Element 13 descriptor: Drive Connector 09
      Element 14 descriptor: Drive Connector 10
      Element 15 descriptor: Drive Connector 11
    Element type: Communication port, subenclosure id: 0 [ti=10]
      Overall descriptor: Ethernet ports
      Element 0 descriptor: Ethernet_port_1
      Element 1 descriptor: Ethernet_port_2

Code:

[root@freenas] ~# sg_ses -j -ff /dev/ses0
  LSI CORP  SAS2X28           0717
  Primary enclosure logical identifier (hex): 50030480019578ff
Slot 01 [0,0]  Element type: Array device slot
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
  Additional Element Status:
    Transport protocol: SAS
    number of phys: 1, not all phys: 0, device slot number: 0
    phy index: 0
      device type: no SAS device attached
      target port for: SATA_device
      attached SAS address: 0x50030480019578ff
      SAS address: 0x50030480019578ec
      phy identifier: 0x0
Slot 02 [0,1]  Element type: Array device slot
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
  Additional Element Status:
    Transport protocol: SAS
    number of phys: 1, not all phys: 0, device slot number: 1
    phy index: 0
      device type: no SAS device attached
      target port for: SATA_device
      attached SAS address: 0x50030480019578ff
      SAS address: 0x50030480019578ed
      phy identifier: 0x0
Slot 03 [0,2]  Element type: Array device slot
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
  Additional Element Status:
    Transport protocol: SAS
    number of phys: 1, not all phys: 0, device slot number: 2
    phy index: 0
      device type: no SAS device attached
      target port for: SATA_device
      attached SAS address: 0x50030480019578ff
      SAS address: 0x50030480019578ee
      phy identifier: 0x0
Slot 04 [0,3]  Element type: Array device slot
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
  Additional Element Status:
    Transport protocol: SAS
    number of phys: 1, not all phys: 0, device slot number: 3
    phy index: 0
      device type: no SAS device attached
      target port for: SATA_device
      attached SAS address: 0x50030480019578ff
      SAS address: 0x50030480019578ef
      phy identifier: 0x0
Temperature [1,0]  Element type: Temperature sensor
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Temperature=19 C
Buzzer [3,0]  Element type: Audible alarm
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Tone indicator: Info=1, Non-crit=1, Crit=1, Unrecov=1
5V [4,0]  Element type: Voltage sensor
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Voltage: 5.08 volts
12V [4,1]  Element type: Voltage sensor
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Voltage: 11.98 volts
5V Line Current Sensor [5,0]  Element type: Current sensor
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Current: 0.76 amps
12V Line Current Sensor [5,1]  Element type: Current sensor
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Current: 0.36 amps
Enclosure [7,0]  Element type: Enclosure
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
Primary Expander [8,0]  Element type: SAS expander
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, Fail=0
  Additional Element Status:
    Transport protocol: SAS
    number of phys: 24
    SAS address: 0x80040350ff789501
      [0] connector element index: 0
      [1] connector element index: 0
      [2] connector element index: 0
      [3] connector element index: 0
      [4] connector element index: 1
      [5] connector element index: 1
      [6] connector element index: 1
      [7] connector element index: 1
      [8] no attached connector
      [9] no attached connector
      [10] no attached connector
      [11] no attached connector
      [12] connector element index: 4
      [13] connector element index: 5
      [14] connector element index: 6
      [15] connector element index: 7
      [16] connector element index: 8
      [17] connector element index: 9
      [18] connector element index: 10
      [19] connector element index: 11
      [20] connector element index: 12
      [21] connector element index: 13
      [22] connector element index: 14
      [23] connector element index: 15
Upstream Connector (Primary) [9,0]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, Mini SAS 4i receptacle (SFF-8087) [max 4 phys]
    Connector physical link=0xff, Fail=0
Downstream Connector (Primary) [9,1]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, Mini SAS 4i receptacle (SFF-8087) [max 4 phys]
    Connector physical link=0xff, Fail=0
Upstream Connector (Secondary) [9,2]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, Mini SAS 4i receptacle (SFF-8087) [max 4 phys]
    Connector physical link=0xff, Fail=0
Downstream Connector (Secondary) [9,3]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, Mini SAS 4i receptacle (SFF-8087) [max 4 phys]
    Connector physical link=0xff, Fail=0
Drive Connector 00 [9,4]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 01 [9,5]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 02 [9,6]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 03 [9,7]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 04 [9,8]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 05 [9,9]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 06 [9,10]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 07 [9,11]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 08 [9,12]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 09 [9,13]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 10 [9,14]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0
Drive Connector 11 [9,15]  Element type: SAS connector
  Enclosure Status:
    Predicted failure=0, Disabled=0, Swap=0, status: OK
    Ident=0, SAS Drive backplane receptacle (SFF-8482) [max 2 phys]
    Connector physical link=0x0, Fail=0

So, from this info I feel like I can conclude that SES-2 works in this setup. It seems to see everything and provide status information and controls them. I haven't tested everything though, like buzzers. To Ericloewe point however, this would lack FreeNAS integration I believe. Basically you can run this stuff via command line but FreeNAS I don't believe will trigger any of this (or ZFS). That is most likely what TrueNAS does, it builts that layer into the software (not sure what they use instead of sg3_util).

This is good enough for me though, because it would seem to me that the setup supports it IF that functionality is ever added into FreeNAS or it is used with a software that does.

 

[JayG30]

Furthermore, you can use the sas2ircu command (works out of the box in FreeNAS). From what I understand this isn't SES-2 Enclosure Management related, but it does let you initiate LED's and stuff. It also appears to see the LSI SAS2X28 as well.

Code:

[root@freenas] ~# sas2ircu list
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.


         Adapter      Vendor  Device                       SubSys  SubSys
Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
-----  ------------  ------  ------  -----------------    ------  ------
   0     SAS2308_1     1000h    86h   00h:02h:00h:00h      15d9h   0691h
SAS2IRCU: Utility Completed Successfully.

[root@freenas] ~# sas2ircu 0 display
LSI Corporation SAS2 IR Configuration Utility.
Version 20.00.00.00 (2014.09.18)
Copyright (c) 2008-2014 LSI Corporation. All rights reserved.

...

Device is a Enclosure services device
  Enclosure #                             : 2
  Slot #                                  : 0
  SAS Address                             : 5003048-0-0195-78fd
  State                                   : Standby (SBY)
  Manufacturer                            : LSI CORP
  Model Number                            : SAS2X28
  Firmware Revision                       : 0717
  Serial No                               : x36557230
  GUID                                    : N/A
  Protocol                                : SAS
  Device Type                             : Enclosure services device
...
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 50030480:11e78600
  Numslots                                : 8
  StartSlot                               : 0
  Enclosure#                              : 2
  Logical ID                              : 50030480:019578ff
  Numslots                                : 13
  StartSlot                               : 0
------------------------------------------------------------------------
SAS2IRCU: Command DISPLAY Completed Successfully.
SAS2IRCU: Utility Completed Successfully.

 

[adamjs83]

Soooo, interesting find perhaps...

http://www.techcable.com/HTML/SFF-8087-cables.htm
Look at model 2136476SB-1M
SATA x4 Fanout to SFF-8087 with Side Band

However my lack of understanding of the details of this leaves me unsure if this will work since the Supermicro X10SL7-F seems to have sideband provided via separate SGPIO pins. However, I guess it is possible that the sideband could be sent in multiple ways...

I figure this boards has been so popular that someone probably has used it with a supermicro case with this or similar backplane and reverse breakout cables. So they would probably know for a fact that this works or doesn't.

Did you ever get this working?

 

[ioquatix]

Thanks for all this information.

I'm thinking of building out a server using:

- SC826TQ-R500LPB : https://www.supermicro.com.tw/products/chassis/2U/826/SC826TQ-R500LPB
- X10SRH-F : https://www.supermicro.com.tw/products/motherboard/Xeon/C600/X10SRH-CF.cfm

Am I correct in assuming there aren't enough SATA ports on the X10SRL-F?

What cables do you need to connect from the MB to the backplane?