Supermicro CSE-846 build - Only boot satadom visible to TrueNAS

[danielstonehouse]

Hi All,

Long term user of two SUPERMICRO truenas purchased through IX systems. Had a bunch of 3TB SAS drives sitting unused after upgrading (got a great deal on 25 4tb SAS drives) - so attempted my own build.

I purchased a second hand

Supermicro SuperChassis CSE-846
Supermicro X9DRI-LN4+
2 x E5-2609v2 Processors
Memory - 128GB ram
Dual Intel 1GB
Dual Intel X540-T2 10GBase-T RJ45

I also sourced a SATADOM separately.

TrueNAS installed and running.

My issue is I can't see any drives in TrueNAS (aside from the SATADOM).

Only 4 flashing blue lights out of 24 drives installed in the backplane. Which correspond to the 4x Seagate Constellation ES.3 Drives. The western digital drives (WD Re SAS drives which have previously been used in my other TrueNAS boxes) show no signs of life.

(also the 10GB card, while visible to the system, doesn't seem to show any activity when plugged into the switch - but thats a problem for a later date).

My hunches:

- Internal Cabling? Only one blue SAS cable going from Backplane to motherboard
- Power requirements for the drives?
- HBA - Should I buy a HBA (or 2) instead of connecting directly to motherboard? If so what spec / ports / cables do I need to maximise performance across 24 drives
- Jumpers on Backplane?


Thanks - would appreciate any input. Have overcome a few hurdles already on this build, and have hit a bit of a wall.

 

[HoneyBadger]

Supermicro X9DRI-LN4+

Assuming you mean the X9DRi-LN4F+ that model has no onboard SAS controller; you'll need to add at least a single HBA to access your drives.

Please also identify your drive backplane model eg: "BPN-SAS2-846EL1" as it will determine if you would be able to benefit from a multiple HBA setup or if you have only a single upstream connection.

 

[danielstonehouse]

Assuming you mean the X9DRi-LN4F+ that model has no onboard SAS controller; you'll need to add at least a single HBA to access your drives.

Please also identify your drive backplane model eg: "BPN-SAS2-846EL1" as it will determine if you would be able to benefit from a multiple HBA setup or if you have only a single upstream connection.

Thanks, what's my best approach to definitively identify the backplane? I can see a serial number.

I suspect it may be a SAS-846-EL

 

[artlessknave]

perhaps you meant X9DR3-LN4F+? if so, that's intel SAS(1?); probably not super useful.
definitely need to know the backplane. first thing I can think of is you have an "A" backplane, not an expander (EL1/EL2).
iirc, the expanders usually have 2 SAS ports very close to each other, but the "A" direct connect has them all over the board.
you almost catch the model# in the picture but just not quite...

 

[danielstonehouse]

perhaps you meant X9DR3-LN4F+? if so, that's intel SAS(1?); probably not super useful.
definitely need to know the backplane. first thing I can think of is you have an "A" backplane, not an expander (EL1/EL2).
iirc, the expanders usually have 2 SAS ports very close to each other, but the "A" direct connect has them all over the board.
you almost catch the model# in the picture but just not quite...

It does seem to have a number of connectors across the backplane board

 

[artlessknave]

I became curious and went on a dive and looked at supermicros X9DR boards....and I think you are right about the board, but that board definitely doesn't have SAS, it's using a miniSAS connector for 4 SATA2 ports, probably the "SCU SATA"

Supermicro | Products | Motherboards | Xeon® Boards | X9DRi-LN4F+

 

[danielstonehouse]

It would appear the forum ate a few messages - so to summarise:

With @artlessknave help (and some rummaging with a cheap endoscopic camera) - I have identified the backplane as a bpn-sas-846A!

My understanding is this passes through 4 drives on each of 6 ports on the backplane

I have purchased 3x 9272-8I 6Gbs PCI-E HBA FW: P20 ( LSI 9207-8i IT Mode

and 6x SAS SFF 8087 Cables

 

[artlessknave]

ummm. it looks like 9272-8I is megaraid, are they pre-crossflashed?
also, note that my original directions were not quite right, a single expander and SAS card would have actually been insufficient anyway.

for posterity, a reconstruction of the missing posts from the forum notifications and our conversation, for both my frayed sanity and anyone else who ends up here:

Can't see a model name on the board at all - but it does indeed look like an "A" backplane based on the description.
Here is a rather janky video showing the many connectors.
Distinctively the connector one on the far left (which had cables plugged into it when purchased, with the other end unplugged - which I figured should be plugged into the motherboard) faces up and the other staggered connectors face down.
https://www.dropbox.com/s/6s04e9ahw7aaj8t/IMG_1112.MOV?dl=0
I just purchased an LSI 9207-8i (IT mode)
Anything more possible than the single connection?
Any concerns about 24 x 3tb drives?
I was hoping for big, fast storage (ie a stream or two of 10g) - but big, slow storage is useful as well!

I think the forum was having issues and it ate our last two messages! The email summary is all I have.
Absolutely you would think the model name would be on there somewhere but I just cant spot it.
Could you make a guess? I'd love to be able to look it up by model name
View attachment 53397

bottom right of this image. now sure how that orients into a chassis, i dont have one of these styles myself. i was looking at getting one which is why I know about the backplanes.
https://www.servercase.co.uk/Images/DBImages/Products/68178_1.jpg
i dunno wtf is up with the forums. I tried to get to Jira too and that failed.
i ususally delete my notifications so i only have some of those....might change that habit...

looks like its on the very left, under the metal and data cables, but seeing the 6 miniSAS makes me pretty confident that its an "A" type, none of the others should have that layout.
the hidden content was:
bpn-sas-846TQ 24x SATA connector direct connection
bpn-sas-846A 6x miniSAS direct connection
bpn-sas2-846el1 2x mini sas single expander
bpn-sas2-846el2 2x mini sas single expander
bpn-sas3-846ab1 (i dont remember exactly, but they are different from el1/2)

the "A" and TQ can technically operate at SAS3, so they have higher overall performance potential than the sas2 expander ones - functionally they are wires. the A one is nicest since you need 6 cables instead of 24.

 

[danielstonehouse]

Yes the cards are all pre flashed to “IT MODE”

I also purchased a Chelsio 10g card - as that’s the card working well in my other IX Systems TrueNAS boxes… sometimes ebay is the quickest path to solving tech issues!

 

[artlessknave]

mmm, yes. i have newer chelsios in my freebsd stuff (pfsense, truenas). that I wish solid performance from

 

[trumee]

@danielstonehouse What is the power consumption for your system?

 

[artlessknave]

I would estimate ~200W idle. possibly up to 400W-ish full bore.
I would be very curious how good my estimate actually is though.

 

[danielstonehouse]

What’s the best way to test power usage? Do I need special hardware?

 

[danielstonehouse]

Put everything together today and working perfectly!

All very smooth (apart from having to improvise a full height PCI bracket for the 10G card - ugly, but it works)!

I went with 3x 8 disk vdevs at z2 - might be redundancy overkill but I am using older used SAS drives

 

[artlessknave]

What’s the best way to test power usage? Do I need special hardware?

usually, yes. i have a discontinued ubiquity power bar i use. I had a kill-o-watt but it doesnt work anymore.

 

[danielstonehouse]

Oh also - right angled mini SAS cables sound like a good idea… but straight would have been easier and better!

 

[artlessknave]

right angled mini SAS cables

ya, it depends on where they are going. you usually dont realize you have the wrong ones until...you have the wrong ones.