SuperMicro SuperStorage Superserver - which backplane is compatible?

[DirkTripleD]

Hi all,

thank you for the very interesting and useful information in this forum!

I built a SuperMicro server for a customer some months ago, intended for a video postproduction system for video editing in full HD with Adobe Premiere Pro.

The system should have the ability to use it for 4K video editing in the future.

It is consisting of this parts:

SuperMicro SuperServer 7049P-TRT
https://www.supermicro.com/en/products/system/4U/7049/SYS-7049P-TRT.cfm

Mentioned backplane here:
SAS / SATA Hard Drive Backplane w/AMI MG9072

Included board: SuperMicro X11DPH-T
https://www.supermicro.com/en/products/motherboard/X11DPH-T

Which ist this chassis:

SuperMicro SuperChassis 745BAC-R1K28B2

SC745BAC-R1K28B2 | 4U | Chassis | Products | Super Micro Computer, Inc.

745BAC-R1K28B2

www.supermicro.com

Mentioned backplane here:
HD Backplane BPN-SAS3-743A 8-port 4U/Tower 3.5-inch SAS3/SAS2/SATA3 12Gbps backplane for SC743/SC745 chassis

I bought some additional 8x 2.5-drive-bays-cages with and without fan (I was not able to decide which one I prefer most) ;) when I bouht the initial server. They are not in use yet.

Just for the records:

I bought

1x INTEL Xeon Scalable 4208 2,10GHZ FC-LGA3647 11M Cache 10.4GT/sec Box CPU
2x 32 GB ECC-RAM DDR4 32GB Samsung M393A4K40CB2-CVF DDR4-2933 regECC DIMM CL21-21-21 Single (bulk)
1x SAMSUNG MZ-V7P1T0BW 970 PRO 1 TB NVMe M.2 for the cache
6x SEAGATE Ironwolf PRO NAS HDD 4TB SATA

The system is working flawlessly.

The customer now wants to expand the system and I am searching for a couple of weeks to find a new rack enclosure.

I found different systems, some with included motherboard and so on. But my hardest problem is to figure out, which backplanes are compatible and recommended for TrueNAS with ZFS.

These ones are available (I could also mention the systems if you want):

In bold I highlighted elements for which I am unsure if they are good or bad!

This system is advised from a SuperMicro reseller, but I think he is not familiar with TrueNAS:

SKU CSE-846XE2C-R1K23B = SuperChassis 846XE2C-R1K23B = SC846XE2C-R1K23B
https://www.supermicro.com/en/products/chassis/4U/846/SC846XE2C-R1K23B

24 x 3.5" hot-swap SAS/SATA drive bay with SES3 - Mini SAS HD (SFF 8643) connectivity

24-port 4U SAS3 12Gbps dual-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD (secondary expander port only connects to dual-port SAS devices, single-port SATA will not have this redundancy feature)

https://www.supermicro.com/en/products/chassis/4U/846/SC846XE2C-R1K23B

He also mentioned this system:

24 x 3.5" hot-swap drive bay

24x 3.5" hot-swap SAS/SATA drive bays supporting SAS3/2 or SATA3 HDDs with 12Gbps throughput

Backplane: 24-port 4U SAS3 12Gbps single-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD

https://www.supermicro.com/en/products/chassis/4U/846/SC846BE1C-R1K23B

This are the results of my own investigation (I of course would like to use actual X11-hardware and have also to check which ones are available):

24 x 3.5" hot-swap drive bay -

24-port 4U SAS 6Gbps direct-attached backplane, support up to 24x 3.5-inch SAS2/SATA3 HDD/SSD

https://www.supermicro.com/en/products/chassis/4U/846/SC846BA-R920B

24 x 3.5" hot-swap SAS/SATA - Mini-i-Pass (SFF 8087) connectivity

24-port 4U SAS 6Gbps direct-attached backplane, support up to 24x 3.5-inch SAS2/SATA3 HDD/SSD

https://www.supermicro.com/en/products/chassis/4U/846/Sc846XA-R1K23B

24 x 3.5" hot-swap SAS/SATA drive bay with SES3 - Mini SAS HD (SFF 8643) connectivity

24-port 4U SAS3 12Gbps dual-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD (secondary expander port only connects to dual-port SAS devices, single-port SATA will not have this redundancy feature)

https://www.supermicro.com/en/products/chassis/4U/846/SC846XE2C-R1K23B

24 x 3.5" hot-swap drive bay - Mini SAS HD (SFF8643) connectivity

24-port 4U SAS3 12Gbps single-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD

https://www.supermicro.com/en/products/chassis/4U/846/SC846XE1C-R1K23B

24 x 3.5" hot-swap drive bay

24x 3.5" hot-swap SAS/SATA drive bays supporting SAS3/2 or SATA3 HDDs with 12Gbps throughput

Backplane: 24-port 4U SAS3 12Gbps dual-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD (secondary expander port only connects to dual-port SAS devices, single-port SATA will not have this redundancy feature)

https://www.supermicro.com/en/products/chassis/4U/846/SC846BE2C-R1K23B

24 x 3.5" hot-swap drive bay

24x 3.5" hot-swap SAS/SATA drive bays supporting SAS3/2 or SATA3 HDDs with 12Gbps throughput

Backplane: 24-port 4U SAS3 12Gbps single-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD

https://www.supermicro.com/en/products/chassis/4U/846/SC846BE1C-R1K23B

Attention: Low-Profile-Expansion-Slots!?

SuperStorage 6049P-E1CR36H with Super Board X11DPH-T

SAS3 (12Gbps) via Broadcom 3108 AOC; HW RAID; RAID 0, 1, 5, 6, 10, 50, 60

36 Hot-swap 3.5" SAS3/SATA3 drivebays including 4 optional NVMe drives [Great!!!]; 2 hot-swap 2.5" SATA3 drive bays (rear) [Great!!!], 2 M.2 NVMe, Expander based backplane [Great!!!]

https://www.supermicro.com/en/products/system/4U/6049/SSG-6049P-E1CR36H.cfm

SuperStorage 6049P-E1CR36L with Super Board X11DPH-T

SAS3 (12Gbps) via Broadcom 3008 AOC, Broadcom 3008 controller, SATA3 (6Gbps) with RAID 0, 1, 5, 10

36 Hot-swap 3.5" SAS3/SATA3 drive bays including 4 optional NVMe drives; 2 hot-swap 2.5" SATA3 drive bays (rear), 2 M.2 NVMe, Expander based backplane [Again: Great!!!]

Maybe I will add a identical second CPU and will double the RAM. I would like to preserve the board but I think it could be in the budget to buy a new system with a preinstalled new board maybe.

Every advice is welcome!

Thank you very much!

 

[Chris Moore]

You could use an external drive shelf like this cabled to your existing server to give you more drive bays:

SC946ED-R2KJBOD | 4U | Chassis | Products | Super Micro Computer, Inc.

946ED-R2KJBOD

www.supermicro.com

 

[Chris Moore]

SAS3 (12Gbps) via Broadcom 3108 AOC; HW RAID; RAID 0, 1, 5, 6, 10, 50, 60

No hardware RAID. Never use that with FreeNAS / TrueNAS

 

[Chris Moore]

Looks like you need to read the ZFS primer on what SAS is good and what is bad... Please read this:

Don't be afraid to be SAS-sy ... a primer on basic SAS and SATA

With the introduction of SAS 12Gbps, seems like "it's time" to do a braindump on SAS. Work in progress, as usual. History By the late '90's, SCSI and PATA were the dominant technologies to attach disks. Both were parallel bus multiple drop...

www.truenas.com

 

[DirkTripleD]

Hello Chris Moore, thank you for your answer.

I am sorry I forgot to mention that I do not want to put the tower into a rack with the conversion kit and I do not want to use it with an additional JBOD enclosure.

Two decades ago I sold SuperMicro motherboards with own tower chassis, but the last two decades I mainly sold complete DELL an HPE servers and I was a little bit 'shocked' of the difference between the SuperMicro chassis and those ones I have been familiar with the last two decades. I really love well designed servers and I am a little bit disappointed of the tower chassis. It is the same with the IPMI, I thought the functions are nowadays the same but I have been wrong. I thought of using OpenBMC / OpenIPMI but that would take too much time.

Thank you for the useful link in post 4. I know the chassis for about 100 harddrives very well but the price is to high for this project and as I know the hard drive caddys are very expensive here in Germany. When I had a look at it months ago they sold it for about 30 US-Dollar each.

I could read the manual of the Broadcom 3108 AOC, if there is any option to set it up in HBA mode. But I already decided not to use it. Only if someone here in this forum would tell me 'that it is a great controller for TrueNAS, just switch on the only HBA option' and so on... But I do not expect that.

This thread was very interesting:

SOLVED - Help Connecting Supermicro Backplane to HBA

Hi, I recently purchased a Supermicro SC847 BE1C4-R1K23LPB for a server I am building, yet I am not sure how to connect the drives on the backplane to the HBA I have. The backplane has 4 ports, but there is space for 24 drives, and I am not sure how to connect them to the two LSI SAS 9305-16i...

www.truenas.com

Maybe someone could tell me which variant of the backplane is the most preferable one for TrueNAS concerning SuperMicro boards:

Single-expander backplane, dual-expander backplane or direct-attached backplane?

Thank you very much!

 

[SeedyRom]

Hi Dirk,
I am newer to these forums but I have a lot of experience in this field. I believe I can answer each of your concerns (hopefully in a way that is understandable ).

I found different systems, some with included motherboard and so on. But my hardest problem is to figure out, which backplanes are compatible and recommended for TrueNAS with ZFS.

The first thing I want to mention is that there is no backplane that is "incompatible" with any OS. The backplane is a fancy SAS expander and nothing more. It needs an HBA or RAID adapter to sit between the backplane and the OS in order to function properly. The thing that is important to the OS would be if the HBA/RAID card is compatible. Although, there may be features that a backplane offers which you may not have the ability to utilize if the OS doesn't support it. I'll explain what I mean in the next answer..

24-port 4U SAS3 12Gbps dual-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD (secondary expander port only connects to dual-port SAS devices, single-port SATA will not have this redundancy feature)

A dual-expander backplane is a feature which only makes sense in the correct circumstance. If you are locked in with using SATA drives and not SAS then this is a worthless upgrade for you.

Typical backplanes with single expanders will support multiple SAS expansion connectors and cascading (connecting additional backplanes or disk shelves together).

With dual-expander backplanes, this has an added feature of supporting failover. To do this, you would install one SAS expansion cable to each backplane controller. In a failure scenario the connection will fail over to the other controller seamlessly. You can also install two HBA cards and connect each to both controllers for even more redundancy if needed. In order for this to work though, you need to set up multipathing which is a feature that is only supported by SAS drives. You could still install SATA drives into the backplane, but you wouldn't be able to use the secondary controller.

Also, TrueNAS does support SAS multipathing.

Backplane: 24-port 4U SAS3 12Gbps single-expander backplane, support up to 24x 3.5-inch SAS3/SATA3 HDD/SSD

If you are using SATA drives, the single-expander backplane is what I would recommend.

24-port 4U SAS 6Gbps direct-attached backplane, support up to 24x 3.5-inch SAS2/SATA3 HDD/SSD

Direct-attached backplanes is something I would not recommend. Each port on an HBA card would be dedicated to four drives only. This is inefficient and can create headaches if you run into failures. You would also need multiple HBA cards to support 24 disks with no chance of cascading into another shelf.

24 x 3.5" hot-swap SAS/SATA - Mini-i-Pass (SFF 8087) connectivity

24-port 4U SAS 6Gbps direct-attached backplane, support up to 24x 3.5-inch SAS2/SATA3 HDD/SSD

This would be your worse option by far. SFF-8087 is the connector that most HBA or RAID cards use. The Mini-i-Pass is a Supermicro naming which is basically a breakout of an HBA port to four SATA connectors. This would require six SAS expansion ports and you would have a spaghetti of SATA cables to manage.

24 x 3.5" hot-swap SAS/SATA drive bay with SES3 - Mini SAS HD (SFF 8643) connectivity

SCSI Enclosure Services. This is a SAS specific feature where the backplane can grab diagnostic info about the SAS drives.

SAS3 (12Gbps) via Broadcom 3108 AOC; HW RAID; RAID 0, 1, 5, 6, 10, 50, 60

If you aren't going to do a hardware RAID, don't use a RAID card. They are more expensive and usually an HBA is faster than a RAID card is when acting like an HBA. You can usually buy an HBA card with more expansion ports for less than what a RAID card would cost. More ports could potentially give you a faster throughput or would allow you to connect another backplane without having to install more cards.

SAS3 (12Gbps) via Broadcom 3008 AOC, Broadcom 3008 controller, SATA3 (6Gbps) with RAID 0, 1, 5, 10

See above. This card only has two expansion ports. You'll spend far less on an HBA card with four ports instead.

Maybe I will add a identical second CPU and will double the RAM. I would like to preserve the board but I think it could be in the budget to buy a new system with a preinstalled new board maybe.

From a cost perspective, I'm in the same boat as Chris. Just buy an external HBA card with mini-SAS HD connectors and a JBOD. It'll be cheaper than building a new unit and adding more drives to an existing TrueNAS environment is pretty painless. Of course, if you just want to replace the chassis because you don't trust it's ability to meet future needs I can get behind that too. I would recommend looking at what the customer's future growth looks like. Maybe it would make sense to buy a 1U or 2U server as a controller and attach a JBOD to it. This would give you the ability to expand in the future with a redundant controller or additional JBODs. You also will have more redundancy in almost every way and you can upgrade or replace one device instead of having to rebuild the whole thing again.

 

[Chris Moore]

I like the top loading options from SuperMicro. https://www.supermicro.com/en/products/top-loading-storage

We have bought ten of them in my branch in the last three years. Nice systems. Just be sure to get the ones that have IT mode SAS controllers.

This is a review on the next generation top loading system, newer than what we have, but it looks really good:

New Supermicro 60-Bay Top Loading Storage Server Hands-on

We get some hands-on time with the new Supermicro 60-bay top loading storage server which features a new focus on serviceability and flexibility

www.servethehome.com

Based on our experience with the ones we bought in 2019, I would expect these to work very well.

This is a link to the model that we purchased:

Supermicro SSG-6049P-E1CR60L 4U Storage Barebone Dual Processor

Barebone 4U Rackmount SuperServer, Dual Intel Xeon Scalable Processors Gen.2, Intel C621 chipset, Up to 6TB DDR4 ECC 2933MHz memory, Broadcom 3008 SAS3, SW RAID (IT Mode), 60 Hot-swap 3.5in drive bays, 2 Rear hot-swap 2.5in drive bays, optional 6 NVMe bays, SIOM support for flexible networking...

www.wiredzone.com

Because there are so many drives, you can create a lot of vdevs. That allows for greater IOPS across the pool. Our workload involves a lot of random seeks for small files and these systems make it possible to get decent performance in that workload. I am looking to expand the pool to even more vdevs with additional disk shelves.

 

[DirkTripleD]

Hi and thanks to all again!

I finally bought a SuperMicro SSG-6049P-E1CR24L with included motherboard SuperMicro X11DPH-T and it works like a charm!

In my opinion it is a highly recommendable system for TrueNAS.

I just put the INTEL Xeon Scalable 4208 CPU and the RAM from the 'old' system (based on SuperMicro X11DPi-NT motherboard) to the new one, booted it from the existing USB-stick and it is working without any issues.

I do not know if this is 'best practice' or not, but if there will be any upcoming issues I just will perform fresh install and a configuration restore.

Before making these changes I upgraded the old FreeNAS system to the newest TrueNAS version.

I had some problems (I am missing some comfortable features) with SuperMicros IPMI because I am more familiar with DELLs iDRAC and HPEs iLO and so on.

I am thinking about buying an AMD EPYC CPU based system (with SuperMicro H12SSL-CT or H12SSL-NT which both seem to be capable of SuperMicros newer IPMI, unfortunately right now only for single processors) for testing purposes, in a smaller (maybe quieter) Chenbro tower chassis for my home office.

Kind regards

d i r k