Connecting Backplane properly at Supermicro 846BE1C-R1K28B

[Scampicfx]

Hey folks,

I just orderer the items for my first big FreeNAS Server.

I ordererd Supermicro X10SRH-CF Mainboard and 846BE1C-R1K28B Chassis.
Link Mainboard: https://www.supermicro.nl/products/motherboard/Xeon/C600/X10SRH-CF.cfm
Link Chassis: https://www.supermicro.nl/products/chassis/4U/846/SC846BE1C-R1K28B

I would like to ask if it is the correct way, to use two SAS-cables, in order to connect the onboard LSI3008 to the backplane of the chassis? I had a look into the manual of the chassis, but i'm not quite sure, if I got it right.

Appendix E-1 of the Chassis Manual describes the backplane.

On page E-9 there is a picture of connecting the backplane:


(There are no cascading systems in my case...)

Question: Is it correct to put two SuperMicro CBL-SAST-0593 (SFF-8643 to SFF-8643) into one J50 port of the backplane?
I'm not sure if I need to use two or only one cable... In my opinion J50 is only one port, so it is able to handle only one cable, isn't it?

Thanks so much for your help! :)

 

[BigDave]

Back when I purchased a 16bay SM chassis (my first rackmount chassis),
I was the most comfortable with getting that type of information directly
from the backplane's manual. You can usually find them in PDF format
and I found the information to be much more detailed than the chassis
manual.

Found the link to your BP manual here.
https://www.supermicro.com/manuals/other/BPN-SAS3-846EL.pdf

according to a quick study, that BP only requires a single SAS cable for
running all 24 drives, as it has a "single expander".

 

[Scampicfx]

Dear guys,

@ BigDave: Thanks for your time reading the manual!

Yesterday I assembled chassis, mainboard, etc. and had a closer look to the backplane. In case anyone comes across this posting:

Even the single expander backplane has 4 SAS ports. Two incoming SAS ports and two outgoing SAS ports for cascading systems.

This means you need to use two cables in order to connect mainboard and backplane properly and at full speed. :)

 

[BigDave]

You would be correct using two cables if you have a backplane with the designation EL2.
If your backplane is designated EL1, (I'm referencing the link you provided) you can use only one cable.
FYI the EL2 model uses the second SAS cable for failover only, it does not increase bandwidth.

 

[Scampicfx]

Dear BigDave,

as mentioned, I am using the single expander backplane and it has 4 SAS ports. Here's the link to the chassis:
https://www.supermicro.nl/products/chassis/4U/846/SC846BE1C-R1K28B (in the chassis manual there are also some details about the backplane).

Of course, it is possible to use only one cable. But Supermicro recommends using two cables for max bandwidth.

FYI the EL2 model uses the second SAS cable for failover only, it does not increase bandwidth.

This depends on where you place the second SAS cable...

When you are using EL2 backplane and setup your cables like the following picture, then yes, in this case, you are only using the second cable for failover scenario!



However, when you are using a setup like in the following picture, you increase your bandwidth! This scenario is possible both with EL1 and EL2 backplane.



The main point I would like to emphasize is, that the connector J49 consists of two SAS ports as well as connector J50 consists of two SAS ports (in total 4 SAS ports, also at single expander backplane).
As you can see in the picture, Supermicro uses two cables to connect all SAS lanes coming from the HBA to the J49 ports. First SAS cable carries SAS 0-3 and second SAS cable carries SAS 4-7, which are 8 in total.

That was the point I asked in my first posting and which now was solved... I wasn't sure if J49 is a single SAS port or if it consists of two! I wanted to share this information in case it might help other users here in the forum! :)

Cheers,
Scampicfx

 

[BigDave]

Well I will have to eat Crow here because I missed (missed it totally) the fact that
your backplane was SAS3.
My sincere apologies Sir!

 

[ioquatix]

Thanks, I'm in the same boat - I wasn't sure. This was useful. I'll try it and report back. Going to buy two cables :)

 

[mifronte]

@Scampicfx

Nice case. Can you tell be about the drawer in the rear of the chassis for HDDs? In the manual on page 4-7, it shows a drive drawer, but I do not see it in the picture on the SuperMicro web site for the 846BE1C-R1K28B. So does the chassis comes with the drive drawer?

 

[Scampicfx]

Dear mifronte,

the chassis you linked to ( SC846BE1C-R1K28B ) does not include this drive drawer option.

This drive drawer is only supported by the new chassis SC846XE1C-R1K23B or SC846XE2C-R1K23B or SC846XA-R1K23B.

It's a bit confusing because all Supermicro SC846 chassis share the same manual.

 

[ioquatix]

In New Zealand, the price for the 826 BE1C was $1800NZD, and the 846 BE1C was $2800. So, given that the 826 suits my requirements I've decided to go for it. Just FYI, I've attached the full quote (in NZD). 12 drive bays is sufficient. One of the reasons I like that 846 is because it supports full profile PCIe cards. But, honestly, I can't see myself using that in a storage server. If I want compute power, I'll probably build a decent workstation in the next 6-12 months and have it near my desk rather than in a rack in the garage (there are also low-profile GTX 1050Ti cards available so perhaps those might be useful for basic compute stuff).

 

[NAS-Plus]

This may be a bit off-topic but if you don't mind I would like to ask.

How much power would a server as is proposed be likely to use? I presume that this would be measured in watts (W) total or watts (W) per power supply?

The two redundant power supplies appear to be 1280W each or 2560W total. From the SM website:
1U 1280W Redundant Platinum 95%+ Super Quiet Power Supply with PMbus

I'm sure that the typical power consumption would be much less than the rated power, but what might it typically run? I realize that the power consumption would be affected by the number of installed hard disk drives. I Would like to assume an installation with 12 and with 24 drives.

 

[ioquatix]

So, if you like I can measure the power usage and report back. I've got a power meter which you put inline with the plug.

The power delivery in the chassis I got is 920Watt and there are two redundant power supplies. I may only have one plugged in to save power.

My rough guess is about 5-10 watts per drive, about 50 watts for the motherboard, and so total power usage about 100-200 watts.

 

[NAS-Plus]

Thanks, this is helpful. I wonder if there is a way to even more precisely forecast the energy consumption.

 

[ioquatix]

So, I think when it comes to spinning disks, the power consumption profile is very non-uniform.

Big disk arrays used to avoid spinning up all disks at once as the initial power draw was (is?) fairly huge - so much so that spinning up all disks in a 12/24 drive system might draw enough current to damage the PSU or blow a fuse.

At a guess (and considering only 5200RPM drives), I'd say that spinning up a disk initially might use anywhere from 20-40 watts. If you spin up 24 disks, and assume 40 watts per drive, thats 960 watts of power required. Once the drives are spinning, and it's typical to idle anywhere from 2-6 watts. When in use, maximum power draw is likely to be between 5-10 watts for standard low power consumer NAS drives.

If you think about this in the context of a drive going 3x the speed (10k or 15k RPM), you might in fact find it draws MORE than 3x the current.

So, power supplies in servers do need to be able to handle a large initial load, but still provide high efficiently for idle times.

If you review the specific specs of the PSU, the drives, the motherboard, CPU, backplane, memory you can probably make a fairly accurate assessment of startup, idle and load power usage. Keep in mind, that while a CPU might use 10-150watts, DRAM modules that need to be constantly refreshed also draw a lot of power, between 0.3-0.6 watts per GB, so for a 32 GB system, that's actually between 10-30 watts which is not insignificant. If you are building a larger system, e.g. 64, 128, 256, it's a significant part of the power budget.

 

[NAS-Plus]

Excellent. That is very helpful. I'm more concerned with the day to day idle and in-use power consumption. I can see though that the startup power requirements would be important to consider too.

You mentioned previously that using only one power supply of the two redundant power supplies would save some power. I have done this myself. Any idea though how much power is typically saved in this situation?

 

[Stux]

LSI 2008 cards draw 15W too. Heavy duty server cans draw significant amounts too.

A dual PSU can actually provide double the current when both modules are in. If you require more than a single PSU to boot then remember you can't reboot/power up the machine when one of the PSUs are out.

Large single CPU 24 bay system should be fine with circa 1000W PSU.

And probably idles well below 300W. Pity I've lost my watt meter thing ;)

 

[ioquatix]

I'm hoping my server idles around the 80-120 watts in it's initial configuration. I'll report back in about two weeks once it's set up and I'll specifically measure the single/dual PSU configuration power draw.

 

[danb35]

So, if you like I can measure the power usage and report back. I've got a power meter which you put inline with the plug.

There's a very good chance you'll be able to see the power consumption via IPMI without the need for any plug-in gadget. It will even give historical graphs for at least the last week, showing high, low, and average over that timeframe, like this:

Edit: I wouldn't recommend removing one of the power supplies--the impact on your power consumption will be marginal.

 

[NAS-Plus]

Excellent. I did not know that the IPMI provided this information.
Is this from the SuperMicro server listed in your email footer information (dual Xeon, 12 drives)?

 

[danb35]

Is this from the SuperMicro server listed in your email footer information (dual Xeon, 12 drives)?

Yes--18 drives, actually.