Supermicro chassis customization, new vs used?

[chackoc]

I'm trying to determine which way to go for a 12 bay chassis. I lean towards Supermicro, but it appears to be pretty difficult to find a new Supermicro chassis outfitted with a power supply that isn't complete overkill. For me, the most promising route is the 500W 826TQ-R500LPB, but that chassis is somewhat hard to find (is it previous gen?) and every other SM chassis seems to come with much larger power supplies.

An alternative I was considering is picking up a used 12 bay chassis on ebay and swapping in one of the smaller PS options listed in this chart. This options makes me nervous though as I don't know enough about server power supplies to figure out what is swappable with what.

I also don't know if the backplanes/cages change significantly between the older chassis on ebay and the newer models. Is that pretty much all the various chassis are: case + power supply + backplane + cages? Is there anything else to them that I should think about when looking at used vs. new?

Does anyone have any thoughts on the wisdom of buying a used chassis on ebay vs buying new? Or about swapping in a smaller power supply?



The simple alternative I've also been considering is just to go with the Fractal Define XL R2 + Seasonic Fanless Platinum 400W. It has the workstation form factor I'm more familiar with, but I would lose the backplane and the easy front-facing HD access.

 

[jyavenard]

I can't comment on the use of second hand chassis.

But I just purchased two of those chassis. They are current, just that no-one stock them... Which is unfortunate as 500W power supply is still over powered for someone who only want to put 12 disks in there and a plain motherboard.... It's a platinum 93+% efficient too.
Here, the only one they do stock are the 1200W model... I'm still puzzled on what you could fit in a 2U chassis that could ever justify such a big power supply.

I think the only thing new is that the backplane supports SGPIO/SES2... which unfortunately isn't directly supported in FreeNAS/FreeBSD (unless you plan of using hardware RAID, but then why would you use freenas)

Just one advice however, make sure you get the supermicro SATA cable kit (4 cables is less than $10)... Making 12 common red sata cables fit through was a massive pain in the !@#!@#$ (it took me over an hour trying to make it neat, and in the end I gave up and picked up some SM one)

The supermicro ones are much thinner and semi-rigid it's much much easier to make them fit in, they are also made of 4 different length which suits perfectly the case..

Alternatively, you can get the E16 version, which has 2 x SAS2 mini SFF-8087 connection, it would be easier to wire to the motherboard (but you'll need the SFF-8087 -> SATA cables if using onboard SATA, or use a SAS card with 8087 connectors)

The other thing is that you'll need a place very remote where to place this chassis. The fans runs at 2700rpm minimum and can go up to 8000rm. They hit 7000rpm as soon as the CPU reaches 60 degres, in which case expect the neighbour to come and knock on your door as he can't sleep thanks to the private jet you're running.

At the end of the day, those chassis are built like a truck, so you can't go wrong second hand.

 

[jgreco]

The size of the power supply is meant both to spin the drives up (can take 2+ amps at 12v times 12 drives) plus also a dual Xeon E5 system board without groaning. The supplies are reasonably efficient, though for FreeNAS they tend to be too big since most mainboards are less than 100 watts loaded. Try to avoid the 1000w+ supplies... the single CPU E5-26xx in a 24 bay chassis here with 12 drives pulls about 250-350 watts, IIRC, so I feel the R920 supplies might be a bit small if I upgrade to 24 drives. Totally adequate for the system as it is today though.

 

[chackoc]

Thanks for the added information. I was planning on going with the X10 motherboard with the onboard LSI chip. So single Xeon and onboard connectors for all 12 drives. Unfortunately I think that means I'll be forced to run 12 SATA cables off the motherboard. But once build time comes I'll be sure and pick up the Supermicro SATA cables.

Do you know how "swappable" the Supermicro power supplies are? Is there one single form factor for all server power supplies (that long narrow shape) or do I need to be careful about tracking which power supplies mate to which chassis?

It does appear I could save some money by buying the chassis used and then swapping in a more appropriate power supply. And going along with the "built like a truck" idea, my understanding with pro grade hardware is that there's not a lot that can go wrong when buying used -- and anything that does go wrong is easily swapped.

 

[jgreco]

Be aware that there's a plethora of backplane options for the SM chassis. For example I've got an 846BE26 which has a SAS expander, which means that I can take a single SFF8087 cable and hook up all the drives. There is nothing magic that I can think of about the SM cables other than that they're reasonable quality cables.

I cannot tell you just how swappable the supplies are, sorry. However the part numbers for any given chassis are included on the chassis product pages, and you can probably see if that leads anywhere useful.

If you choose to buy used, I do suggest that you buy from someone who seems to be selling several or many units, not just one. The people who bought these a few years ago and are now moving to Amazon S3 and other cloud solutions probably just bought Supermicro prebuilts, which some recovery firm is partsing out on eBay now. No guarantees of course, but that's what I'd look for.

 

[jyavenard]

Do you know how "swappable" the Supermicro power supplies are? Is there one single form factor for all server power supplies (that long narrow shape) or do I need to be careful about tracking which power supplies mate to which chassis?

Everything is pre-assembled. The individual power supply can be monitored via ipmi (voltages, power, fans and overall status). Both of them show up.

You can very easily swap a module of the battery, just press the latch and pull it out. Though, if the chassis is mounted in a cabinet, accessing the rear may prove difficult.
As soon as you pull a battery out, the chassis alarm will start ringing until you plugged it back in. You can do it under load with no problem.

For all the time I've used redundant power supplies, I never had one fail on me so I can't tell you how easy it is to source one once they die several years later. I would assume its a standard across 2U power supplies.

 

[jyavenard]

The size of the power supply is meant both to spin the drives up (can take 2+ amps at 12v times 12 drives) .

According to the WD red spec, the maximum they can pull is 1.75A, so in this chassis that can take a maximum of 12 drives, the 500W power supply is more than enough...

Also when plugged on the LSI controller, at boot time you can clearly see that the controller spin them up one at a time...

Another thing I've noticed with zfs vs hardware raid or Linux md: on the later, when you wrote on the device, all drives light are active immediately at once...
With zfs, this almost never happens.

When I was running my benchmarks, I could see the disk activity LEDs all one for several hours while the test was going, while with zfs it was more cyclical... There's little doubt in my mind now that zfs is kinder to the disks :)

 

[jgreco]

Redundant power supplies are NOT standardized, alas. And quality varies. We used to see the ones in AIC chassis die on a semiregular basis.

The SuperMicro ones do not seem to be particularly cheap (either in terms of quality or cost to replace). But at least the part numbers are listed on the website...

 

[jgreco]

You need to be careful. 12 drives x 12 volt x 1.75A is 252 watts, but just because a supply is rated 500 watts does not mean it can deliver all of that as 12V. Better to actually do the legwork and make the calculations.

 

[jyavenard]

From what I've seen, the rating of the SM power supply seems to be how much 12V the can supply.
The 500W can supply 41.7A @ 12V (61.7A for the 740W one etc..)

In their efficiency rating document, you can see that their full load test draws 40.07A at 12V
http://www.supermicro.com/products/powersupply/80PLUS/80PLUS_PWS-501P-1R.pdf

So really, the 500W is more than efficient for the 12 bay chassis. With room to spare...

The thing that surprised me however, is that at the wall, I see the UPS+chassis using 160W, yet the UPS states 80W usage, and each power supply reports 50W.. So not sure which stuff use what...

 

[chackoc]

It sounds like if I'm careful about knowing what parts I'm getting, a used chassis plus a new, smaller power supply might be the way to go. I'll have to do more research into the backplane options but I don't think I need the newest hardware and features for my needs.

Can you guys expand a bit on the SAS backplanes you both mentioned? With the X10 board I was considering, the onboard LSI 2308 is exposed as 8 SAS ports. How would the wiring look from one of the SAS expander backplanes to this board? If stepping up to a SAS expander backplane would allow me to connect backplane to MB via just a few cables that might be worth considering further.

 

[chackoc]

I will do the math (or at least the rough calculation) once I get closer to finalizing a spec list. But from everything I've read it seems like 500W should cover me for a 12 drive system with a single Xeon MB. My concern going in was that most of the 12 bay chassis I was seeing were specced with 800W+ supplies, which seemed overly large.

 

[jyavenard]

It sounds like if I'm careful about knowing what parts I'm getting, a used chassis plus a new, smaller power supply might be the way to go. I'll have to do more research into the backplane options but I don't think I need the newest hardware and features for my needs.

Can you guys expand a bit on the SAS backplanes you both mentioned? With the X10 board I was considering, the onboard LSI 2308 is exposed as 8 SAS ports. How would the wiring look from one of the SAS expander backplanes to this board? If stepping up to a SAS expander backplane would allow me to connect backplane to MB via just a few cables that might be worth considering further.

If using the TQ backplane, then don't worry about it, just use plain SATA cables. I have that motherboard and I used the 8 ports to the LSI adapter , then 2 of the SATA3 + 2 of the SATA2 from the intel controller.

All I'm saying is that the available hole to pass all the SATA cables is very narrow (you can't pass cables underneath the 3 fans, there's no space), to while you can do it with your standard SATA cables; it's a pain. The SM ones are round and very thin, they are easier to handle it looks much neat & nicer.

http://2.bp.blogspot.com/-HJ_89P9SToU/UoMxZjoAy1I/AAAAAAAACbA/_kbuZvXDg74/s1600/IMG_1282.jpg
I've managed to pass one SGPIO underneath the fan, but it's squashed (which for SGPIO doesnt matter at all, and it's only 4 wires)

I did the math, and the 500W mode is plenty sufficient, with RED drive, even at pick you barely pass the 50% load.

Looks like we're going to end up with the same setup !

 

[jgreco]

Wow that board is floatin' around in there :)

 

[jyavenard]

hehe..

I just made some power measurement...

Chassis is loaded with only 6 drives right now.
Boot peak: 150W
One PSU only: 72W
Two PSU, one not inserted: 84W
Two PSU loaded: 92W

that was measured at idle..

So the power redundancy cost me $43.8 a year (25c/kWh here)

Edit: the power supplies report 37W in, 27W out (the other 30W in, 41W out) so we are at less than 10% load with an efficiency of only 73%...

 

[jgreco]

Fans running full on the supplies? I think I determined that accounted for maybe 10W when adding the 2nd supply...

 

[jyavenard]

all fans are connected via the motherboard. The PSU themselves have 2 fans each, which you can hear running at all time (even with the PC "off" (IPMI is on)). IPMI states fan1 is rotated at 3840prm.

At the time I measured idle; the chassis fans were running at 2700rpm each (there are three of them).

Still, there's a long way to go between what I measure (71W) and what the PSU reports (37W)

 

[jgreco]

The PSU fans are what I meant.

 

[JohnK]

SuperBiiz seems to have stock http://www.superbiiz.com/detail.php?name=CA-826TQ5B

 

[chackoc]

Yeah, that's roughly the same price I was seeing elsewhere. I'm leaning towards buying the chassis used though. It looks like I can either save a fair bit of money or perhaps stretch to a larger chassis by going the used route.