Supermicro Build - Using most of the hardware

[Mirfster]

So then based on that info, something like this should do out of the box. Maybe more RAM and obviously HDDs.

http://www.ebay.com/itm/Dell-PowerE...088919?hash=item3a945ae897:g:BdQAAOSwjyVT0D5l

Um, Yes and No... :)

I am very familiar with that seller and their stuff is top notch. At least two (if not three of my systems came from them).
With that said:

1. The Perc H700 is not going to work. It can't be cross-flashed to JBOD/IT Mode; so it is a waste and you would have to replace it with a card that could be (IBM M1015, Perc H200, LSI 9211 etc.)
2. They are only providing one CPU; which more than likely means that they are not including the heatsink for a 2nd CPU. Then you would have to purchase that later if you decided to add another CPU
3. It only has 8 GB of RAM (not the greatest; especially for the price)
4. Doesn't really say if it includes the drive tray (and screws, you always want the screws too). While picture shows them, the description does not..
   
5. Overall seems a bit pricey for what they are including

The one I originally linked is the better deal. I have bought from them as well.

1. Has the H200 already, so all you have to do is cross-flash it (maybe if you ask they will even cross-flash it for you; not really needed though since it is pretty simple)
   
2. Has two CPUs with heatsinks
   
3. Has 6 times the RAM
4. Does include the trays (still ask and make sure you get the screws as well)
5. Overall same price as the one linked (with shipping and all)
6. Comes with rails (good if you are rack mounting)
7. This is a "Turn Key" solution IMHO

Edit: I just looked at the pictures as well. In the 3rd one it appears as if the H200 they are including is the "Mezzanine" model (connects directly to the Motherboard instead of using a PCIe Slot). That is a nice surprise, since alone that card is ~$130.00 itself. I use them and have bought most of mine afterwards. Example: search eBay for the item which is part number : 85M9R

 

[depasseg]

Can you shed some light on what type of controller/backplane I need to be looking for, for 3TB+ support? Such as SATA 3.0gbps support or such.

You want SAS-2 (6gpbs).

So look for a supermicro with "E16" in it. (the 6 designates 6gbps)

 

[Mirfster]

Just to ease your mind on the C2100/FS12-TY; I contacted the Seller:

Code:

From: esisoinc
To: mirfster
Subject: Re: Details about item: mirfster sent a message about DELL FS12-TY C2100 2x QUAD CORE L5630 2.13GHz 48GB RAM 12x TRAY H200 #131728700629
Sent Date: Feb-26-16 10:09:42 PST
Dear mirfster,

we dont have it sorry.
- esisoinc

From: mirfster
To: esisoinc
Subject: Re: Details about item: mirfster sent a message about DELL FS12-TY C2100 2x QUAD CORE L5630 2.13GHz 48GB RAM 12x TRAY H200 #131728700629
Sent Date: Feb-26-16 10:05:20 PST
Dear esisoinc,

Thanks, was wondering about the Mezzanine H200, since in the pictures (#3) it definitely looks like it from where the cables are coming from and nothing is in the PCIe slots. :)

Not a big deal either way. I appreciate your fast response.
- mirfster

From: esisoinc
To: mirfster
Subject: Re: Details about item: mirfster sent a message about DELL FS12-TY C2100 2x QUAD CORE L5630 2.13GHz 48GB RAM 12x TRAY H200 #131728700629
Sent Date: Feb-26-16 09:43:11 PST
Dear mirfster,

1) Does the backplane on this model have the two Mini-SAS connections (SFF-8087)? yes
2) Will the system support hard drives over 2TB? yes
3) Does the system come with the screws for the Trays/Caddies? yes
4) The H200 that is included, is this the "Mezzanine" model? no
- esisoinc

From: mirfster
To: esisoinc
Subject: Details about item: mirfster sent a message about DELL FS12-TY C2100 2x QUAD CORE L5630 2.13GHz 48GB RAM 12x TRAY H200 #131728700629
Sent Date: Feb-26-16 09:34:34 PST
Dear esisoinc,

Hello, can you please provide a couple answers?

1) Does the backplane on this model have the two Mini-SAS connections (SFF-8087)?
2) Will the system support hard drives over 2TB?
3) Does the system come with the screws for the Trays/Caddies?
4) The H200 that is included, is this the "Mezzanine" model?

Thanks for any info you can provide.
- mirfster

 

[silvesj]

You guys are awesome, and I think I have a better idea of what is needed now. Thanks sooo much. I'll keep you posted with any other questions.

 

[silvesj]

@Mirfster - One more question, what is your power consumption like with the DELL FS12-TY C2100 ?

 

[Mirfster]

It has dual 750w PSUs, but I haven't really tracked how much it is using. The L5630 is listed as 40w, where the others can range from 40w up to 130w.

 

[Klontje]

Sorry for off-topic questions but was triggered because of the powerusage question. You guys got me all warm about FreeNAS and at first I wanted to build a Skylake build because I want lot of disks and energy efficiency. But when I saw the prices on those Dell C2100 I thought, wow...

I do have some questions and one was the powerdraw, I saw the CPU's only have a TDP of 40w which is pretty low but am curious about overall system draw. Another point is noise. I don't have to host it in my bedroom like some other guys here but noise is often an issue with rack-mounted gear. Was wondering how and where you can host 19" equipment in a regular home. Problems I see when building a 19"rack in a closet build of drywall isolation and a door is cooling. Hope to hear some opinions.

And if this post is inappropriate here let me know, I'll create a new topic.

 

[Mirfster]

I saw the CPU's only have a TDP of 40w which is pretty low but am curious about overall system draw.

Not all the CPUs have the 40w TDP, as stated depending on the CPU chosen it can range from 40w to 130w each (system can hold 2 CPUs). I would not consider these as "energy efficient" as newer systems, but don't think they are any worse than comparable systems from the same time frame. I will see about getting some info for overall system draw together. I have three of them running in my Server Rack at the moment (1 - FreeNas, 1 - ESXi and 1 - Hyper-V) as well as one in a cubicle. My Dell 42U Server Rack is located in a small walk in closet within 10 feet of my cubicle (Yes, I have two cubicles in a room in the basement that is my "office"..).

As to noise, I always mention to NOT update the BMC Firmware above v1.7. This is because after that the fans run at full speed constantly (issue with the BMC Firmware) and it gets loud. Dell is not going to fix/address this. So back to the noise info, with all three of the Servers running, I used a decibel meter (a real one not some phone app) and averaged 58-60DB for the entire rack. I honestly think the Switch or Raritan PDU are contributing a lot to the noise. These things are super quiet IMHO. I constantly use speaker phone for meetings and no one ever knows about the servers. Keep in mind that when you cold power up, it will be a little loud as the system POST and the fans run through the self tests, but will quiet down shortly afterwards. This does not really occur on reboots.

Personally, I have put these things through the ringer and they still have performed flawlessly. Which finishing my basement "office" (converted guest room); I had the Server Rack housed in my garage. They sat there running all summer long without any specialized cooling. Image days/months on end @ 93F+ in a garage with concrete flooring constantly radiating heat...

So, are these for everyone? Meh, I don't know; but they work great for me.
Are they the most energy efficient? Compared to today's systems... Highly doubt it, but let me get some data together
Are they loud/noisy? Not at all IMHO

One thing to note is that most of the systems (if not all) you get from sellers will be the FS12-TY (avoid the AMD models, which if I recall have "NV" in the model). While 99.9% identical to a full on Dell C2100, they were made as "Cloud Servers" as a fork and are NOT officially supported by Dell. Not too much concern since even a C2100 would be past warranty. Drivers, Parts and Firmware for C2100s has worked fine on them as far as my experience.

Whew, sorry for the long reply. I really really have to get a post together with all this info to make it easy for everyone...

 

[Mirfster]

K, here is some info on the power consumption:

Server #1

• OS: Microsoft Server 2012 R2 DataCenter
• CPU(s): 2x Xeon X5650 @ 2.6GHz
• RAM: 24 GB
• Perc H700 Controller
• OS Drive(s): 2x 250GB SSD
  
• Data Drive(s): 12x 2TB (HGST)
  
• QLogic 10 GB SPF+ (Dual Port)
• Dual 750w PSU

Server #2

• OS: ESXi vSphere 5.5 Update 3
• CPU(s): 2x Xeon X5650 @ 2.6GHz
• RAM: 64 GB
• Perc H700 Controller
• OS Drive(s): 2x 32GB SSD
  
• Data Drive(s): 12x 2TB (HGST)
  
• QLogic 10 GB SPF+ (Dual Port)
• Dual 750w PSU

Server #3

• OS: FreeNAS-9.3-STABLE-201602031011
• CPU(s): 2x Xeon L5639 @ 2.13GHz
• RAM: 32 GB
• Perc H200 (Mezzanine Model) - Cross Flashed to 9211
  
• OS Drive(s): 2x 32GB SSD
  
• Data Drive(s): 12x 2TB (HGST)
• Intel 10 GB SPF+ (Mezzanine Model Dual Port)
  
• Dual 750w PSU

Now with that said, here is the power consumption via my Raritan PSU:
Server #1 (Outlets 2 & 12) = ~ 215 Watts
Server #2 (Outlets 1 & 11) = ~ 216 Watts
Server #3 (Outlets 3 & 13) = ~203 Watts

Again sorry for the long post, but figured it would assist the OP as well.

Hope this info helps.

 

[Mirfster]

Hey silvesj, let us know which card you went with that you are trying to cross-flash. In the meantime, take a look at this thread "Add raid card after configuration" where I provided info for cross-flashing. If we know which card you have we can easily assist (as long as it is a card that can be cross-flashed). :)

 

[silvesj]

Thanks for the reply! I'll read over your post. The card is PERC H200 RAID CONTROLLER

Just so I don't provide the wrong info:

http://www.ebay.com/itm/13172870062...8a5cd0b29fc&bu=44032488820&cp=1&sojTags=bu=bu

 

[Mirfster]

Yep should not really have too many problems flashing that card. I use those (both PCI and Mezzanine Models). Only issue you really may have it if the H200 has a really old version <= P06; then you would have to flash it up to that version P07 first (I think with Dell Firmware if I recall correctly); then you can flash it up to LSI P20.

If you have any questions feel free to ask.

 

[silvesj]

1st - I assume this is the drive I need to apply if my version is older. http://www.dell.com/support/home/us/en/19/Drivers/DriversDetails?driverId=N9C88

2nd - I installed FreeNAS 9.10 last night, this AM went to configure it and realize it has no IP. I go to the server, configure network interfaces, and it doesn't suggest any. So ifconfig shows no eth interfaces? Have you ever run in to any issues with this?

 

[Mirfster]

1. Should be, but don't worry about that yet. Let's see where it is at first
2. Never had any issues with it getting an IP
   • Make sure the NICs are enabled in the BIOS
     
   • Make sure you have the attached the proper NIC to your switch/router
     • Use any of the two that are on the back (stacked)
     • Top one is NIC #1 Bottom is NIC #2
     • The one by itself is the Dedicated BMC; used for Web Access/IPMI
3. Worse case, you should be able to add the Interface in the Shell/Console (Can walk you through this if needed)

 

[silvesj]

Got the NICs working -_- was disabled as you said but also IMO in a weird spot. But network is working.

Have the warning about the h200 - Firmware version 7 does not match driver version 20 for /dev/mps0. Please flash controller to P20 IT firmware.

and sas2flash output:

LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2008(B2)

Controller Number : 0
Controller : SAS2008(B2)
PCI Address : 00:05:00:00
SAS Address : 544a842-0-224d-ea00
NVDATA Version (Default) : 07.00.00.19
NVDATA Version (Persistent) : 07.00.00.19

 

[Mirfster]

So when you run sas2flash -list can you copy/paste the output here in CodeTags?

Would look similar to this:

Code:

Welcome to FreeNAS
[root@ASC-FREENAS01] ~# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved
        Adapter Selected is a LSI SAS: SAS2008(B1)
        Controller Number              : 0
        Controller                     : SAS2008(B1)
        PCI Address                    : 00:01:00:00
        SAS Address                    : 099152d-0-0029-216d
        NVDATA Version (Default)       : 14.01.00.08
        NVDATA Version (Persistent)    : 14.01.00.08
        Firmware Product ID            : 0x2213 (IT)
        Firmware Version               : 20.00.04.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9211-8i
        BIOS Version                   : 07.39.00.00
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : SAS2 Mezz
        Board Assembly                 : N/A
        Board Tracer Number            : N/A
        Finished Processing Commands Successfully.
        Exiting SAS2Flash.

 

[silvesj]

Sorry didn't realize I was impatient. Had to wait a few seconds more for the second half.

root@nas ~]# sas2flash -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

Adapter Selected is a LSI SAS: SAS2008(B2)

Controller Number : 0
Controller : SAS2008(B2)
PCI Address : 00:05:00:00
SAS Address : 544a842-0-224d-ea00
NVDATA Version (Default) : 07.00.00.19
NVDATA Version (Persistent) : 07.00.00.19
Firmware Product ID : 0x2713 (IR)
Firmware Version : 07.15.08.00
NVDATA Vendor : Dell
NVDATA Product ID : H200A
BIOS Version : 07.11.10.00
UEFI BSD Version : 07.02.04.00
FCODE Version : N/A
Board Name : PERC H200A
Board Assembly : N/A
Board Tracer Number : N/A

Finished Processing Commands Successfully.
Exiting SAS2Flash.


And I assume the BMC firmware is above what you recommended as the fans are loud !

Thanks in advance!

 

[Mirfster]

Firmware Version : 07.15.08.00

From memory, you should be fine to flash this with the LSI 9211 IT Firmware; since it is at >=7. Otherwise you would first have to bring it up.

And I assume the BMC firmware is above what you recommended as the fans are loud !

If your BMC Firmware is above 1.7, then the system will sound like a rocket taking off constantly. Pretty much I never go above version 1.70.64097; and even lower on is fine. I would suggest flashing your BMC Firmware first, since the sound can get annoying. Keep in mind that with the proper Firmware it may be loud when it first starts up from a complete power-off, but that should be only for a very short time.

 

[silvesj]

OK, so then I should follow your following steps for applying the LSI firmware? And I should be able to downgrade the BMC firmware correct?

1. Copy the "2118it.bin" and "mptsas2.rom" to a folder on FreeNas
   • In my example, it is a folder called "P20.4"
   • Located in: "\\ASC-FREENAS01\MediaShare\P20.4" <--- Windows Explorer
   • Located in: "/mnt/DataPool01/MediaShare/P20.4/" <--- FreeNas
2. SSH (Putty) to FreeNas
3. Change to the directory that has the files
   • Example Command: cd /mnt/DataPool01/MediaShare/P20.4
4. Execute the command to grab SAS Address (just in case)
   • Example Command: sas2flash -list
   • Copy the "SAS Address" Info (just for safe keeping)
5. Execute the "sas2flash" command to flash
   • Example Command: sas2flash -o -f 2118it.bin -b mptsas2.rom
   • *** I prefer the card to have the BIOS...
6. Exit Putty and Reboot when done (I am not 100% sure if this is needed, but I do it anyways)
   • Can delete the folder and files you created; if desired

 

[Mirfster]

OK, so then I should follow your following steps for applying the LSI firmware?

Yep, should be fine

And I should be able to downgrade the BMC firmware correct?

Correct, I think I got my older BMC from: http://poweredgec.dell.com/latest_fw.html (Just grab the one labeled "07/12/2012 pec2100bmc170.zip").

I use "socflash.EXE" instead of what they use "DCSFWUD.EXE". Can't recall right now why, but I have not had any issues with it. If you can't get it to flash with theirs, then let me know and I will see about getting you the "socflash.EXE" as well as instructions on how I do it.