Xeon server build, looking for recommendations on parts please

[andyjay777]

• Case was looking at the Fractal Define 7 XL, but there might be better options? Prefer something as quiet as possible
• Also, not sure on PSU yet, with up to 14 drives, going to need a few molex connectors I guess :o

Cheers,
Deeda

I am running the Define XL with 16 HDDs and it's pretty quiet, while still maintaining good airflow to most of the drives. I say most... there are two pairs of drives at the bottom on the same level as the power supply and the 2nd pair run warmer by 7-8 deg over the rest.

There are other comments on power supply sizing (and some good articles on this forum). Went for a Corsair RM1000x. It's got 6 SATA power connectors on the PSU and I could use my spare SATA power cables to supply all the drives (with no more than 4 drive per cable). Some photos here.

 

[Deeda]

I am running the Define XL with 16 HDDs and it's pretty quiet, while still maintaining good airflow to most of the drives. I say most... there are two pairs of drives at the bottom on the same level as the power supply and the 2nd pair run warmer by 7-8 deg over the rest.

There are other comments on power supply sizing (and some good articles on this forum). Went for a Corsair RM1000x. It's got 6 SATA power connectors on the PSU and I could use my spare SATA power cables to supply all the drives (with no more than 4 drive per cable). Some photos here.

Thanks for the info on the Define XL.

Next mystery I am trying to work out is the mounting holes on the bottom of SAS drives. With my current TrueNAS build in a Define R5, the SAS drive holes on the bottom of the drives are different to SATA drives, so didn't line up with the R5 mounting trays, seen here.

The drive bays for the Define 7 XL look different, but I can't work out if the holes will line up, as there are no measurements I can find anywhere? Define 7 XL drive trays.

 

[sretalla]

Next mystery I am trying to work out is the mounting holes on the bottom of SAS drives. With my current TrueNAS build in a Define R5, the SAS drive holes on the bottom of the drives are different to SATA drives, so didn't line up with the R5 mounting trays, seen here.

The drive bays for the Define 7 XL look different, but I can't work out if the holes will line up, as there are no measurements I can find anywhere? Define 7 XL drive trays.

I guess that's what you get when you mix enterprise (SAS) with consumer gear.

I would have assumed the relatively small number of manufacturers would have meant that things like that were super-standardized, but you just blew that out of the water.

The holes on the sides of the drives are probably more standard/aligned, but that doesn't help you with that case.

You'll need to get out your drill and your good reading glasses to make sure you mark and drill accurately.

 

[Deeda]

Yep the side holes line up... just not those bottom ones ffs... ah well, and yep I think you are right about the drill. Just a shame to have to drill holes in few hundred dollars worth of extra drive holders...

 

[neofusion]

Why?
The holes add value.
I've done the same in my R5 with much success. You do not need the full mouse ear hole configuration either, as long as you don't make the holes tiny you can work the rubber dampeners through each hole with some finagling. I did file them down a bit after drilling to remove some of the sharp edges; less risk of accidental cuts to fingers and/or rubber dampeners that way.

The only somewhat unrelated issue I had was vibration/resonance in the case, but that was solved by adding electrical tape to the sides of the drive sleds as well as the places where the side panels touch the chassis. I am still amazed at how well it eliminated the noise.

 

[Etorix]

I guess that's what you get when you mix enterprise (SAS) with consumer gear.

There's no need to throw enterprise SAS gear into the mix: I've seen a similar issue mixing SATA drives from WD (3 side holes) and Seagate (2 side holes).

I would have assumed the relatively small number of manufacturers would have meant that things like that were super-standardized, but you just blew that out of the water.

Nope. Three manufacturers, all building their drives from platters supplied by a single company, could not fully agree where to put screw holes in the casings.

 

[Deeda]

Ok so first day of new build. Hit a snag already. Getting a code 53 on post, which seems to suggest memory issue...

I'm certain the memory, shown here should be ok? Motherboard says it supports DDR4 2400 on E5-1600/2600 V4 CPUs https://www.asrockrack.com/general/productdetail.asp?Model=EPC612D4U-8R#Specifications

CPU is a Xeon E5-2680 v4.

Tried 4 sticks, and now just trying the 1 stick, same result.

The motherboard manual says, "Problem related to memory. Please re-install CPU and memory then clear CMOS."

I am getting a display on the monitor, it just sits on "System Initializing". I assume if the CPU wasn't working or faulty, I would get nothing on the monitor?

 

[Deeda]

Ok so, there is a sticker on the BIOS chip that says, "P1.00". I guess (not sure) that indicates it is running BIOS v1.00?

If so that might be the issue with the memory error, as BIOS version 2.0 or above is required for v4 series. I did see this before I purchased, but was hoping newer stock would come with a new BIOS. Version 2.0 released in 2016

Now, without entering The Matrix, how do you update the BIOS if you can't boot...

 

[NugentS]

Sometimes you have to buy (off ebay, generally) a cheap chip to boot the board, to update the BIOS so that you can then use the actual CPU you want ti use.

 

[Etorix]

Now, without entering The Matrix, how do you update the BIOS if you can't boot...

Plug a cable in the BMC management port, find out which address it got, log in to the BMC UI and see if you can update from there. No CPU required.

 

[andyjay777]

Ok so, there is a sticker on the BIOS chip that says, "P1.00". I guess (not sure) that indicates it is running BIOS v1.00?

If so that might be the issue with the memory error, as BIOS version 2.0 or above is required for v4 series. I did see this before I purchased, but was hoping newer stock would come with a new BIOS. Version 2.0 released in 2016

Now, without entering The Matrix, how do you update the BIOS if you can't boot...

The link shows the board supports bios “Instant Flash”. Without going into the manual I think this means you can plug a Usb stick into the board and push the flash button to update the bios. Can be done with or without a CPU with the system off (but ATX power connected).
Check the manual as the ASRock board I had required you to rename the file to creative.rom.

Might save you removing the CPU and re-pasting etc.

Good luck!
(I had a terrible time with a 2nd hand ASRock board (not ASRock Rack) and could never get it to post with a new Xeon which required the latest bios, but worked fine on old CPUs… I hope you get on better)

 

[Deeda]

Plug a cable in the BMC management port, find out which address it got, log in to the BMC UI and see if you can update from there. No CPU required.

Thanks for the tip. I got into the BMC. Couldn't find any mention of updating the BIOS via the BMC. So I updated the BMC to the latest version. Same result though, no update BIOS in the BMC interface. Found a Asrock FAQ that said to update BIOS via BMC requires "BMC AST2500", seems I have AST2400 of course lol

The link shows the board supports bios “Instant Flash”. Without going into the manual I think this means you can plug a Usb stick into the board and push the flash button to update the bios. Can be done with or without a CPU with the system off (but ATX power connected).
Check the manual as the ASRock board I had required you to rename the file to creative.rom.

Might save you removing the CPU and re-pasting etc.

Good luck!
(I had a terrible time with a 2nd hand ASRock board (not ASRock Rack) and could never get it to post with a new Xeon which required the latest bios, but worked fine on old CPUs… I hope you get on better)

Also checked out Instant Flash. The instructions indicate that to use Instant Flash you either enter the BIOS, or press F6 when you see the "Instant Flash" option during bootup. Which it seems the system doesn't get that far on boot https://www.asrockrack.com/support/faq.asp#InstantFlash

 

[andyjay777]

Also checked out Instant Flash. The instructions indicate that to use Instant Flash you either enter the BIOS, or press F6 when you see the "Instant Flash" option during bootup. Which it seems the system doesn't get that far on boot https://www.asrockrack.com/support/faq.asp#InstantFlash

Oh no. See if your local computer store has an older version of the CPU that will boot and get them to upgrade the bios. Will cost you some money… may or may not be cheaper than buying the older CPU yourself (as someone suggested).

 

[Deeda]

Got my hands on a 2630 v2 and thought I'd be home free. Doesn't fit the socket, lol wtf... yes I should learn to read, as the motherboard does state it supports v3 or higher CPUs...

 

[Deeda]

Ok so some progress today!

BIOS updated and v4 CPU working. Also got the onboard SAS controller flashed to IT mode...

 

[neofusion]

Ok so some progress today!

BIOS updated and v4 CPU working. Also got the onboard SAS controller flashed to IT mode...

Congratulations!

With the SAS3008 based HBA I would aim for firmware version 16.00.12.00.
See if you can find a slightly more up-to-date version.

 

[Deeda]

Congratulations!

With the SAS3008 based HBA I would aim for firmware version 16.00.12.00.
See if you can find a slightly more up-to-date version.

Bugger I thought that 16.00.10.00 was the latest? I grabbed it from the Broadcom website and that was the latest they had?

 

[blanchet]

There is a special firmware version for LSI 9300 series running TrueNAS.
It is available only on the TrueNAS forum.

LSI 9300-xx Firmware Update

JoshDW19 submitted a new resource: LSI 9300-xx Firmware Update - Fixes a controller reset issue in LSI 9300-xx HBA cards Hey Community, If you are using an LSI 9300 HBA with FreeNAS or the soon-to-be TrueNAS CORE, you may experience some performance issues causing the controller to reset...

www.truenas.com

 

[Deeda]

Thanks for the tip @blanchet and @neofusion. I also received the additional PCIe HBA in the mail today, so have now updated both the onboard and PCIe HBAs to 16.00.12.00

The downloaded zip for version 16.00.12.00 didn't include BIOS or UEFI updates, so I assume just stick with the ones that came with 16.00.10.00?

 

[blanchet]

The downloaded zip for version 16.00.12.00 didn't include BIOS or UEFI updates, so I assume just stick with the ones that came with 16.00.10.00?

Yes you can use the sas3flash utility that you want.
If you want to update from FreeDOS, follow this instructions

SOLVED - Remotely flash LSI 9300 with FreeDOS Live CD

Abstract This guide explains how to create a custom FreeDOS Live CD to remotely upgrade the firmware of LSI 9300 HBA on a Supermicro server. The custom LiveCD is created from a Linux computer. This guide is based on the instructions from https://www.youtube.com/watch?v=IwPsadNho9M This guide...

www.truenas.com