Suggestions for motherboard upgrade?

[danb35]

Yeah, creeping featuritis is a big problem...

 

[sfatula]

A lot of money (when combined with other components you'll need) to just get IPMI! Must be a huge hassle. I find I am not using it very often but it's definitely handy.

 

[Ericloewe]

Speaking of which, I got hyped up by a BMC firmware changelog for the X11SPH-nCTF that mentioned virtual media now working with HTML5, but I was disappointed by the fact that there still is no such option visible.

 

[Whattteva]

A little late, but I use Supermicro X11-SPI-TF.

• It has plenty of RAM slots (8) and supports LRDIMM's up to 128 GB/stick if you want, though those sticks are very expensive. I use 4x 64 GB sticks at the moment.
• HTML5-based IPMI - check.
• Only single socket, so not a problem here and no need to worry about needing two CPU's to use all the RAM slots.
• Intel obviously.
• Has integrated 10x SATA3 + 2 SATADOM.
• IPMI includes onboard VGA if required.
• It has onboard copper 2x 10GbE.
• I think it has 1 or 2 M.2 slots, not sure.
• Availability is ok, not as high as Dell or HP obviously.
• Price is 350-500'ish maybe. You could get lucky and land a deal. Mine was $600 with a Xeon 4210T + cooler all bundled.

 

[danb35]

A lot of money (when combined with other components you'll need) to just get IPMI!

True, though the only other components were RAM, a pair of heatsinks, and I'll need to get a HBA (haven't decided yet whether to upgrade to a 3008-based card from my current onboard 2308). Oh, and a few cables.

There were other factors--the current board is getting kind of long in the tooth (I've had it seven years, and it was used when I bought it), and while I'm not seeing any impending signs of failure, the possibility's starting to nag at me. And there's been the occasional (though rare in the last couple of years) memory error, always in the same DIMM.

But IPMI is kind of addictive--once you've used it, you don't want a server without it. And the Java-based implementation of it is increasingly hard to support--the only way I've found thus far to make it work is with a Windows 7 VM, which right now only lives on one of my machines (though I suppose I could put it on my Proxmox cluster). I don't spend a huge amount of time away from home, but it's enough that I want this to work--particularly since it always seems that problems start just after I leave home.

 

[sfatula]

True, though the only other components were RAM, a pair of heatsinks, and I'll need to get a HBA (haven't decided yet whether to upgrade to a 3008-based card from my current onboard 2308). Oh, and a few cables.

There were other factors--the current board is getting kind of long in the tooth (I've had it seven years, and it was used when I bought it), and while I'm not seeing any impending signs of failure, the possibility's starting to nag at me. And there's been the occasional (though rare in the last couple of years) memory error, always in the same DIMM.

But IPMI is kind of addictive--once you've used it, you don't want a server without it. And the Java-based implementation of it is increasingly hard to support--the only way I've found thus far to make it work is with a Windows 7 VM, which right now only lives on one of my machines (though I suppose I could put it on my Proxmox cluster). I don't spend a huge amount of time away from home, but it's enough that I want this to work--particularly since it always seems that problems start just after I leave home.

You don't need a new CPU!? But yeah that's starting to get old, the old MB. A rare memory error is never a good thing either.

The java based implementation is awful, I agree, it was always a hassle to use for me. As I use an arm based puny Pi 4b for desktop for the past year (yes, I do everything on it that isn't on truenas and it works just fine as a desktop replacement for me, even build Truenas containers on it, code, run video processing, etc), java was a no go, it simply won't run on it. Java will, but not IPMI. I wouldn't have a server without IPMI either. Remote fixing is beautiful.

I did used too. My x10 was just over $100 (and it was unopened! a lucky find) and the e5-2698v3 was $70 used. Didn't see any need for newer CPU as that one screams anyway and pretty much allows for all the "apps" I'd ever use and VMs as well.

 

[danb35]

You don't need a new CPU!?

No, the board includes CPUs. As @Etorix said, I thought that made it a pretty good deal--though at the price those chips are going for, maybe not quite as outstanding as I'd thought; looks like they're around $30 each. I guess I also need heatsink compound; haven't dealt with that for a good while now.

 

[Ericloewe]

looks like they're around $30 each

Holy %!#& that is insanely cheap, what the hell happened!? I've literally seen LGA1366 Westemere EP parts going for a bit more than that, and they sure as hell don't have 14 cores.

 

[sfatula]

$30 for a Xeon Gold? I missed that part sorry, but wow if true!

 

[danb35]

$30 for a Xeon Gold?

CD8067303592500 INTEL XEON PROCESSOR GOLD 6132 2.60GHZ 19.25M 14 CORES 140W H0 | eBay

PART NUMBER CD8067303592500. PROCESSOR NUMBER GOLD 6132. INCLUDE WITH PROCESSOR. INSTRUCTION SET 64-BIT. LITHOGRAPHY 14 NM. not for PC.

www.ebay.com

Intel Xeon GOLD 6132 (SR3J3)-2.60GHZ PROCESSOR | eBay

Find many great new & used options and get the best deals for Intel Xeon GOLD 6132 (SR3J3)-2.60GHZ PROCESSOR at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Intel Xeon Skylake SR3J3 2.60 GHz GOLD-6132 FCLGA3647 CPU Processor for sale online | eBay

Find many great new & used options and get the best deals for Intel Xeon Skylake SR3J3 2.60 GHz GOLD-6132 FCLGA3647 CPU Processor at the best online prices at eBay! Free shipping for many products!

www.ebay.com

Definitely less than I paid for my E5-2670s.

 

[sfatula]

Wow! That is cheap, and a little better than my 2698 v3. Definitely cheaper. I'd buy that!

 

[danb35]

So, an update. Apparently I should have looked around eBay a little more carefully; the same seller had boards that included the appropriate heatsinks in other listings. But when I asked them if they had the heatsinks, I asked about a separate listing, so that's what they told me about--and ended up sending the wrong, or at least incomplete, heatsinks--they didn't have the CPU retention clips that are now required with LGA3647. So they're swapping them out, which is good, and doing it without much in the way of hassle, which is also good--but I still don't have the right heatsinks until they get me the replacements.

Motherboard is here. It includes two 8 GB sticks of RAM, but I don't think I'm going to use them. RAM is here, all 128 GB of it. HBA is here. SAS breakout cables are here. SAS 8643-to-8087 cables (yeah, I went for the 3008 HBA) are ordered but not here yet. Maybe everything will be here by next weekend to finish up.

 

[danb35]

So it's probably time to put this to bed. No, the vendor for the heatsinks didn't swap them out; they just refunded my money once they got them back. Ordered a pair from another vendor, they arrived quickly, and were the correct parts. So today was the big day.

Replacing the old mobo with the new wasn't too tough. I needed to move two of the standoffs in the chassis, but once I did, everything fit nicely. Props to Supermicro for making that relatively painless. Then it was update BIOS (or "embedded motherboard firmware", if you like), then update BMC, then update HBA firmware. Then futz with the boot order settings in the BIOS to get it to boot from the right device. The only problem I ran into is that apparently those "predictable network interface names" aren't, so now the two ports on my Chelsio NIC are enp94s0f4 and enp94s0f4d1 rather than whatever they were before. So destroy the LAGG interface, recreate it with the two correct ports, and it's all good.

So everything's up and running. And now I have a HTML5 virtual console, which will make it much easier to remote-admin the thing in the future. And it seems to be running the fans a little slower, which I don't mind, even though the server room is now far enough from the house that it doesn't much matter any more. I'll want to make sure temps stay reasonable, but

Now if only upgraded CPUs for my MicroServer Gen10+ were more reasonably priced...

Edit: the newer IPMI firmware also means that it's easier to update its TLS certificate, and even to automate that using a script like this: https://gist.github.com/arnebjarne/54dbab54e5fb82043a4835c0250840b4. If you're using the SmallStep CA software to run a local CA, make sure to issue the cert using a RSA key rather than EC, as the firmware apparently doesn't support EC certs.