PSA: CTRL+F5 is your friend

[Ericloewe]

So, having recently acquired an X10SDV-4C-TLN2F that had a weird 7.x version BMC firmware (latest publicly available is 3.88, there's a 3.9x out there which I suspect introduces random user passwords and only ships on new boards), I did something really stupid and flashed 3.88 without backing up the old firmware.

Well, that mostly worked. Everything seemed fine, except that much of the webGUI was broken. No sensors list at all, HTML5 iKVM didn't work (but Java did!), all sorts of strange broken things.
I reset the thing to factory defaults, reflashed it multiple times, rebooted a ton of times... always to no success. I even wrote this up over at the STH forums in hopes of finding some leads.

The sudden realization that I'd wasted hours of my life hit me after I realized that IPMIview and IPMItool both worked fine, over the LAN and over the LPC bus, respectively. Everything worked, except the webGUI - because the pre-flash version of some assests was cached by the browser!

After a quick CTRL+F5, everything is working normally.

(Fun fact: this thing actually has my first 10GbE NIC ever. @jgreco - apparently you ran into a weird firmware/driver issue with the Broadwell-D NIC in 10Gb mode. Did you ever look into a fix for the immediate issue or did you just throw -DA4 cards at the problem?)

 

[jgreco]

apparently you ran into a weird firmware/driver issue with the Broadwell-D NIC in 10Gb mode.

If by "weird" you mean that Supermicro sent me the firmware and eprom updater tool for a different board, and it hosed it but good. Then when they eventually sent me the correct firmware, I was having significant trouble installing it because ... if I remember correctly it was identifying it as the board the previous update had been targeted at. It was dead to drivers that tried to probe it, and Supermicro wanted to RMA the board rather than figure it out, which wasn't too feasible at the time.

Since the stupid thing only takes 128GB RAM, it is only a small part of the environment here. The normal config here is 4x 10GbE in 2x2 failover, of which the Broadwell-D was handling two and a Solarflare was handling two, so I just let it fail over and ignored it. I know it wasn't working. But -- weirdly -- I then did put in a Supermicro AOC-STG-I4S or whatever the quad X710 Supermicro card is, and after flashing the X710 to appropriate firmware levels, I found the Broadwell was suddenly working too.

The weird thing is that networking seems to be getting worse. The X552 crap is worse than the X520 (which remains the only "minimal screwing around" card I know of), and the X710 literally has four(?) different firmware revisions supported by ESXi 6.7. I mostly did the upgrade for better VF support and because the Solarflares didn't support ESXi 7. Argh.

 

[Ericloewe]

The weird thing is that networking seems to be getting worse.

You mean that the hardware is degrading? Or a drop in quality from the X520 to the X552 part of Boradwell-D?

 

[jgreco]

You mean that the hardware is degrading? Or a drop in quality from the X520 to the X552 part of Boradwell-D?

An increased dependence on firmware and OEM competence to make the product work....?

 

[Ericloewe]

Had me scared there for a moment.

Yeah, I noticed that back in 2014/2015 when this stuff came out. It was rough in terms of early support. I suspect they did a big redesign for Xeon-D, because they wouldn't have been using close to leading-edge nodes for NICs at that time - probably leading to big firmware changes. The TDP budget was also damn tight, considering an X520 is probably around a quarter of the TDP of a low-end Broadwell-D, once you take out the PHY.
Having everything integrated in a design for small systems also makes OEMs work harder, because they can't just take a reference design, throw it on a microATX board, and sprinkle in some details like they might for a larger thing.

 

[jgreco]

Yeah, I noticed that back in 2014/2015 when this stuff came out. It was rough in terms of early support.

That'd be more easily accepted if we weren't talking about 2020.

 

[Ericloewe]

Ouch.

 

[mistermanko]

ecause the pre-flash version of some assests was cached by the browser!

Been there, done that. I've really learned to appreciate this extension: cache cleaner https://addons.mozilla.org/en-US/firefox/addon/clearcache/

 

[Ericloewe]

They want AI to analyze images, I just want an AI that clears the correct caches automagically. Is that too much to ask for?
Yes it is.

As a sidenote, I believe this also fixes a weird bug with the user management on another Supermicro board (I think it was the A1SRi-2758F, which uses the same BMC firmware image) - in that case, literally everything I tried was working, except that user passwords weren't updated correctly.

 

[mistermanko]

Say, how do you get these freaking java applets running? I had to boot up a windows vm just to install old java runtimes to open these stupid IMPI KVMs. I can't add boot iso's via HTML5 iKVM.

 

[Ericloewe]

Ideally, you don't. If you're stuck with older systems that didn't get HTML5 iKVM clients (Supermicro X9 and earlier, Dell Gen 12 and earlier, etc.), Windows or Linux both work - but you need the correct Java thing.

Say, how do you get these freaking java applets running?

They're not applets, those have been personae non gratae for a decade+, fortunately. They're actually Java Web Start apps... which is something that's also been deprecated by mainline Java, albeit much more recently.

The good news is that the old terrible experience is slightly improved, thanks to these guys: https://openwebstart.com/

Basically install that, then access the iKVM client as usual.

 

[KevDog]

@Ericloewe
I'm trying to use openwebstart, but when opening the jnlp file it says no iKVM64 in java.library.path. Any way to fix? I'm not too familiar with OpenWebStart

 

[Whattteva]

Don't use the file that it gives you to download. Run the standalone app. Much better experience and it's what I'm using.

 

[KevDog]

Don't use the file that it gives you to download. Run the standalone app. Much better experience and it's what I'm using.

Which standalone app are you referring too? ipmiView?

 

[jgreco]

Which standalone app are you referring too? ipmiView?

IPMIview 2.0. Much better tool.

 

[KevDog]

IPMIview 2.0. Much better tool.

I'm using IPMIview 1.9 on MacOS Monterey and this is all I'm getting. It's like authetication isn't working or something. I'm trying to get to terminal.

 

[jgreco]

Does that even have remote console capability?

 

[KevDog]

Does that even have remote console capability?

Not sure what you're exactly referring to -- either the ipmiView version or MOBO version -- I used to be able to using the remote console via JVM through the web interface:

 

[Whattteva]

Well, that mostly worked. Everything seemed fine, except that much of the webGUI was broken. No sensors list at all, HTML5 iKVM didn't work (but Java did!), all sorts of strange broken things.
I reset the thing to factory defaults, reflashed it multiple times, rebooted a ton of times... always to no success. I even wrote this up over at the STH forums in hopes of finding some leads.

The sudden realization that I'd wasted hours of my life hit me after I realized that IPMIview and IPMItool both worked fine, over the LAN and over the LPC bus, respectively. Everything worked, except the webGUI - because the pre-flash version of some assests was cached by the browser!

After a quick CTRL+F5, everything is working normally.

You see, this is super weird to me cause I think you were the one that actually told me that I could upgrade my X10 board to HTML5 by doing a firmware upgrade + clear browser cache.

 

[jgreco]

Not sure what you're exactly referring to -- either the ipmiView version or MOBO version -- I used to be able to using the remote console via JVM through the web interface:

There is no "MOBO version". There is a web server on the IPMI controller on the motherboard that is also capable of launching a Java console viewer, but this is not IPMIVIEW.

I am saying use IPMIview. 2.0. This is a package distributed by Supermicro that is actually a Java app. I don't see a remote console option in whatever the Apple "IPMIview" thing is, which is more or less how I remember ancient IPMIview 1 to be like.