PSA: (Rare) Skylake/Kaby Lake Hyperthreading bug

[Mark Levitt]

Nope, but I'm just pointing out that there is a intel-ucode available which apparently fixes the issue without requiring a bios update.

Yes, but that update won't fix the issue across a reboot. I.e., microcode updates have to be applied every time the machine is booted.

 

[ioquatix]

That's right, it's part of the boot process on my system.

 

[Mark Levitt]

OK,

Having done some more digging, it seems that this FreeBSD package has been updated to the microcode that fixes the hyper threading data corruption issue:
https://www.freshports.org/sysutils/devcpu-data/

Installing the package in FreeNAS doesn't work and I tried installing it in a Jail but, as I expected, the Jail can't write the microcode to the CPU.

Any idea how to get this installed to the FreeNAS boot on FreeNAS 11?

 

[Ericloewe]

There's a ticket for that in the bug tracker. Add that information to it, and please post the ticket number here, while you're at it.

 

[joeschmuck]

Well there is a bug report on this problem however iXsystems has stated it was deiffered to be fixed by a third party. I personally feel that is a bad choice by iXsystems since there does exist an easy fix. And they should want to update any TrueNAS systems they sold as well.

I updated the ticket with my two cents.

 

[mav@]

None of existing TrueNAS systems are affected, since Skylake Xeon E5 v5 CPUs are not even released by Intel yet. Mentioned comments on the bug report are my own, and my personal feeling is that this in significant part is an example of mass hysteria (possibly not without commercial side).

 

[rs225]

From a FreeNAS perspective, this probably only matters if you are running a VM that hits it, or a plug-in. I think the core system is going to be fine. BIOS solution is probably acceptable here.

The main problem is distribution of the microcode is probably Internet-only from Intel's URL(excepting big OS makers). That isn't going to work for a lot of installations.

 

[Mark Levitt]

From a FreeNAS perspective, this probably only matters if you are running a VM that hits it, or a plug-in. I think the core system is going to be fine. BIOS solution is probably acceptable here.

The main problem is distribution of the microcode is probably Internet-only from Intel's URL(excepting big OS makers). That isn't going to work for a lot of installations.

First, it seems to me that a lot of people use FreeNAS as a Plex server and want to do heavy transcoding. That seems like a primary use case for FreeNAS. I also don't believe motherboard vendors are going to be very quick to update their BIOS releases if they do it at all. Why would they when there is an intel microcode release available for most OS platforms.

Second, no the lack of internet connectivity is not an issue. The FreeBSD package "devcpu-data" is distributed with the intel microcde. It does not download it on the fly at install or runtime.

The current FreeBSD package already contains the relevant microcode. This just needs to be added to FreeNAS.

 

[Amir Yalon]

None of existing TrueNAS systems are affected, since Skylake Xeon E5 v5 CPUs are not even released by Intel yet. Mentioned comments on the bug report are my own, and my personal feeling is that this in significant part is an example of mass hysteria (possibly not without commercial side).

I doubt anyone is trying to cause mass hysteria, especially doubt there is a commercial side to it. Granted, the set of instructions that trigger it is rarely produced by compilers, yet the problem is real; so much so, that Intel has reported errata and produced a fix.

We can only speculate that FreeNAS itself is not affected because it is not written in OCaml, and not compiled with GCC. But until you have actively ruled it out by checking every CPU instruction, you can’t say for sure that it is not affected. Not to mention, users may be running OCaml programs on Linux in a bhyve VM, or compile them with GCC in a jail (if that’s possible), so again, the problem is real even on FreeNAS.

In any case, I’d be grateful if anyone can update here as soon as Supermicro have released a BIOS update with the fix. I’ve sent an email to support@supermicro.com, not sure what kind of response to expect.

 

[Mark Levitt]

We can only speculate that FreeNAS itself is not affected because it is not written in OCaml, and not compiled with GCC. But until you have actively ruled it out by checking every CPU instruction, you can’t say for sure that it is not affected. Not to mention, users may be running OCaml programs on Linux in a bhyve VM, or compile them with GCC in a jail (if that’s possible), so again, the problem is real even on FreeNAS.

From the reports I've seen, the bug is triggered by code that is small and in a tight loop. Compilers usually don't do that. That being said, I'd imagine some video encoders might have some hand-optimised inner loops that could trip this bug.

I worked out how to install the FreeBSD package and get it to load the microcode that fixes the issue so hyper threading back on for me.

 

[rs225]

First, it seems to me that a lot of people use FreeNAS as a Plex server and want to do heavy transcoding. That seems like a primary use case for FreeNAS. I also don't believe motherboard vendors are going to be very quick to update their BIOS releases if they do it at all. Why would they when there is an intel microcode release available for most OS platforms.

Does Plex hit this? I would be more likely to believe Java JITs or a JavaScript engine can hit this than Plex. Plex may be optimized, but it is the same code running everywhere; it would have already turned up. Motherboard vendors will release the update because we are talking about it, system integrators are going to demand it for any Skylake/Kaby-lake product, and nobody wants their hardware coming back to them by the carton. OS patches are great, but the BIOS has always been the intended mechanism.

The current FreeBSD package already contains the relevant microcode. This just needs to be added to FreeNAS.

You are correct about the .pkg. I was looking at the ports, which fetch it from Intel or AMD. So they presumably could include it.

 

[Mark Levitt]

You are correct about the .pkg. I was looking at the ports, which fetch it from Intel or AMD. So they presumably could include it.

Plex does do transcoding so perhaps. I mean, there's no question it's very rare, but if a transcoding job causes data corruption would most people notice or would they just assume its a Plex bug or random event. In reality, is this bug likely to be triggered any more or less than a cosmic ray flipping a bit in RAM? Probably not, but everyone insists on only ECC ram for FreeNAS.

Anyway, I took a look at the actual package and it's basically just the firmware files and an rc.d init script to load them into the CPU at boot time. I ended up installing the package in a Jail and then just copying the firmware files and scripts from outside the jail to where they expect to be.

 

[Ericloewe]

Does Plex hit this?

Highly doubtful. We don't see complaints about Plex mysteriously breaking stuff all the time, and we would see them.

 

[Amir Yalon]

I’ve sent an email to support@supermicro.com, not sure what kind of response to expect.

Good news, the prompt reply from Supermicro indicates that they are actively working on it, and the new revision BIOS will be released as soon as internal validation is finished.

 

[Mark Levitt]

Good news, the prompt reply from Supermicro indicates that they are actively working on it, and the new revision BIOS will be released as soon as internal validation is finished.

Hmm. I emailed Asrock Rack and have yet to hear anything from them.

 

[joeschmuck]

Hmm. I emailed Asrock Rack and have yet to hear anything from them.

Supermicro has an important customer base that they prefer to keep happy so they buy more product in the coming years. And I have noticed that Supermicro does answer you if you submit a request. Sometimes it's not what you want to hear but they don't leave you in the dark.

Guess I'll check for my BIOS update in a few weeks.

 

[DrKK]

Hmm. I emailed Asrock Rack and have yet to hear anything from them.

Now you know why so many of us are Supermicro fanboys. :)

 

[melbow]

Good news, the prompt reply from Supermicro indicates that they are actively working on it, and the new revision BIOS will be released as soon as internal validation is finished.

Just a small update FYI - I also contacted Supermicro today about this (got an almost instant reply too ), specifically about the X11SSM-F: so BIOS 2.0B contains the fix for this hyperthreading issue and is currently still in testing/validation and no fixed release date as of yet.

 

[tedthom]

Hi all!
Just wanted to let you know that the 2.0b BIOS update for the Supermicro X11SSM-F has been posted on the Supermicro website sometime last week, it looks like.
I applied the update yesterday and it went smoothly! [Just FYI, I used the files and tutorials from the SevenForums web site on how to create an MS-DOS bootable Flash Drive]
Hyper-threading is now enabled again on my side!
Thank you Supermicro.
Happy flashing!

 

[droeders]

Looks like there's an update to the IPMI firmware as well - version 1.37.

My boards are at 1.13. Does anyone know where I can find a list of fixes? The release notes in the zip file for 1.37 are worthless.