PSA: (Rare) Skylake/Kaby Lake Hyperthreading bug

[Omega]

See: https://lists.debian.org/debian-devel/2017/06/msg00308.html

There appears to be a bug with Skylake/Kaby Lake processors (the v5/v6 ones) with active Hyperthreading that in rare situations can lead to unpredictable behaviour. This is apparently fixed with a microcode update for Skylake (I don't know if those are applied by FreeNAS, so it might be fine and not an issue), Kaby Lake not yet according to the Debian warning.

It might not be a big issue, but if you have one of the affected processors, you might want to do a bit of research. I was about to look closer into it, but then realised that my systems are (with the exception of a MacBookPro), all on Haswell...

Apologies if this is already flagged, a search didn't yield anything.

 

[joeschmuck]

Interesting read. I'll have to check on my CPU, I'm sure it's affected but...

 

[Ericloewe]

Interesting. This seems to be the following entry in the Skylake Xeon E3 errata: https://www.intel.com/content/dam/w...es/xeon-e3-1200v5-spec-update.pdf#G2.14032447

The microcode patch is probably going to be in the next BIOS version for most boards, that'll be the definitive "fix".

 

[ioquatix]

What does the BIOS patch do?

 

[Ericloewe]

What does the BIOS patch do?

Presumably fix whatever bad logic is currently in the microcode.

 

[ioquatix]

I see, but how does this work? Does it upload some changed firmware to the CPU?

 

[ioquatix]

It looks like if you have the latest intel microcode package, it also fixes the issue?

This is not all - you need intel-microcode '...with base version 3.20170511.1' (which is not available on every distro yet AFAIK), and then you need to reboot for it to take effect.

From Reddit: https://www.reddit.com/r/programmin...l_skylakekaby_lake_processors_broken/djdw5h8/

 

[Omega]

Yes, the latest microcode package fixes Skylake (according to the original link - haven't verified). But as I said, I am not sure if FreeNAS would apply them; FreeBSD is capable to do so (cpuctl), but afaict, cpuctl_load isn't set in loader.conf (at least on my FreeNAS, 9.10.2-U4 - and with the caveat that I am not native to FreeBSD, so might have missed a place where it is set).

Updated BIOSes does sound like the nicest solution :)

 

[c32767a]

I see, but how does this work? Does it upload some changed firmware to the CPU?

A lot of the CPU's onboard microcode can be uploaded/replaced by either the BIOS at boot time or the OS. From what I've seen in the writeups, This will be completely fixed only with a BIOS update that includes new CPU microcode.

 

[Ericloewe]

I don't think any CPU from the last ~15 years can boot to a usable state without the BIOS uploading at least some of the microcode.

 

[Mark Levitt]

Yes, the latest microcode package fixes Skylake (according to the original link - haven't verified). But as I said, I am not sure if FreeNAS would apply them; FreeBSD is capable to do so (cpuctl), but afaict, cpuctl_load isn't set in loader.conf (at least on my FreeNAS, 9.10.2-U4 - and with the caveat that I am not native to FreeBSD, so might have missed a place where it is set).

Updated BIOSes does sound like the nicest solution :)

The problem is that waiting for the motherboard manufacturers to update their BIOS releases could take months, if they even bother.

 

[Dice]

So, for us SM users, we won't take on any action except for disabling HT and than sitting around until next SM bios update? Or is it something that we should do?

 

[Mark Levitt]

So, for us SM users, we won't take on any action except for disabling HT and than sitting around until next SM bios update? Or is it something that we should do?

Given that there is a microcode update from Intel for Skylake processors, I would have expected FreeNAS to apply it in the next release.

 

[ioquatix]

I already updated intel-ucode as part of boot process to latest one. How can we test if issue is fixed?

 

[Mark Levitt]

I already updated intel-ucode as part of boot process to latest one. How can we test if issue is fixed?

How did you update? I'm not really familiar with FreeBSD, but it seems like the package containing intel microcode is "devcpu-data" which doesn't seem to be installed on FreeNAS.

 

[Omega]

Also, I believe the microcode updates aren't persistent and need to be applied on each boot.

 

[Mark Levitt]

Also, I believe the microcode updates aren't persistent and need to be applied on each boot.

Yes. There seems to be a FreeBSD package "devcpu-data" that contains various microcode updates and, when installed and enabled, loads them into the CPU on boot. It's not installed on FreeNAS though.

I guess we have to wait for the FreeBSD devs to update that package, then for FreeNAS devs to include it in a FreeNAS release.

(Or hope that the motherboard manufacturers release a bios update, but I won't hold my breath on that one).

 

[ioquatix]

How did you update? I'm not really familiar with FreeBSD, but it seems like the package containing intel microcode is "devcpu-data" which doesn't seem to be installed on FreeNAS.

I should apologise for not being clear. I'm running on Arch Linux.

 

[Mark Levitt]

I should apologise for not being clear. I'm running on Arch Linux.

Ah. Doesn't really apply to booting FreeNAS then.

 

[ioquatix]

Nope, but I'm just pointing out that there is a intel-ucode available which apparently fixes the issue without requiring a bios update.