xHCI, Skylake, Supermicro X11 rant

[Ericloewe]

So, after initial testing showed that my new server was ready to start proper testing, which means trying to install FreeNAS 9.3.

The catch is that it's a Supermicro X11SSM-F. I'd already updated the BIOS and BMC/IPMI firmware (still going to need a BIOS update in the near future, probably, with the microcode patch for the Skylake processors.

One of the apparently cool features of the 1.0b BIOS was this option:

Windows 7, much like FreeNAS 9.3, doesn't include an xHCI driver by default, so installing it is a bit painful on Skylake. I assumed this feature would present USB devices as ATA devices or some such thing, as a temporary workaround.

Ok, so let's see what boot options we have on a motherboard like this:

1. Boot from a smorgasbord of USB devices (which get somewhat arbitrarily assigned to floppy disk drives, pen drives and HDDs - an 8GB Toshiba flash drive got recognized as a floppy disk drive one boot and as a pen drive the next)
2. Boot from IPMI virtual devices. These are mounted as USB devices
3. Network boot
4. SATA devices, obviously

Since this is a rant, let's go through these points and see what the problems are:

1. Well, everyone knows about Skylake only supporting xHCI, which means the FreeNAS 9.3 installer won't work. The "nice little option" doesn't help at all with FreeNAS. Boo.
2. I certainly expected IPMI to mount the virtual USB devices autonomously. The BMC is connected to the host via a number of interfaces, including PCI-e. It seemed only logical that an internal PCI-e USB controller would be used. But, that seems to not be the case, since the FreeNAS 9.3 installer won't boot. With or without the "nice little option" I mentioned earlier.
   
3. The infrastructure needs to be setup, so it's a bit painful.
4. Not quite as easy to use as USB, but it seems I'll have to use of the hard drives as temporary storage for the installer.

It looks like I'll be acquiring a USB 2.0 card to facilitate all the OS installations I was going to try. But IPMI virtual drives not working is going to be a major pain for the average user.

 

[joeschmuck]

I think I'm missing something or maybe there could be a little clarification...
Why would adding a USB 2.0 add-on card allow some OS installations?
Also do I understand correctly that IPMI doesn't actual allow booting from an image?

I haven't downloaded the BIOS and BMC firmware, thought I'd wait until I got it running before I trashed it.

And it's too bad the "nice little option" didn't do what you were hoping for.

 

[Ericloewe]

Why would adding a USB 2.0 add-on card allow some OS installations?

The USB 2.0 card would naturally use the EHCI driver, which FreeNAS does load by default. So I'd be able to boot from a USB device - if the BIOS is smart enough to handle PCI-e USB controllers.

Also do I understand correctly that IPMI doesn't actual allow booting from an image?

It does, but apparently uses the PCH for USB functionality, meaning that its virtual devices use the xHCI driver, which FreeNAS, again, doesn't load by default.

And it's too bad the "nice little option" didn't do what you were hoping for.

I think it's just bugged and wouldn't even work with Windows 7. Though I'll try it out just for fun. Supermicro might like the feedback.

 

[Ericloewe]

So I tried Windows 7 and the results were a bit surprising.

The installer actually loads correctly. Amazingly, Windows Preinstallation Environment seems to use BIOS services to read the install device, since it worked without the compatibility option being turned on. That is, until user input was required. Keyboard and mouse were dead, real and IPMI.

As I expected, turning on that option in the BIOS did absolutely nothing. Keyboard and Mouse still dead.

 

[Ericloewe]

A Haswell Z97 system is behaving similarly. It's an AMI BIOS, very similar under the hood to the one Supermicro uses, but with more bells and whistles.

The installation was hanging at mountroot from /dev/iso9660. Fortunately, messing around with the (more plentiful) USB options on that board did allow the installer to boot. Unfortunately, it's now crapping out when trying to write to the SSD. I hope a quick wipe of the GPT will solve this.

 

[joeschmuck]

Well I hope a new firmware comes out to fix these things. It won't be the first time to be on the cutting edge and have issues ;)

 

[Frallan]

I just installed 9.3 on my X11SSM-F using a spare SSD which I just dd'ed over the image onto. I tried IPMI boot but the installer would fail halfway through the boot sequence where it tried to access the media as cdrom 9660 device.

Where did you find the 1.0b BIOS? I only found X11SSF BIOS when I went looking?

 

[Ericloewe]

Where did you find the 1.0b BIOS? I only found X11SSF BIOS when I went looking?

http://www.supermicro.com/support/resources/results.aspx

I just installed 9.3 on my X11SSM-F using a spare SSD which I just dd'ed over the image onto. I tried IPMI boot but the installer would fail halfway through the boot sequence where it tried to access the media as cdrom 9660 device.

Yeah, that's my experience. I did try with one of the hard drives and it still wouldn't mount, but that might've been a bodged dd (turns out there's an issue with some of the SATA drives that I'm still narrowing down).

I have gotten 9.3 to run by installing to the SSD on a different machine, but, as I explained, it was a pain.

 

[joeschmuck]

So my RAM and CPU arrived, I did not update the motherboard firmware at all and am just running my MemTest86 6.3.0 (free) on the RAM. Wish I had the paid version to test out the ECC RAM but I'm not paying that price for a home one time use. Okay, I might use it every time I build a computer but that is about once every 3 to 5 years.

 

[Ericloewe]

So my RAM and CPU arrived, I did not update the motherboard firmware at all and am just running my MemTest86 6.3.0 (free) on the RAM. Wish I had the paid version to test out the ECC RAM but I'm not paying that price for a home one time use. Okay, I might use it every time I build a computer but that is about once every 3 to 5 years.

And they're kind enough to not give you more than a year of updates, so no support for the latest and greatest when that time comes.

 

[Frallan]

http://www.supermicro.com/support/resources/results.aspx

Cheers! Got it.

 

[joeschmuck]

Just asking before I make a mistake here...

This link is for the IPMI firmware upgrade...
http://supermicro.com/support/bios/firmware0.aspx

I found mine which is listed as firmware 1.13 and my current version is .52 so i think upgrading it would be a good idea. I'm just asking for confirmation before I take this leap.

 

[joeschmuck]

So I updated the BMC to version 1.13. That didn't fix my IPMI access which is crappy. The Virtual drive window never displays anything. I'll figure this thing out, just maybe not tonight. The user guide is poor as well since it switches between many versions of the tool. Maybe I need to try an older version of the tool.

 

[Ericloewe]

The Virtual drive window never displays anything

The one in the IPMI GUI? That one's hopeless, never got it to work.

The one you want is in the IPMI viewer java thingy itself. It works as expected (except for the whole USB issue...).

 

[joeschmuck]

I figured it out.

 

[Mirfster]

Side note for MS Windows users with Skylake: http://windows.microsoft.com/en-us/windows/skylake-support

Looks like they will try their best to make you go up to Windows 10..

Skylake systems supported on Windows 7 and Windows 8.1
Last updated: February 8, 2016
The following is a list of computer manufacturers (OEMs) who have identified hundreds of PC models with Intel's 6th generation of processors, also known as Skylake, that will be supported on Windows 7 and Windows 8.1. Support will be provided until July 17, 2017 and its recommended that these systems be upgraded to Windows 10 as soon as possible. After July 2017, the most critical Windows 7 and Windows 8.1 security updates will be addressed for these configurations, and will be released if the update does not risk the reliability or compatibility of the Windows 7 and Windows 8.1 platform on other devices. For more information regarding Microsoft's support lifecycle for Windows, please see the Support Lifecycle FAQ.

For each of the supported systems listed on the OEM pages, the OEM has committed to additional testing, regular validation of Windows Updates, and publishing drivers and firmware for Windows 10 on Windows Update which will help unlock the security and power management benefits of Windows 10 once the systems are upgraded.

 

[gpsguy]

"Try their best" is putting it kindly.

I am a bit PO'd with their current strong armed tactics to force Win 7 users to upgrade to Windows 10.

Extended support for Win 7 ends on 1/14/2020. If I choose to stay on Win 7, I shouldn't be tricked into an accidental upgrade. I'm running GWX Control Panel on a couple of machines in an effort to block it.

Looks like they will try their best to make you go up to Windows 10..

Sent from my phone

 

[Ericloewe]

Good news everyone!

It's XHCI on Skylake! None of that EHCI rubbish.

Yes, this is the latest FreeNAS 9.10 nightly. Haven't done much testing, but USB works!

Now, it's up to Supermicro to get the BIOS annoyances fixed.

 

[Mirfster]

Nice!