Supermicro and Kaby Lake issue?

[beczkabe]

Hi,

I've started completing the hardware for my (ESXi) FreeNAS setup and I had some problems (and regrets) already.
I wanted to ask those of you who also have Kaby Lake setup if you also experience such issues.

I've bought:
- Supermicro X11SSH-LN4F
- E3-1245v6 (stock cooler)
- 1x Samsung DDR4 16GB 2400GB CL17 (M391A2K43BB1-CRC)

I've put those into Lian-Li case with Corsair 750W psu.
I've updated mobo bios to the newest: R 2.0c
I've updated mobo IPMI firmware to the newest: R 1.39.

I've started with Memtest86+ for 20h, there were no errors.

With default bios settings every system I boot hangs. That was the case for FreeNAS, ESXi, bunch of Linux distros (Arch, Ubuntu, Fedora). The only one that didn't hang was Openmediavault which I guess had old kernel. Even FreeBSD 12-current hanged with ACPI on.

I've wasted a lot of time looking into possible fixes. Moved memory between slots, checked cpu temps etc and messed with the bios settings.
The only thing that helped with the situation was disabling completely CPU C-States in Bios (I've tried disabling one option after another from C-States as well).
With C-States disabled I could boot ESXi for example.

Another option was disabling ACPI under FreeBSD (FreeNAS) and Linux. That helped as well.
With ACPI disabled I've ran Intel Processor Diagnostic Tool multiple times - all successful. I've also burned memory with Google's stressapptest (which according to the Internet can detect mem issues faster than Memtest86+) - also no issues detected.

So the question is - do you guys have similar experience with Kaby Lake platform? How one can better check if mobo/cpu has no issuess? Finally can you recommend the system with specific version that works with Kaby Lake & CPU C-States turned on and ACPI turned on so I could test my mobo/cpu/ram?

I've already ordered more memory, better cooler, perc h310, different psu; wdc 8tb red drives are being checked in different computer.

My dream was to have ESXi server with FreeNAS, pfsense, plex, etc vms; I've taken Kaby Lake cpu with igpu so I could do the passthrough igpu to vm and have hardware decoding when possible.
I've chosen LN4F version so I could passthrough 2x nic if needed; both ssh-f & ssh-ln4f have 2x pcie 8x so I've just taken the one with more network cards.

Taking into consideration how much time I've already wasted on this setup I already regret I haven't bought xeon-d or x10 :(

Any advices are welcome. If I could point the faulty hardware I could RMA it but currently I have no idea how to do it since all checks passed.
Not using ACPI will drain more money from my pocket but I guess I can live with this. I'd like to know if my hardware is healthy though.

Thanks,
Beka

 

[joeschmuck]

First of all a single stick of RAM must be installed into the blue slot furthest away from the CPU, don't just move it around thinking it will work. This is in the user manual.

I wouldn't have ordered any additional parts until you figured out what is going on.

Lastly, it sounds like you have tried a lot of things so you should probably open up a ticket with Supermicro Tech Support. They are very good and will respond within a few days if not sooner.

Things you could look into are: Reseating the CPU, making sure nothing is bent.
CPU Stress Test.

Your problem must be the Motherboard or CPU based on your comments in your posting.

When that second stick of RAM comes in, pop that into the second blue slot and try again. It shouldn't make a difference however what do you have to lose?

 

[wblock]

Modern systems pretty much require ACPI. It is not likely to be useful without it.

 

[Ericloewe]

Lastly, it sounds like you have tried a lot of things so you should probably open up a ticket with Supermicro Tech Support.

+1

The X11SSH-F/LN4F BIOS may be broken in that version.

 

[r0nski2000]

I can't help you, but fwiw, I am running e3-1245v6 with x11ssm(whatever bios it came with, I think 2.0a), 1x16gb corsair ram, and had no problems booting up. The only time I had problems was with the h200i cards installed.

 

[Johnnie Black]

The only time I had problems was with the h200i cards installed.

Just in case you don't already know it works after the tape mod, same for the perc h310.

 

[beczkabe]

Thanks guys for your comments!

I've contacted the shop and they have asked me to do several other tests.
Now I've sent the motherboard to the shop - they will test it with different cpu & mem to determine if that is actually mobo fault or not.

I've tested bunch of other operating systems - all of them hanged during boot. I'll try to post as much info as I can - it may help somebody else.

RedHat Enterprise Linux 7.3 boot cd:
- non-uefi boot mode - hangs
- uefi boot mode - hangs
- rescue non-uefi boot mode - hangs

RedHat Enterprise Linux 6.7 boot cd:
- non-uefi boot mode - hangs

Ubuntu 14.04.03 boot cd:
- non-uefi boot mode - hangs
- uefi boot mode - hangs
- non-uefi boot mode & acpi=off - hangs

FreeBSD 10.2 boot cd:
- non-uefi boot mode - installer starts but hangs immediatelly
- non-uefi boot mode & acpi=off - panic

SLE 12 boot dvd:
- non-uefi boot mode - hangs
- uefi boot mode - hangs
- non-uefi boot mode & acpi=off - kernel panic

Windows 10 dvd:
- non-uefi boot mode - hangs
- uefi boot mode - hangs


From not supported operating systems I've tried:
RedHat Enterprise Linux 7.4 boot dvd:
- non-uefi boot mode - hangs
- non-uefi boot mode & acpi=off - hangs

ESXi 6.5 u1:
- non-uefi boot mode - hangs

FreeBSD 12-current boot cd:
- hangs

 

[joeschmuck]

Just in case you don't already know it works after the tape mod, same for the perc h310.

You don't need to do this for the motherboard in question, there is a pair of jumpers to assist in this called SMBus to PCI. I don't like placing tape on a edge connector, it could fall off and become debris in the slot.

 

[joeschmuck]

Now I've sent the motherboard to the shop - they will test it with different cpu & mem to determine if that is actually mobo fault or not.

Hope they figure it out swiftly so you can get back on your feet and get your system established.

 

[Johnnie Black]

You don't need to do this for the motherboard in question, there is a pair of jumpers to assist in this called SMBus to PCI.

Yeah, but won't that disable the SMbus to PCI for all slots? It's supposed to improved performance when it's on, though I agree that don't really like using tape despite never having any issues with it.

 

[joeschmuck]

Yeah, but won't that disable the SMbus to PCI for all slots? It's supposed to improved performance when it's on, though I agree that don't really like using tape despite never having any issues with it.

When I ran this I didn't see any perfomance issues. All I could say is I didn't like the PERC H310 myself, I thought it was a slow card overall, but YMMV.

 

[Johnnie Black]

I didn't like the PERC H310 myself, I thought it was a slow card overall,

Same performance as any other LSI SAS2008 controller once crossflashed, maybe yours had issues.

 

[joeschmuck]

Same performance as any other LSI SAS2008 controller once crossflashed, maybe yours had issues.

It might have but it was free so I can't really complain. Or maybe I just had higher expectations ;)

 

[Chris Moore]

Yeah, but won't that disable the SMbus to PCI for all slots? It's supposed to improved performance when it's on, though I agree that don't really like using tape despite never having any issues with it.

I have five of the Dell H310 cards right now in various systems and never had a problem with SMbus. No tape, no jumpers. That's a problem only with certain system boards that don't know how to handle the card.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Johnnie Black]

That's a problem only with certain system boards that don't know how to handle the card.

I never said it was a general problem, I have several my self on Supermicro X9 series boards, the X11 series require the mod (or jumper change).

 

[r0nski2000]

Just in case you don't already know it works after the tape mod, same for the perc h310.

I did the jumper mod - only mentioned the issue in case it was applicable to the op.

I couldn't figure out what the side effects from the jumper mod are, and since I couldn't notice any, I just left it. My performance requirements are pretty low, so even if there are some issues I probably wouldn't notice/care.

 

[beczkabe]

Short summarize after almost a year.

With no other ideas I sent the motherboard to the shop.
They have tested it and said that it worked for them :) but Supermicro replaced the mobo anyways.

I've completed the setup with replacement motherboard and haven't had any issues since then. Everything just works.

 

[Chris Moore]

What is the final hardware configuration after all this?