SuperMicro X10SRL-F-O Faulty?

[gbooker]

I have a Supermicro X10SRL-F-O running FreeNAS for almost 3 months now and lately I've been experiencing issues with the USB buses spontaneously disconnecting. This was a significant issue when I was previously booting off a mirror of thumb drives, but after I started booting off a SATA DOM, the problem persisted.

Symptoms:
When it occurs, the following shows up in messages:

Nov 17 17:00:03 pond ugen0.2: <vendor 0x8087> at usbus0 (disconnected)
Nov 17 17:00:03 pond uhub2: at uhub0, port 1, addr 2 (disconnected)
Nov 17 17:00:03 pond ugen1.2: <vendor 0x8087> at usbus1 (disconnected)
Nov 17 17:00:03 pond uhub3: at uhub1, port 1, addr 2 (disconnected)
Nov 17 17:00:03 pond ugen0.3: <vendor 0x0557> at usbus0 (disconnected)
Nov 17 17:00:03 pond uhub4: at uhub2, port 6, addr 3 (disconnected)
Nov 17 17:00:03 pond ugen1.3: <CPS> at usbus1 (disconnected)
Nov 17 17:00:03 pond ugen1.4: <vendor 0x05e3> at usbus1 (disconnected)
Nov 17 17:00:03 pond uhub5: at uhub3, port 2, addr 4 (disconnected)
Nov 17 17:00:03 pond ugen0.4: <vendor 0x0557> at usbus0 (disconnected)
Nov 17 17:00:03 pond ukbd0: at uhub4, port 1, addr 4 (disconnected)
Nov 17 17:00:03 pond ugen1.5: <Mitsumi Electric> at usbus1 (disconnected)
Nov 17 17:00:03 pond uhub6: at uhub5, port 4, addr 5 (disconnected)
Nov 17 17:00:03 pond ums0: at uhub4, port 1, addr 4 (disconnected)
Nov 17 17:00:03 pond ugen1.6: <Logitech> at usbus1 (disconnected)
Nov 17 17:00:03 pond ums1: at uhub6, port 1, addr 6 (disconnected)
Nov 17 17:00:03 pond ugen1.7: <Mitsumi Electric> at usbus1 (disconnected)
Nov 17 17:00:03 pond ukbd1: at uhub6, port 3, addr 7 (disconnected)
Nov 17 17:00:03 pond uhid0: at uhub6, port 3, addr 7 (disconnected)
Nov 17 17:00:04 pond usbus0: port reset timeout
Nov 17 17:00:04 pond usbus1: port reset timeout
Nov 17 17:00:04 pond uhub_reattach_port: port 1 reset failed, error=USB_ERR_TIMEOUT
Nov 17 17:00:04 pond uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling port 1
Nov 17 17:00:04 pond uhub_reattach_port: port 1 reset failed, error=USB_ERR_TIMEOUT
Nov 17 17:00:04 pond uhub_reattach_port: device problem (USB_ERR_TIMEOUT), disabling port 1

(With the last two lines repeating numerous times) Shortly after this is logged, the system reboots, and works fine for an average of a few days before it happens again.

Twice, the above did not happen but the ethernet changed to to 100Mbps instead of 1Gbps and stopped responding completely. The messages log only reported DNS and route failures. It notices when I disconnect and reconnect the ethernet cable, but network never responds until reboot.

System
X10SRL-F-O, with Xeon E5 1620v3 processor, 32G of ECC Crucial RAM (4x8G), and 3 IBM M1015 cards in IT firmware (P16). Internal sensors consistently report temperatures below 45°C. BIOS and Firmware are both running the latest version (1.0c and 02.18. I updated them both in the hopes of fixing the problem). USB devices have been in total: a USB hub connecting a keyboard and mouse, CyberPower UPS, and for the first two months, two Sandisk thumb drives. No other USB devices have ever been connected to this machine.

Tech Support
I contacted Supermicro to ask if my board was faulty and the response was to immediately blame FreeNAS. In my reply I told them that I know of many others who use FreeNAS on this same MB without issue. The next reply suggested that I turn of XHCI in the BIOS (note: there is no "XHCI" setting to disable, but there is a "XHCI Handoff" that can be disabled. I asked if this is the correct setting and have not yet received a response). Does this change make sense to anyone?


So, I have nearly no confidence in Supermicro resolving this issue so I thought I'd ask here. I know people on this forum have the same board so is there some setting in my BIOS that is screwing this up? Is my MB faulty and in need of replacement (it is still in warranty but I have to act quickly)? Thanks for the help.

 

[joeschmuck]

Why don't you list your hardware setup (be as detailed as you can) and the version of FreeNAS you are running.

I'd like to tell you that USB other than for a few things is not well supported under FreeNAS but you need to answer the question so we can offer some real help.

 

[gbooker]

Sorry for not including the list the first time. I'm pretty certain I got everything here but if anything is missing, let me know:

Hardware:

• Chasis: Norco 4224 with 120mm Fanwall with backplanes
  
• Motherboard: Supermicro X10SRL-F-O
• Processor: Intel Xeon E5-1620v3
• Memory: 4x8G Crucial ECC memory: DDR4 2133 (PC4 17000) (CT4K8G4RFS4213)
• Boot drive: 16G SATA DOM (Supermicro SSD-DM016-PHI)
  
• HBAs: 3x IBM M1015 Flashed with IT firmware P16
• Power Supply: Antec 750W (HCP-750)
• CPU Cooler: NH-U9DXi4 (running around 80%)
• Fan Wall Fans: 3xNF-P12 (running at full speed)
• Rear fans: 2xNF-A8 (running around 80%)
• UPS: CyberPower CP1000PFCLCD PFC

Software:

• FreeNAS 9.3 running on metal on Stable tree. Currently up to date (201511040813)

Main Pool:

• raidz2: 6x2TB WD Black (WD2002FAEX)
  
• raidz2: 6x4TB WD Red (WD40EFRX)

USB devices:

• Above mentioned UPS (now disconnected but previously was always connected)
• Belkin travel USB hub connecting: (sometimes connected but failures have occurred both with and without)
  
  • Old Apple keyboard (the thick one) and single button Apple USB mouse (the generation after the puck)
• Previously booted off of two Sandisk 16G thumb drives but now are disconnected. Didn't change failure rate.

Ethernet:

• First port connected to Netgear gigabit switch
• IPMI also connected to same switch
• Second port unconnected.

Lastly, I have a monitor connected to the VGA which stays off unless there's a problem.

 

[joeschmuck]

I would not use a USB hub with FreeNAS just to simplify the troubleshooting.

1) Turn off your FreeNAS.
2) Unplug all your USB devices.
3) Turn on your FreeNAS.
4) In the GUI, System --> Advance --> Show Console Messages in Footer, checked and Save it. You should have messages at the bottom of your GUI window.
5) Click on the Console Messages in the footer window to enlarge it.
6) Plug in the UPS to a USB 2.0 port (black, not blue). Watch the messages pop up and you are looking to see which port the UPS connect to.
7) Configure your UPS to use that port. Hopefully you know how to do that. Set the driver to "Cyber Power Systems ups 2".

Now see how things work, do not use the USB 3.0 connections (blue) as these are less compatible depending on the system. If your system works flawlessly for a period of time, you could use the second USB 2.0 port for a keyboard but when troubleshooting a problem, you need to control your testing and not be in a hurry.

 

[gbooker]

In the interest of controlling my testing, I'm not going to conduct these steps right now. Thus far the machine has been up for 2 days and I want to see if it stays so. Currently I am testing the "XHCI Handoff" disabled with the UPS disconnected. If the machine does its lockup/reboot or I deem the test successful (likely another week), then I'll follow your steps exactly and report back here. Note: This is the first boot with the UPS disconnected from power on.

I can tell you I've done these steps in the past except my UPS was connected at boot. In other words, I've had instances where the hub, keyboard, and mouse were never connected from boot to failure. Previously, I used dmesg to find the UPS port which was /dev/ugen1.9. I had the UPS configured and FreeNAS did see it as well as notice power interruptions.

I have never touched the USB 3 ports with anything.

 

[joeschmuck]

When you say the UPS is not connected, you do mean just the USB cable, not that the computer is not on UPS power, right?

Well hope things work out.

 

[gbooker]

Yes, I've always been talking about the USB cable only. The computer has always been pulling power through the UPS; I see no reason to change that.

 

[DearestDreamer]

Your setup looks similar to mine, except for the chassis and I'll probably be running without a UPS. I'll also be putting the OS on SSDs instead of SATADOMs. Once I assemble everything I'll tell you how my setup comes along, since we're both using the same motherboard, CPU and HBA.

 

[gbooker]

Update: Having previously disabled XHCI handoff, I have now hit 12 days of uptime. The UPS's data cable (USB) is not connected but the Belkin hub along with the keyboard and mouse stayed connected from boot (I didn't disconnect them because I didn't want to taint the experiment).

 

[gbooker]

Just in case anyone else runs across this thread via a search or whatnot, I have continued to enjoy continual uptime with the system only going down for software updates. It has been nearly two months since my last issue. Given the USB devices are connected, it's seems that disabling "XHCI Handoff" in the BIOS has solved this issue.