Repeated USB_ERR_TIMEOUT with TrueNAS 12

Lee Spangler

Dabbler
Joined
Sep 30, 2014
Messages
26
I have been receiving repeated messages regarding the handling of USB, USB Hubs and USB allocation since moving to TrueNAS 12. What concerns me is that my drives are continually writing as these errors appear.

I have attempted to resolve this through a bios update (didn't help) and making sure all USB ports were unused as some threads suggested might be helpful.

I then tried to rollback to 11.3 and that now shows the same errors. So, I went back to TrueNAS 12.

I have included the entries from the messages logfile below and have attached the whole messages logfile.

I am using a P9X79 Deluxe Asus motherboard with 12 GB RAM. I boot from a SSD (SATA connected) with no additional devices attached.

Any insights are welcome.
-Lee




Nov 5 00:00:01 WDWNas usb_alloc_device: Failure selecting configuration index 0:USB_ERR_TIMEOUT, port 1, addr 1 (ignored)
Nov 5 00:00:01 WDWNas ugen1.2: <vendor 0x2109 product 0x0810> at usbus1
Nov 5 00:00:02 WDWNas usb_alloc_device: device init 3 failed (USB_ERR_TIMEOUT, ignored)
Nov 5 00:00:02 WDWNas ugen1.3: <Unknown > at usbus1 (disconnected)
Nov 5 00:00:02 WDWNas uhub_reattach_port: could not allocate new device
Nov 5 00:00:02 WDWNas uhub0: at usbus1, port 1, addr 1 (disconnected)
Nov 5 00:00:02 WDWNas ugen1.2: <vendor 0x2109 product 0x0810> at usbus1 (disconnected)
Nov 5 00:00:02 WDWNas uhub0: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
Nov 5 00:00:03 WDWNas uhub0: 4 ports with 4 removable, self powered
Nov 5 00:00:04 WDWNas xhci0: Resetting controller
Nov 5 00:00:09 WDWNas usb_alloc_device: Failure selecting configuration index 0:USB_ERR_TIMEOUT, port 1, addr 1 (ignored)
Nov 5 00:00:09 WDWNas ugen1.2: <vendor 0x2109 product 0x0810> at usbus1
Nov 5 00:00:10 WDWNas usb_alloc_device: device init 3 failed (USB_ERR_TIMEOUT, ignored)
Nov 5 00:00:10 WDWNas ugen1.3: <Unknown > at usbus1 (disconnected)
Nov 5 00:00:10 WDWNas uhub_reattach_port: could not allocate new device
Nov 5 00:00:10 WDWNas uhub0: at usbus1, port 1, addr 1 (disconnected)
Nov 5 00:00:10 WDWNas ugen1.2: <vendor 0x2109 product 0x0810> at usbus1 (disconnected)
Nov 5 00:00:10 WDWNas uhub0: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
Nov 5 00:00:10 WDWNas uhub0: 4 ports with 4 removable, self powered
Nov 5 00:00:11 WDWNas xhci0: Resetting controller
Nov 5 00:00:16 WDWNas usb_alloc_device: Failure selecting configuration index 0:USB_ERR_TIMEOUT, port 1, addr 1 (ignored)
Nov 5 00:00:16 WDWNas ugen1.2: <vendor 0x2109 product 0x0810> at usbus1
Nov 5 00:00:17 WDWNas usb_alloc_device: device init 3 failed (USB_ERR_TIMEOUT, ignored)
Nov 5 00:00:17 WDWNas ugen1.3: <Unknown > at usbus1 (disconnected)
Nov 5 00:00:17 WDWNas uhub_reattach_port: could not allocate new device
Nov 5 00:00:17 WDWNas uhub0: at usbus1, port 1, addr 1 (disconnected)
Nov 5 00:00:17 WDWNas ugen1.2: <vendor 0x2109 product 0x0810> at usbus1 (disconnected)
Nov 5 00:00:18 WDWNas xhci0: Controller reset timeout.
Nov 5 00:00:19 WDWNas xhci0[2318]: Last message 'Controller reset tim' repeated 1 times, suppressed by syslog-ng on WDWNas.local
Nov 5 00:00:19 WDWNas uhub0: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
Nov 5 00:00:20 WDWNas uhub0: 4 ports with 4 removable, self powered
Nov 5 00:00:21 WDWNas xhci0: Resetting controller
Nov 5 00:00:21 WDWNas usb_alloc_device: device init 2 failed (USB_ERR_TIMEOUT, ignored)
Nov 5 00:00:21 WDWNas ugen1.2: <Unknown > at usbus1 (disconnected)
Nov 5 00:00:22 WDWNas uhub_reattach_port: could not allocate new device
Nov 5 00:00:23 WDWNas usb_alloc_device: device init 2 failed (USB_ERR_TIMEOUT, ignored)
Nov 5 00:00:23 WDWNas ugen1.2: <Unknown > at usbus1 (disconnected)
Nov 5 00:00:23 WDWNas uhub_reattach_port: could not allocate new device
Nov 5 00:00:23 WDWNas uhub0: at usbus1, port 1, addr 1 (disconnected)
Nov 5 00:00:23 WDWNas uhub0: <0x1b21 XHCI root HUB, class 9/0, rev 3.00/1.00, addr 1> on usbus1
Nov 5 00:00:23 WDWNas uhub0: 4 ports with 4 removable, self powered
Nov 5 00:00:24 WDWNas xhci0: Resetting controller
 

sretalla

Powered by Neutrality
Moderator
Joined
Jan 1, 2016
Messages
9,702
If you're connecting your pool disks via USB, you're not listening to any of the advice in the forums about how to do it right.

This is exactly the kind of problem that we in the forum are warning about due to the nature of ZFS transaction groups and how bad USB controllers are at handling that huge burst of activity.

You can share more details about your hardware setup and maybe we can advise further, but right now it looks to me like you're just using an inappropriate disk connection path. (you need to go to SAS/HBA or SATA to make it reliable)

Some of what's happening could be related to a scrub and/or potentially SMR disks, so there's that on top of what I already mentioned.
 

Lee Spangler

Dabbler
Joined
Sep 30, 2014
Messages
26
No, I am not accessing any of my disk pools via USB. All my disks are connected via SATA, including the boot drive.

What entry led you to that conclusion? Because it isn't the case at all. Maybe correcting that issue may resolve my errors.

I am considering making a new install and then restoring the config. That seems to be a "radical" step. I don't understand what has caused the problem.

Of course, the ultimate symptom is that eventually the USB ports (all of the them 2.0 and 3.0) all fail to work (including the keyboard). I don't find that to be acceptable.

-Lee
 

HoneyBadger

actually does care
Administrator
Moderator
iXsystems
Joined
Feb 6, 2014
Messages
5,110
Try setting the following tunable as type "loader" and reboot

hw.usb.xhci.use_polling=1

XHCI (USB 3.0) has been problematic for a while now - your board may have an implementation that isn't well-liked in TN12; not sure why it would continue failing on the rollback to 11.3 tough.

Can you disable USB3 in the BIOS and leave it with USB2 only?
 

Lee Spangler

Dabbler
Joined
Sep 30, 2014
Messages
26
I will try the change to the tunable.

I have disabled the USB 3 controllers and still get the errors.

I'll let the thread know how this works.

-Lee
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,972
If you are not using any USB port then I'd recommend disabling USB support in the BIOS completely. But you must have a keyboard and/or mouse connected, right? Are you using one of the ports 1-4 (reference your user manual) which are USB 2.0 ports?

Stupid question time... This looks like one UBS controller (1.1 and 1.2)
vendor 0x2109 product 0x0810
This indicates it's a USB 3 hub, likely the VIA Super speed hub on your motherboard.

So in this situation I'd go into the BIOS and disable the LEGACY USB 3.0 SUPPORT, set to AUTO the LEGACY USB SUPPORT, and disable EHCI HAND-OFF. Now I'd power off the machine and disconnect the keyboard and mouse. Lastly , power on and cross your fingers.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,972
And you can also try to just disable the Asmedia USB 3.0 controller alone, see if that works. Yes, it's a separate selection vice the legacy support.

EDIT: If the messages go away, then start turning on the USB support one at a time, eventually to just leave the problem child disabled.
 

Lee Spangler

Dabbler
Joined
Sep 30, 2014
Messages
26
OK - everyone has been very helpful.

I have turned off the Asmedia USB 3.0 onboard controllers (there are two).
I have also turned Legacy USB to Auto; Legacy 3.0 to Disabled, and EHCI Hand off disabled.
I also disabled the EHCI controllers (even though those are 2.0) and disabled all USB ports except the keyboard.
I then even disabled all USB ports including the keyboard and ...

It is still reporting the messages.
 

joeschmuck

Old Man
Moderator
Joined
May 28, 2011
Messages
10,972
Wow, I'm not sure why the error messages are still occurring when you disabled the USB support. I wish I had another trick up my sleeve but I don't.

Well I searched the internet and found that you are not the only one to have this problem and it also exists in FreeBSD 12 (what TrueNAS is based on). Suggestions are to create that tunable for loader that was previously mentioned and also add
Code:
hint.ahci.0.msi=0


I then tried to rollback to 11.3 and that now shows the same errors. So, I went back to TrueNAS 12.
Were you previously on FreeNAS 11.3? And which subversion were you on. U5 or something earlier? And when you rolled back, how did you do that exactly?

Good luck, I hope you figure it out.
 

Lee Spangler

Dabbler
Joined
Sep 30, 2014
Messages
26
On the rollback, I set 11.3-U5 as a "keep" boot environment and made sure I did NOT upgrade my ZFS. I then go into system/boot and set it to 11.3-U5 as the active environment. I just don't understand why I can't get rid of the messages even when I disable USB entirely. In my google searches, there appears to be an issue with FreeBSD and some ASUS motherboards which generate this message.
 
Joined
Aug 12, 2021
Messages
6
I seem to be having the exact same issue with this configuration:
Xeon E5-1230v6
SuperMicro X11SSH-LN4F
2 x Kingston 16 GB ECC
2 x Kingston Valueram 8 GB ECC
6 x HGST Ultrastar 7K6000
Toshiba 16gb usb 2
nmve 120 gb Samsung

None of the settings in BIOS have helped us boot so far. We're stuck in this loop:
2021-08-13_13-40-39_screenshot.png
 

Inspecteur

Cadet
Joined
Aug 29, 2021
Messages
4
If you are not using any USB port then I'd recommend disabling USB support in the BIOS completely. But you must have a keyboard and/or mouse connected, right? Are you using one of the ports 1-4 (reference your user manual) which are USB 2.0 ports?

Stupid question time... This looks like one UBS controller (1.1 and 1.2)

This indicates it's a USB 3 hub, likely the VIA Super speed hub on your motherboard.

So in this situation I'd go into the BIOS and disable the LEGACY USB 3.0 SUPPORT, set to AUTO the LEGACY USB SUPPORT, and disable EHCI HAND-OFF. Now I'd power off the machine and disconnect the keyboard and mouse. Lastly , power on and cross your fingers.
The configuration recommended by joe in the BIOS worked, but I also had to disable the intel xHCI mode and set it to DISABLED. I also wasn't sure if a usb device connected to the NAS could have been casing this USB_Timeout error. Ended up finding a tiny usb drive plugged in with a fresh install of TrueNAS on it. I think it was meant to be a way to quickly get back up in running in case my main boot drive fails, but that may have been the root of the problem, I also removed it after the fact.

Might be better practice to keep it handy but not plugged in 24/7, lesson learned
 
Top