Frequent reboots since 12.0-U2 update (cxgb driver panic)

[Andrew Ostrom]

I installed the 12.0-U2 update to my system (was running 12.0-U1) a few days ago. Up until then my system was running with no issues. Since the U2 update I get an alert several times a day stating that the system has rebooted unexpectedly. I am not a Unix/Linux guru by any stretch of the imagination - my expertise was in the now obsolete VAX/VMS and RSTS/E operating systems.

My configuration is:

Supermicro SuperChassis 846E1-R1200B Dual Xeon, 24 x HDD Storage Server​

Supermicro Motherboard X9DRi-F​

Dual Intel Xeon E5-2690 v2 3Ghz​

128GB Ram (16x 8GB PC3-12800R)​

BPN-SAS2-846EL1 Backplane​

2 LSI SAS9207-8i​

2 120GB Kingston SSD Plus drives for system​

Dual PWS-1K21P-1R 1200W 80 Gold Plus power supplies​

Vdev1 (RAID-Z2) - 8 x 3TB Seagate SAS ES.3 ST3000NM0023​

Vdev2 (RAID-Z2) - 8 x 4TB Seagate SAS ES.3 ST4000NM0023​

Vdev3 (RAID-Z2) - 8 x 4TB Seagate SAS ES.3 ST4000NM0023​

Chelsio 10Gbe (10Gbase-SR using fiber to Aruba switch)​

Looking at the files info.* in /data/crash, they are all similar to this:

Dump header from device: /dev/da23p1​

Architecture: amd64​

Architecture Version: 4​

Dump Length: 1607168​

Blocksize: 512​

Compression: none​

Dumptime: Thu Feb 18 09:24:00 2021​

Hostname: freenas.ostrom.org​

Magic: FreeBSD Text Dump​

Version String: FreeBSD 12.2-RELEASE-p3 7851f4a452d(HEAD) TRUENAS​

Panic String: trying to coalesce 8 packets in to one WR​

Dump Parity: 2839218024​

Bounds: 0​

Dump Status: good​

I'm not sure exactly what this means, but the SMART status for da23 is good - I have started a "long" test on da23, which will take a few hours.

I think I am posting the right files here, but please tell me if I am missing anything.

Thanks for the help.

 

[Samuel Tai]

This isn't a problem with da23. The crash is in the cxgb driver trying to combine packets. Try disabling all offload options for this NIC.

 

[Samuel Tai]

See http://web.mit.edu/freebsd/head/sys/dev/cxgb/cxgb_sge.c. Your panic is definitely in the cxgb driver, either in the transmit or the receive queue.

 

[Samuel Tai]

To regain stability, try rolling back to 12.0-U1, and then upgrading manually to U1.1 to avoid the data loss bug with asynchronous copy-on-write. You can get a manual update for U1.1 from http://download.freenas.org/12.0/STABLE/U1.1/TrueNAS-12.0-U1.1-manual-update.tar. Then, submit a bug report. This may be a driver regression from upstream.

 

[Andrew Ostrom]

OK, I will do that. Thanks for the help.
------------
Update - I am now running 12.0-U1.1, and will file a bug report in Jira.
------------
Update 2 - Bug filed - https://jira.ixsystems.com/projects/NAS/issues/NAS-109486?filter=allo*****sues
------------
Update 3 - 2/22/2021 - the bug has been reported by at least one more user, and reproduced by a member of the dev team, so it's being worked. Thanks!
-----------
Update 4 - 2/22/2021 - A fix has been developed and is being tested.
-----------
Update 5 - 2/24/2021 - The fix has been committed to TrueNAS 12.0-U3 and also to FreeBSD main.