Loss of connection stability upgrading from Core 13.0-U3.1 to 13.0-U4

[firsway]

Hi, I have observed issues with my network after performing an upgrade of TrueNAS Core 13.0-U3.1 to 13.0-U4.
Initially I thought this might just be related to the Realtek based NIC onboard the MB, however then realised that pings to an IP interface on my Intel 82599 10GB LC-LC card were also failing. I noticed that pings both to the Realtek and to the Intel had spurious success, enough to take out my local iSCSi availability, and also remote access to NFS for my Plex service.
After trying a few things like restarting the interfaces, a reboot, checking the cables, the network switch, without any improvement, I rolled back to 13.0-U3.1 which removed the issue.
Here is my interface driver list;


root@freenas3[~]# pciconf -lv | grep -A1 -B3 network
ix0@pci0:5:0:0: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10fb subvendor=0x15d9 subdevice=0x0611
vendor = 'Intel Corporation'
device = '82599 10 Gigabit Dual Port Network Connection'
class = network
subclass = ethernet
ix1@pci0:5:0:1: class=0x020000 rev=0x01 hdr=0x00 vendor=0x8086 device=0x10fb subvendor=0x15d9 subdevice=0x0611
vendor = 'Intel Corporation'
device = '82599 10 Gigabit Dual Port Network Connection'
class = network
subclass = ethernet
re0@pci0:7:0:0: class=020000 rev=0x16 hdr=0x00 vendor=0x10ec device=0x8168 subvendor=0x1458 subdevice=0xe000
vendor = 'Realtek Semiconductor Co., Ltd'
device = 'RTL8111/8168/8411 PCI Express Gigabit Ethernet Controller'
class = network
subclass = ethernet

There wasn't a great deal in the other logs that I could see that looked obvious. Is there any other specific logs that I should look at and provide to help diagnose this?
BTW I rolled back the system on 22nd March, and connectivity has been solid since. All connections are uplinked to a Cisco Nexus 3064-PQ. The fibre links ix0 and ix1 are trunk links, re0 is configured to talk to an access port.

Any ideas? Thanks in advance

 

[firsway]

Anybody with any thoughts on this please?

 

[Samuel Tai]

Are all your interfaces on the same subnet? This is a known unsupported configuration, which can result in random behavior.

 

[firsway]

Are all your interfaces on the same subnet? This is a known unsupported configuration, which can result in random behavior.

`No, they are all separate subnets. re0 is on my main /24 subnet. ix0 is a trunk with currently 2x VLANs defined, both separate /28, ix1 is also a trunk with 1x /28 VLAN. There is no overlap onto the /24 from any of the /28s, all completely separate IP ranges, in any case I don't think the GUI will allow such a thing..

 

[Samuel Tai]

Do any of these interfaces have associated bridges or LAGGs? Best practice for upgrades is to delete network configuration down to just your management interface beforehand, and then recreate the VLANs/LAGGs/bridges after a successful upgrade.

 

[firsway]

Do any of these interfaces have associated bridges or LAGGs? Best practice for upgrades is to delete network configuration down to just your management interface beforehand, and then recreate the VLANs/LAGGs/bridges after a successful upgrade.

Hi, No bridges or LAGGs. Not sure if there is going to be an effort to delete/recreate config for each upgrade as the 10G interfaces have iSCSi portals bound to the VLAN addresses for those interfaces? These portals are linked to datastores for my ESXi environment. In any case these portals have been there forever, and the setup has persisted through multiple upgrades in the past. First time this has been a problem to be honest.

 

[blanchet]

Few years ago, I had stability issue after upgrading TrueNAS to version 12.0.
Indeed, ZFS replication was faster in the new version and it disturbed ESXi.
Finally I have used a dedicated NIC for ZFS replication, and it has solved my issue.

ESXi + TrueNAS 12.0 NFS Lost Connection

I have total 6 Esxi Hosts from 6.7 ~ 7.x and mounted 3 TrueNAS 12.0 U5 to U6 NFS share, I found the all ESXi hosts will lost the NFS datastore suddenly in all TrueNAS hosts randomly without any error log, once the connectivity lost, the storage capacity will shown 0 and the mounted guest OS will...

www.truenas.com

 

[Kataryno]

Hello firsway,

Did you found out anything new related with this bug? It happened to me today, the same behaviour after upgraded to U4. Network instability, jails not starting, etc.

 

[firsway]

Hello firsway,

Did you found out anything new related with this bug? It happened to me today, the same behaviour after upgraded to U4. Network instability, jails not starting, etc.

Hi, sorry for delay responding. Sorry, I did not. Only responses I've had are what you see above. I'm going to wait until a further upgrade comes out, and then perhaps try again with that.

 

[Kataryno]

Hi, sorry for delay responding. Sorry, I did not. Only responses I've had are what you see above. I'm going to wait until a further upgrade comes out, and then perhaps try again with that.

No problem! Thanks for the reply. In fact the U5 will be released soon, maybe could help us with our problem.

Strange how no one else complained about this, we are the unlucky ones :P