Network connectivity dropped in all Jails restarting them doesn't work. How to troubleshoot this issue?

[Mannekino]

Hi,

For the second time in the past 1-2 weeks network connectivity inside all my Jails stopped. Restarting the Jails doesn't help restore connectivity.

The Jails are starting fine but afterwards they cannot be pinged from other devices in the network nor I can ping to other devices or resolve hostnames inside the Jail.

The previous time this happened I restarted the entire TrueNAS server but it's currently running a scrub and obviously this doesn't provide a proper solution.

I've checked various log files on the TrueNAS server but so far I've been unable to find any errors.

How can I go about and start troubleshooting this problem? What logs and other things do I need to check?

Thank you.

 

[Patrick M. Hausen]

Did you change and save the network settings? In that case a reboot is the only remedy I know. Changing anything about the network will disconnect running jails and VMs in my experience.

 

[Mannekino]

Nope, I haven't changed any settings. I just loaded up all the web interfaces of my Jails to do some stuff and none of them opened up. After looking at the network graph it seems like everything stopped yesterday (15th) around 08:45 my time. I tried looking through some log file to find something happening at that time but so far I've found nothing.

I'm planning on doing a reboot after the scrub is done which should be in a couple of hours. But since this is the second time in the last week I'm afraid some underlaying issue has sprung up which is likely to repeat itself.

 

[Mannekino]

Unfortunately it happened again. Just tried to load up all the web interfaces of my Jails and none of them are opening. Traffic dropped around 02:00 last night. I'm at a loss to where even start troubleshooting this. Any help is greatly appreciated. I will postpone restarting my TrueNAS server in case I get some replies from people who want to help me and check stuff while it's not working.

It's just the Jails. I can still access the TrueNAS web UI and through SSH.

 

[Patrick M. Hausen]

ifconfig and dmesg on the NAS host itself would be a good start.

Second - standard question: did you disable hardware offloading in the settings for the physical interface your jails are connected to?

 

[Mannekino]

Hi Patrick, thanks for you reply.

I didn't change any network settings in a long time. The server has been running well for over a year. After it first happened I decided to both upgrade TrueNAS and the Jails but that didn't help. I only have one physical interface connected which is also being used for IPMI.

I have a test Jail also which I did start after the last reboot. I just started it and there's no network connectivity in that Jail also. I can ping from inside the Jail to outside targets and I can't ping the IP address of the Jail from my PC. I can experiment with the settings of this Jail if you want.

Here is the output of ifconfig

Code:

# ifconfig
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=81249b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,LRO,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:25:90:74:57:e1
        media: Ethernet autoselect
        status: no carrier
        nd6 options=1<PERFORMNUD>
em1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: em1
        options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
        ether 00:25:90:74:57:e0
        inet 192.168.178.3 netmask 0xffffff00 broadcast 192.168.178.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:dc:dd:47:50:00
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.9 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 14 priority 128 path cost 2000
        member: vnet0.8 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 13 priority 128 path cost 2000
        member: vnet0.7 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000
        member: vnet0.6 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000
        member: vnet0.5 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 10 priority 128 path cost 2000
        member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 7 priority 128 path cost 2000
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 6 priority 128 path cost 2000
        member: em1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 2 priority 128 path cost 20000
        groups: bridge
        nd6 options=1<PERFORMNUD>
vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: grafana as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:3e:10:b8
        hwaddr 02:83:d2:34:f6:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: transmission as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:bb:2e:01
        hwaddr 02:b3:57:de:83:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: jackett as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:96:a7:32
        hwaddr 02:62:fb:8d:2d:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: plex as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:7e:eb:bb
        hwaddr 02:d3:e8:31:99:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.5: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: sabnzbd as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:2e:9c:cb
        hwaddr 02:7c:c9:13:3b:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.6: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: tautulli as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:6f:2e:44
        hwaddr 02:a3:2f:1b:18:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.7: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: sonarr as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:60:65:8c
        hwaddr 02:dc:03:ea:1c:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.8: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: lidarr as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:bf:cd:2e
        hwaddr 02:51:ca:d3:23:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>
vnet0.9: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: radarr as nic: epair0b
        options=8<VLAN_MTU>
        ether 02:ff:60:e3:63:cb
        hwaddr 02:c0:84:41:19:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=1<PERFORMNUD>

And dmesg

Code:

# dmesg
---<<BOOT>>---
Copyright (c) 1992-2020 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 12.2-RELEASE-p9 2ee62d665f0(HEAD) TRUENAS amd64
FreeBSD clang version 10.0.1 (git@github.com:llvm/llvm-project.git llvmorg-10.0.1-0-gef32c611aa2)
VT(efifb): resolution 800x600
CPU: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz (2400.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1dbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000000<IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 34359738368 (32768 MB)
avail memory = 33141325824 (31606 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <SUPERM SMCI--MB>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 hardware threads
random: unblocking device.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
Launching APs: 1 7 2 5 3 6 4
Timecounter "TSC-low" frequency 1200036432 Hz quality 1000
random: entropy device external interface
kbd1 at kbdmux0
mlx5en: Mellanox Ethernet driver 3.5.2 (September 2019)
nexus0
efirtc0: <EFI Realtime Clock> on motherboard
efirtc0: registered as a time-of-day clock, resolution 1.000000s
aesni0: No AES or SHA support.
padlock0: No ACE support.
cryptosoft0: <software crypto> on motherboard
acpi0: <SUPERM SMCI--MB> on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
atrtc0: registered as a time-of-day clock, resolution 1.000000s
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
apei0: <ACPI Platform Error Interface> on acpi0
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 16 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mps0: <Avago Technologies (LSI) SAS2008> port 0xe000-0xe0ff mem 0xdfa40000-0xdfa4ffff,0xdfa00000-0xdfa3ffff irq 16 at device 0.0 on pci1
mps0: Firmware: 20.00.07.00, Driver: 21.02.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
em0: <Intel(R) PRO/1000 Network Connection> port 0xf020-0xf03f mem 0xdfc00000-0xdfc1ffff,0xdfc25000-0xdfc25fff irq 20 at device 25.0 on pci0
em0: Using 1024 TX descriptors and 1024 RX descriptors
em0: Using an MSI interrupt
em0: Ethernet address: 00:25:90:74:57:e1
ehci0: <Intel Cougar Point USB 2.0 controller> mem 0xdfc24000-0xdfc243ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
usbus0: 480Mbps High Speed USB v2.0
pcib2: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.4 on pci0
pci3: <ACPI PCI bus> on pcib3
em1: <Intel(R) PRO/1000 Network Connection> port 0xd000-0xd01f mem 0xdfb00000-0xdfb1ffff,0xdfb20000-0xdfb23fff irq 16 at device 0.0 on pci3
em1: Using 1024 TX descriptors and 1024 RX descriptors
em1: Using 2 RX queues 2 TX queues
em1: Using MSI-X interrupts with 3 vectors
em1: Ethernet address: 00:25:90:74:57:e0
ehci1: <Intel Cougar Point USB 2.0 controller> mem 0xdfc23000-0xdfc233ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
usbus1: 480Mbps High Speed USB v2.0
pcib4: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci4: <ACPI PCI bus> on pcib4
vgapci0: <VGA-compatible display> mem 0xde000000-0xdeffffff,0xdf800000-0xdf803fff,0xdf000000-0xdf7fffff irq 23 at device 3.0 on pci4
vgapci0: Boot video device
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Cougar Point AHCI SATA controller> port 0xf070-0xf077,0xf060-0xf063,0xf050-0xf057,0xf040-0xf043,0xf000-0xf01f mem 0xdfc22000-0xdfc227ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
ahciem0: <AHCI enclosure management bridge> on ahci0
acpi_button0: <Power Button> on acpi0
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart0: console (115200,n,8,1)
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
uart2: <16550 or compatible> port 0x3e8-0x3ef irq 10 on acpi0
ipmi0: <IPMI System Interface> port 0xca2,0xca3 on acpi0
ipmi0: KCS mode found at io 0xca2 on acpi
ichwd0: <Intel Cougar Point watchdog timer> on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc87ff pnpid ORM0000 on isa0
superio0: <Nuvoton NCT6776> at port 0x2e-0x2f on isa0
wbwd0: <Nuvoton NCT6776 (0xc3/0x33) Watchdog Timer> at WDT ldn 0x08 on superio0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
Timecounters tick every 1.000 msec
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
ugen1.1: <Intel EHCI root HUB> at usbus1
ugen0.1: <Intel EHCI root HUB> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ipmi0: IPMI device rev. 1, firmware rev. 3.52, version 2.0, device support mask 0xbf
ipmi0: Number of channels 2
ipmi0: Attached watchdog
ipmi0: Establishing power cycle handler
Trying to mount root from zfs:freenas-boot/ROOT/12.0-U5 []...
Root mount waiting for: CAM usbus0 usbus1
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
Root mount waiting for: CAM usbus0 usbus1
ugen0.2: <vendor 0x8087 product 0x0024> at usbus0
uhub2 on uhub1
uhub2: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus0
ugen1.2: <vendor 0x8087 product 0x0024> at usbus1
uhub3 on uhub0
uhub3: <vendor 0x8087 product 0x0024, class 9/0, rev 2.00/0.00, addr 2> on usbus1
Root mount waiting for: CAM usbus0 usbus1
uhub3: 6 ports with 6 removable, self powered
uhub2: 6 ports with 6 removable, self powered
Root mount waiting for: CAM usbus0 usbus1
ugen0.3: <Winbond Electronics Corp Hermon USB hidmouse Device> at usbus0
ums0 on uhub2
ums0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
ugen1.3: <Corsair Commander PRO> at usbus1
ukbd0 on uhub2
ukbd0: <Winbond Electronics Corp Hermon USB hidmouse Device, class 0/0, rev 1.10/0.01, addr 3> on usbus0
kbd2 at ukbd0
uhid0 on uhub3
uhid0: <Corsair Commander PRO, class 0/0, rev 2.00/0.07, addr 3> on usbus1
ugen1.4: <vendor 0x05e3 USB2.0 Hub> at usbus1
uhub4 on uhub3
uhub4: <vendor 0x05e3 USB2.0 Hub, class 9/0, rev 2.00/32.98, addr 4> on usbus1
uhub4: MTT enabled
Root mount waiting for: CAM usbus1
uhub4: 4 ports with 4 removable, self powered
Root mount waiting for: CAM
Root mount waiting for: CAM
ses0 at ahciem0 bus 0 scbus7 target 0 lun 0
ses0: <AHCI SGPIO Enclosure 2.00 0001> SEMB S-E-S 2.00 device
ses0: SEMB SES Device
ada0 at ahcich0 bus 0 scbus1 target 0 lun 0
ada0: <SAMSUNG MZ7LM240HCGR-00005 GXT1003Q> ACS-2 ATA SATA 3.x device
ada0: Serial Number S2E5NXAGA01048
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 228936MB (468862128 512 byte sectors)
ada0: quirks=0x3<4K,NCQ_TRIM_BROKEN>
ses0: ada0 in 'Slot 00', SATA Slot: scbus1 target 0
ada1 at ahcich1 bus 0 scbus2 target 0 lun 0
ada1: <SAMSUNG MZ7LM240HCGR-00005 GXT1003Q> ACS-2 ATA SATA 3.x device
ada1: Serial Number S2E5NXAGA01052
ada1: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 228936MB (468862128 512 byte sectors)
ada1: quirks=0x3<4K,NCQ_TRIM_BROKEN>
ses0: ada1 in 'Slot 01', SATA Slot: scbus2 target 0
ada2 at ahcich5 bus 0 scbus6 target 0 lun 0
ada2: <SuperMicro SSD SOB20R> ACS-2 ATA SATA 3.x device
ada2: Serial Number SMC0515D94519B854045
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 512bytes)
ada2: Command Queueing enabled
ada2: 30208MB (61865984 512 byte sectors)
ses0: ada2 in 'Slot 05', SATA Slot: scbus6 target 0
da0 at mps0 bus 0 scbus0 target 4 lun 0
da0: <ATA WDC WD80EFAX-68K 0A81> Fixed Direct Access SPC-4 SCSI device
da0: Serial Number VDGWGP1D
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 7630885MB (15628053168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 5 lun 0
da1: <ATA WDC WD80EFAX-68K 0A81> Fixed Direct Access SPC-4 SCSI device
da1: Serial Number VDGV0SPD
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 7630885MB (15628053168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 7 lun 0
da3: <ATA WDC WD80EFAX-68K 0A81> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number VAHTEG6L
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 7630885MB (15628053168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 6 lun 0
da2: <ATA WDC WD80EFAX-68K 0A81> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number VAJ08P2L
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 7630885MB (15628053168 512 byte sectors)
lo0: link state changed to UP
em1: link state changed to UP
GEOM_MIRROR: Device mirror/swap0 launched (2/2).
GEOM_ELI: Device mirror/swap0.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: software
GEOM_MIRROR: Device mirror/swap1 launched (2/2).
GEOM_MIRROR: Device mirror/swap2 launched (2/2).
GEOM_ELI: Device mirror/swap1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: software
GEOM_ELI: Device mirror/swap2.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:     Crypto: software
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> UCP/8/48/0x3f8<EDG,THR,REA,WRI,INV,QUA,PRC> UCF/1/48/0x60<REA,WRI>
em1: link state changed to DOWN
em1: link state changed to UP
CPU: Intel(R) Xeon(R) CPU E31260L @ 2.40GHz (2400.07-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206a7  Family=0x6  Model=0x2a  Stepping=7
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x1dbae3ff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,POPCNT,TSCDLT,XSAVE,OSXSAVE,AVX>
  AMD Features=0x28100800<SYSCALL,NX,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  Structured Extended Features3=0x9c000400<MD_CLEAR,IBPB,STIBP,L1DFL,SSBD>
  XSAVE Features=0x1<XSAVEOPT>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
Security policy loaded: MAC/ntpd (mac_ntpd)
bridge0: Ethernet address: 02:dc:dd:47:50:00
epair0a: Ethernet address: 02:83:d2:34:f6:0a
epair0b: Ethernet address: 02:83:d2:34:f6:0b
epair0a: link state changed to UP
epair0b: link state changed to UP
epair0a: changing name to 'vnet0.1'
em1: link state changed to DOWN
bridge0: link state changed to UP
em1: promiscuous mode enabled
vnet0.1: promiscuous mode enabled
lo0: link state changed to UP
em1: link state changed to UP
uhid0: at uhub3, port 3, addr 3 (disconnected)
uhid0: detached
epair1a: Ethernet address: 02:b3:57:de:83:0a
epair1b: Ethernet address: 02:b3:57:de:83:0b
epair1a: link state changed to UP
epair1b: link state changed to UP
epair1a: changing name to 'vnet0.2'
epair1b: changing name to 'epair0b'
vnet0.2: promiscuous mode enabled
lo0: link state changed to UP
tun0: link state changed to UP
tun0: changing name to 'wg0'
epair2a: Ethernet address: 02:62:fb:8d:2d:0a
epair2b: Ethernet address: 02:62:fb:8d:2d:0b
epair2a: link state changed to UP
epair2b: link state changed to UP
epair2a: changing name to 'vnet0.3'
epair2b: changing name to 'epair0b'
vnet0.3: promiscuous mode enabled
lo0: link state changed to UP
epair3a: Ethernet address: 02:d3:e8:31:99:0a
epair3b: Ethernet address: 02:d3:e8:31:99:0b
epair3a: link state changed to UP
epair3b: link state changed to UP
epair3a: changing name to 'vnet0.4'
epair3b: changing name to 'epair0b'
vnet0.4: promiscuous mode enabled
lo0: link state changed to UP
epair4a: Ethernet address: 02:7c:c9:13:3b:0a
epair4b: Ethernet address: 02:7c:c9:13:3b:0b
epair4a: link state changed to UP
epair4b: link state changed to UP
epair4a: changing name to 'vnet0.5'
epair4b: changing name to 'epair0b'
vnet0.5: promiscuous mode enabled
lo0: link state changed to UP
epair5a: Ethernet address: 02:a3:2f:1b:18:0a
epair5b: Ethernet address: 02:a3:2f:1b:18:0b
epair5a: link state changed to UP
epair5b: link state changed to UP
epair5a: changing name to 'vnet0.6'
epair5b: changing name to 'epair0b'
vnet0.6: promiscuous mode enabled
lo0: link state changed to UP
epair6a: Ethernet address: 02:dc:03:ea:1c:0a
epair6b: Ethernet address: 02:dc:03:ea:1c:0b
epair6a: link state changed to UP
epair6b: link state changed to UP
epair6a: changing name to 'vnet0.7'
epair6b: changing name to 'epair0b'
vnet0.7: promiscuous mode enabled
lo0: link state changed to UP
epair7a: Ethernet address: 02:51:ca:d3:23:0a
epair7b: Ethernet address: 02:51:ca:d3:23:0b
epair7a: link state changed to UP
epair7b: link state changed to UP
epair7a: changing name to 'vnet0.8'
epair7b: changing name to 'epair0b'
vnet0.8: promiscuous mode enabled
lo0: link state changed to UP
epair8a: Ethernet address: 02:c0:84:41:19:0a
epair8b: Ethernet address: 02:c0:84:41:19:0b
epair8a: link state changed to UP
epair8b: link state changed to UP
epair8a: changing name to 'vnet0.9'
epair8b: changing name to 'epair0b'
vnet0.9: promiscuous mode enabled
lo0: link state changed to UP
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b

 

[Patrick M. Hausen]

So it's an Intel interface - which is good. Did you disable hardware offloading? If not, please do so and reboot.

 

[Mannekino]

Alright, I checked the box for "Disable hardware offloading" under the settings of the interface em1 and it's currently rebooting. Here are the specs of my server. So after a reboot everything will probably work again and I'll have to wait for a couple of days before it happens again (hopefully not). Unless there's some more checking we can do of log files.

• Motherboard: Supermicro X9SCM-F
• Chassis: Fractal Design Node 804
• CPU: Intel Xeon E3-1260L
• CPU cooler: Noctua NH-L9i (this has a 4-pin PWM fan)
• Memory: 32GB ECC (4x 8GB Kingstong ValueRAM KVR1333D3E9SK2/16G)
• HBA: Dell PERC H200 (flashed in IT mode)
• Storage SDD: 2x Samsung PM863 240GB connected through the two onboard SATA3 (6Gb/s) ports
• Storage HDD: 4x Western Digital Red 8TB (connected through the HBA)
• Storage extra: Supermicro SATA DOM for the OS
• Corsair Commander Pro for addressing the PWM fans

Edit:
Jails have connectivity again, now have to wait again and see what happens.

 

[Mannekino]

Unfortunately it happened again. Took about 4 days. Currently the TrueNAS system is running the scrub on my primary pool so I can't reboot the server. When I run dmesg I see these messages again at the bottom. This must be some kind of clue to help investigate further.

Code:

arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b
arp: 192.168.178.3 moved from 02:ff:60:3e:10:b8 to 00:25:90:74:57:e0 on epair0b
arp: 192.168.178.3 moved from 00:25:90:74:57:e0 to 02:ff:60:3e:10:b8 on epair0b

Network connectivity dropped around 06:30 (CEST) this morning.

I only see these two MAC addresses in dmesg output:

• 00:25:90:74:57:e0 (this is the MAC address of the em1 physical interface
• 02:ff:60:3e:10:b8 (this is ether MAC address of the grafana jail.

I think I will try and reboot the server again and start all the Jails except for the grafana jail and see what happens.

Is there any other way to restore network connectivity to the Jails without having to reboot the system?

Other suggestions welcome.

 

[DATAstrm]

Did you find a solution to this?

 

[Mannekino]

Not exactly. I managed to figure out by proces of elimination that the Grafana Jail was causing the problem so I just turned that one off for like 2 months. After that period I did some regular maintenance and updated all the Jails (both a Jail version update and software inside the Jail). After that update I turned the Grafana Jail back on again and since then it has been fine.

So still not sure what the problem was exactly but it has been running fine for a while now.

 

[DATAstrm]

Thanks for the update!

I'm running into a problem now where all my jails will simultaneously lose connectivity. This all happens silently. The frustrating part is that it's hard to diagnose because it can take up to 22 days before it happens-and there's nothing in the logs to indicate a problem.

 

[clusty]

Not exactly. I managed to figure out by proces of elimination that the Grafana Jail was causing the problem so I just turned that one off for like 2 months. After that period I did some regular maintenance and updated all the Jails (both a Jail version update and software inside the Jail). After that update I turned the Grafana Jail back on again and since then it has been fine.

So still not sure what the problem was exactly but it has been running fine for a while now.

This is so frustrating...
I am pulling my hair and I think I might have a similar problem :(