I am hoping that someone else has experienced (and been able to resolve) this issue, or at least can help me figure out where to look to pinpoint the problem. I have found similar threads on this forum and on reddit, but I didn't find anything that resolved my problem (and honestly I couldn't understand much of what they were saying).
About a week and a half ago, all of my jails lost networking. By that I mean they quit working, presumably because they couldn't talk to the outside anymore. From a shell inside the jail, I cannot ping the default router (192.168.1.1) like I used to.
I have a pretty simple set up that was chugging along without issue for quite some time, so this issue has surprised me. I only have 4 jails (one is created by a plugin):
1. The MineOS plugin/jail
2. A jail for Plex
3. A jail that runs a mysql server
4. A jail that runs a tomcat server (that communicates with the mysql server)
I do not use DHCP for any of my jails. I have a static IP for all 4. I then have set my router (Netgear Nighthawk R6900) to only assign IPs starting above the range that my static IPs are in. I also have each jail IP as a reserved IP in the router, and I can (when things are working) see each of these IPs as "attached devices" inside the router gui.
The first time things quit working, I was able to resolve it by restarting the TrueNAS server. Everything was fine for a few hours, then it happened again. This time a TrueNAS server reboot did not resolve the problem. Instead, I shut down the server, my modem, and router. Then started each one up in order. This resolved the problem for about a day and a half until last night when it happened again. Currently things are down.
From the threads I already read, there is a lot of talk about vnet, nics, and bridges, which honestly is above my paygrade. The fact that it was working fine, but now it is not, makes me think that something has changed. Ok, so what changed? Yes, I did do some things before the jails went haywire. This is what I can remember doing:
1. I upgraded TrueNAS from TrueNAS-12.0-U2 to TrueNAS-12.0-U4
2. I needed to upgrade the java version inside the mineos jail so that I could use the latest minecraft server jar. I had a heck of a time doing this upgrade and ended up just blowing away the plugin and reinstalling it, then upgrading java. This appeared to be working
3. I attempted to install the gitlab plugin a few times but it never would complete
4. I did not change any kind of network settings anywhere
Part of me wants to think it has something to do with the MineOS upgrade. It seems like when the issue happens, it's while my two kids are playing Minecraft. But I haven't made that conclusive connection, and it's hard to believe that a problem with one plugin can mess up all the others.
With that said, because all the jails are messed up, it makes me think it's a higher level server or networking issue. I just don't understand how it can work for a while and then stop working.
Here is the ifconfig from the host:
Here's the TrueNAS Network Summary screen:
Here's the Network Global Config screen:
Anyone have any thoughts or suggestions?
About a week and a half ago, all of my jails lost networking. By that I mean they quit working, presumably because they couldn't talk to the outside anymore. From a shell inside the jail, I cannot ping the default router (192.168.1.1) like I used to.
I have a pretty simple set up that was chugging along without issue for quite some time, so this issue has surprised me. I only have 4 jails (one is created by a plugin):
1. The MineOS plugin/jail
2. A jail for Plex
3. A jail that runs a mysql server
4. A jail that runs a tomcat server (that communicates with the mysql server)
I do not use DHCP for any of my jails. I have a static IP for all 4. I then have set my router (Netgear Nighthawk R6900) to only assign IPs starting above the range that my static IPs are in. I also have each jail IP as a reserved IP in the router, and I can (when things are working) see each of these IPs as "attached devices" inside the router gui.
The first time things quit working, I was able to resolve it by restarting the TrueNAS server. Everything was fine for a few hours, then it happened again. This time a TrueNAS server reboot did not resolve the problem. Instead, I shut down the server, my modem, and router. Then started each one up in order. This resolved the problem for about a day and a half until last night when it happened again. Currently things are down.
From the threads I already read, there is a lot of talk about vnet, nics, and bridges, which honestly is above my paygrade. The fact that it was working fine, but now it is not, makes me think that something has changed. Ok, so what changed? Yes, I did do some things before the jails went haywire. This is what I can remember doing:
1. I upgraded TrueNAS from TrueNAS-12.0-U2 to TrueNAS-12.0-U4
2. I needed to upgrade the java version inside the mineos jail so that I could use the latest minecraft server jar. I had a heck of a time doing this upgrade and ended up just blowing away the plugin and reinstalling it, then upgrading java. This appeared to be working
3. I attempted to install the gitlab plugin a few times but it never would complete
4. I did not change any kind of network settings anywhere
Part of me wants to think it has something to do with the MineOS upgrade. It seems like when the issue happens, it's while my two kids are playing Minecraft. But I haven't made that conclusive connection, and it's hard to believe that a problem with one plugin can mess up all the others.
With that said, because all the jails are messed up, it makes me think it's a higher level server or networking issue. I just don't understand how it can work for a while and then stop working.
Here is the ifconfig from the host:
Code:
em0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 150 0 options=812099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,VLAN _HWFILTER> ether 6c:2b:59:d9:89:1d inet 192.168.1.26 netmask 0xffffff00 broadcast 192.168.1.255 media: Ethernet autoselect (1000baseT <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384 options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6> inet6 ::1 prefixlen 128 inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2 inet 127.0.0.1 netmask 0xff000000 groups: lo nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL> pflog0: flags=0<> metric 0 mtu 33160 groups: pflog bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 ether 02:83:6e:a1:24:00 id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15 maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200 root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0 member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 8 priority 128 path cost 2000 member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 7 priority 128 path cost 2000 member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 6 priority 128 path cost 2000 member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 5 priority 128 path cost 2000 member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP> ifmaxaddr 0 port 1 priority 128 path cost 20000 groups: bridge nd6 options=1<PERFORMNUD> vnet0.1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: mineos as nic: epair0b options=8<VLAN_MTU> ether 62:2b:59:c5:1d:5f hwaddr 02:9d:8b:66:16:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=1<PERFORMNUD> vnet0.2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: mysql as nic: epair0b options=8<VLAN_MTU> ether 6e:2b:59:9d:e6:cc hwaddr 02:15:c7:05:56:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=1<PERFORMNUD> vnet0.3: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: pms as nic: epair0b options=8<VLAN_MTU> ether 6e:2b:59:57:7e:c0 hwaddr 02:98:39:79:54:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=1<PERFORMNUD> vnet0.4: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: associated with jail: tomcat as nic: epair0b options=8<VLAN_MTU> ether 6e:2b:59:32:e3:88 hwaddr 02:e5:89:0e:15:0a groups: epair media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>) status: active nd6 options=1<PERFORMNUD>
Here's the TrueNAS Network Summary screen:
Here's the Network Global Config screen:
Anyone have any thoughts or suggestions?