Jail losing network connectivity

[boggie1688]

Hi,

I have a jail that contains 3 plugins, the jail is also using OpenVPN. After 2 days, the plugins lose connectivity to the outside internet. This only recently started happening in the last month.

If I restart the jail, everything is fine.

Allow_Tunnel is checked, in the Network Properties of the Jail.

Any advice on what to look for here?

 

[dlavigne]

FreeNAS version?
Anything in /var/log/messages around the time connectivity is lost?

 

[boggie1688]

A little newb here, so bear with me.
I ran this command:
ls /var/log/messages

The output was just:
/var/log/messages

I believe this means there is nothing in that folder, so no logs.

I should note, when this happens. I log into the jail, and try to ping google.com. No response. Also I am on Freenas 11.2 latest stable train.

I caught this issue last night around 7PM, so my guess is that in roughly 48 hours it will pop up again. For now the jail and its plugins are running just fine. Weird right?

 

[dlavigne]

Try this one: more /var/log/messages (it should open the file, use your page down key or return bar to scroll).

 

[boggie1688]

Try this one: more /var/log/messages (it should open the file, use your page down key or return bar to scroll).

I dumped the messages into a text file and then cut out the repeating errors. Looks like the loss of connectivity happened around 10AM yesterday, and was not resolved until I restarted the plugin that evening. It just looks like openvpn complaining that it has lost connection. Nothing else.

Or maybe I'm interpreting this incorrectly.

EDIT: I stand corrected. I forgot I was installing a new router around that time. This might be the root cause of these errors, but the jail has been suffering from this issue for over a month.

I've also dumped the Freenas root messages below.

My guess is we need to wait for it to happen again and then quickly inspect the messages.

Thanks for being so eager to help.

Jail Messages:

Code:

Dec  4 10:00:00 sonarr newsyslog[39981]: logfile turned over due to size>100K
Dec  4 10:01:05 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:01:05 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:01:05 sonarr openvpn[6479]: Could not determine IPv4/IPv6 protocol
Dec  4 10:01:05 sonarr openvpn[6479]: SIGUSR1[soft,init_instance] received, process restarting
Dec  4 10:03:00 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:03:00 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:03:00 sonarr openvpn[6479]: Could not determine IPv4/IPv6 protocol
Dec  4 10:03:00 sonarr openvpn[6479]: SIGUSR1[soft,init_instance] received, process restarting
Dec  4 10:06:15 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:06:15 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:06:15 sonarr openvpn[6479]: Could not determine IPv4/IPv6 protocol
Dec  4 10:06:15 sonarr openvpn[6479]: SIGUSR1[soft,init_instance] received, process restarting
Dec  4 10:11:50 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:11:50 sonarr openvpn[6479]: RESOLVE: Cannot resolve host address: us-california.privateinternetaccess.com:1198 (hostname nor servname provided, or not known)
Dec  4 10:11:50 sonarr openvpn[6479]: Could not determine IPv4/IPv6 protocol
Dec  4 10:11:50 sonarr openvpn[6479]: SIGUSR1[soft,init_instance] received, process restarting
Dec  4 10:16:50 sonarr openvpn[6479]: TCP/UDP: Preserving recently used remote address: [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  4 10:16:50 sonarr openvpn[6479]: UDP link local: (not bound)
Dec  4 10:16:50 sonarr openvpn[6479]: UDP link remote: [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  4 10:17:50 sonarr openvpn[6479]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Dec  4 10:17:50 sonarr openvpn[6479]: TLS Error: TLS handshake failed
...
Dec  4 19:23:01 sonarr openvpn[6479]: UDP link local: (not bound)
Dec  4 19:23:01 sonarr openvpn[6479]: UDP link remote: [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  4 19:23:01 sonarr openvpn[6479]: write UDP: No buffer space available (code=55)
Dec  4 19:24:01 sonarr openvpn[6479]: TLS Error: TLS key negotiation failed to occur within 60 seconds (check your network connectivity)
Dec  4 19:24:01 sonarr openvpn[6479]: TLS Error: TLS handshake failed
Dec  4 19:24:01 sonarr openvpn[6479]: SIGUSR1[soft,tls-error] received, process restarting
Dec  4 19:28:12 sonarr openvpn[6479]: /sbin/ifconfig tun0 destroy
Dec  4 19:28:12 sonarr openvpn[6479]: SIGTERM[hard,init_instance] received, process exiting
Dec  4 19:28:12 sonarr dhclient[6186]: connection closed
Dec  4 19:28:12 sonarr dhclient[6186]: exiting.
Dec  4 19:28:13 sonarr syslogd: exiting on signal 15
Dec  4 19:28:18 sonarr syslogd: kernel boot file is /boot/kernel/kernel
Dec  4 19:28:26 sonarr openvpn[60770]: WARNING: file '/usr/local/etc/openvpn/pass.txt' is group or others accessible
Dec  4 19:28:26 sonarr openvpn[60770]: OpenVPN 2.4.8 amd64-portbld-freebsd11.3 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Nov 15 2019
Dec  4 19:28:26 sonarr openvpn[60770]: library versions: OpenSSL 1.0.2o-freebsd  27 Mar 2018, LZO 2.10
Dec  4 19:28:26 sonarr openvpn[60771]: TCP/UDP: Preserving recently used remote address: [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  4 19:28:26 sonarr openvpn[60771]: UDP link local: (not bound)
Dec  4 19:28:26 sonarr openvpn[60771]: UDP link remote: [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  4 19:28:26 sonarr openvpn[60771]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Dec  4 19:28:29 sonarr openvpn[60771]: [ab5c2054fc36a48cdeb33fa793a42dac] Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  4 19:28:30 sonarr openvpn[60771]: TUN/TAP device /dev/tun0 opened
Dec  4 19:28:30 sonarr openvpn[60771]: /sbin/ifconfig tun0 10.89.10.6 10.89.10.5 mtu 1500 netmask 255.255.255.255 up
Dec  4 19:28:30 sonarr openvpn[60771]: Initialization Sequence Completed
Dec  5 08:29:32 sonarr openvpn[60771]: event_wait : Interrupted system call (code=4)
Dec  5 08:29:32 sonarr openvpn[60771]: /sbin/ifconfig tun0 destroy
Dec  5 08:29:32 sonarr openvpn[60771]: SIGTERM[hard,] received, process exiting
Dec  5 08:29:33 sonarr dhclient[60478]: connection closed
Dec  5 08:29:33 sonarr dhclient[60478]: exiting.
Dec  5 08:29:33 sonarr syslogd: exiting on signal 15
Dec  5 08:31:06 sonarr syslogd: kernel boot file is /boot/kernel/kernel
Dec  5 08:31:12 sonarr openvpn[14971]: WARNING: file '/usr/local/etc/openvpn/pass.txt' is group or others accessible
Dec  5 08:31:12 sonarr openvpn[14971]: OpenVPN 2.4.8 amd64-portbld-freebsd11.3 [SSL (OpenSSL)] [LZO] [LZ4] [MH/RECVDA] [AEAD] built on Nov 15 2019
Dec  5 08:31:12 sonarr openvpn[14971]: library versions: OpenSSL 1.0.2o-freebsd  27 Mar 2018, LZO 2.10
Dec  5 08:31:12 sonarr openvpn[14973]: TCP/UDP: Preserving recently used remote address: [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  5 08:31:12 sonarr openvpn[14973]: UDP link local: (not bound)
Dec  5 08:31:12 sonarr openvpn[14973]: UDP link remote: [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  5 08:31:12 sonarr openvpn[14973]: WARNING: this configuration may cache passwords in memory -- use the auth-nocache option to prevent this
Dec  5 08:31:12 sonarr openvpn[14973]: [38d2411b9565c879bd26b761f4781cbd] Peer Connection Initiated with [AF_INET]XXX.XXX.XXX.XX:XXX
Dec  5 08:31:13 sonarr openvpn[14973]: TUN/TAP device /dev/tun0 opened
Dec  5 08:31:13 sonarr openvpn[14973]: /sbin/ifconfig tun0 10.0.10.6 10.0.10.5 mtu 1500 netmask 255.255.255.255 up
Dec  5 08:31:13 sonarr openvpn[14973]: Initialization Sequence Completed

Freenas Root Messages:

Code:

Dec  4 00:00:00 freenas syslog-ng[2435]: Configuration reload request received, reloading configuration;
Dec  4 00:00:00 freenas syslog-ng[2435]: Configuration reload finished;
Dec  4 08:18:54 freenas ugen6.2: <SEJIN SEJIN USB joint Keyboard> at usbus6 (disconnected)
Dec  4 08:18:54 freenas ukbd0: at uhub4, port 2, addr 2 (disconnected)
Dec  4 08:18:54 freenas ukbd0: detached
Dec  4 08:31:26 freenas ugen6.2: <SEJIN SEJIN USB joint Keyboard> at usbus6
Dec  4 08:31:26 freenas ukbd0 on uhub4
Dec  4 08:31:26 freenas ukbd0: <Joint Keyboard> on usbus6
Dec  4 08:31:26 freenas kbd2 at ukbd0
Dec  4 09:40:39 freenas kernel: re1: link state changed to DOWN
Dec  4 09:40:39 freenas kernel: re1: link state changed to DOWN
Dec  4 10:12:02 freenas kernel: re1: link state changed to UP
Dec  4 10:12:02 freenas kernel: re1: link state changed to UP
Dec  4 10:13:17 freenas dhclient: New IP Address (re1): 192.168.XXX.XXX
Dec  4 10:13:17 freenas dhclient: New Subnet Mask (re1): 255.255.XXX.0
Dec  4 10:13:17 freenas dhclient: New Broadcast Address (re1): 192.168.XXX.XXX
Dec  4 10:13:17 freenas dhclient: New Routers (re1): 192.168.XXX.XXX
Dec  4 11:53:21 freenas kernel: Limiting icmp unreach response from 422 to 200 packets/sec
Dec  4 11:53:21 freenas kernel: Limiting icmp unreach response from 422 to 200 packets/sec
Dec  4 11:53:22 freenas kernel: Limiting icmp unreach response from 401 to 200 packets/sec
Dec  4 11:53:22 freenas kernel: Limiting icmp unreach response from 401 to 200 packets/sec
Dec  4 11:53:23 freenas kernel: Limiting icmp unreach response from 379 to 200 packets/sec
Dec  4 11:53:23 freenas kernel: Limiting icmp unreach response from 379 to 200 packets/sec
Dec  4 11:53:24 freenas kernel: Limiting icmp unreach response from 458 to 200 packets/sec
Dec  4 11:53:24 freenas kernel: Limiting icmp unreach response from 458 to 200 packets/sec
Dec  4 11:53:25 freenas kernel: Limiting icmp unreach response from 317 to 200 packets/sec
Dec  4 11:53:25 freenas kernel: Limiting icmp unreach response from 317 to 200 packets/sec
Dec  4 11:53:26 freenas kernel: Limiting icmp unreach response from 678 to 200 packets/sec
Dec  4 11:53:26 freenas kernel: Limiting icmp unreach response from 678 to 200 packets/sec
Dec  4 11:53:27 freenas kernel: Limiting icmp unreach response from 724 to 200 packets/sec
Dec  4 11:53:27 freenas kernel: Limiting icmp unreach response from 724 to 200 packets/sec
Dec  4 11:53:28 freenas kernel: Limiting icmp unreach response from 824 to 200 packets/sec
Dec  4 11:53:28 freenas kernel: Limiting icmp unreach response from 824 to 200 packets/sec
Dec  4 11:53:29 freenas kernel: Limiting icmp unreach response from 778 to 200 packets/sec
Dec  4 11:53:29 freenas kernel: Limiting icmp unreach response from 778 to 200 packets/sec
Dec  4 11:53:30 freenas kernel: Limiting icmp unreach response from 831 to 200 packets/sec
Dec  4 11:53:30 freenas kernel: Limiting icmp unreach response from 831 to 200 packets/sec
Dec  4 11:53:31 freenas kernel: Limiting icmp unreach response from 699 to 200 packets/sec
Dec  4 11:53:31 freenas kernel: Limiting icmp unreach response from 699 to 200 packets/sec
Dec  4 11:53:32 freenas kernel: Limiting icmp unreach response from 852 to 200 packets/sec
Dec  4 11:53:32 freenas kernel: Limiting icmp unreach response from 852 to 200 packets/sec
Dec  4 11:53:33 freenas kernel: Limiting icmp unreach response from 682 to 200 packets/sec
Dec  4 11:53:33 freenas kernel: Limiting icmp unreach response from 682 to 200 packets/sec
Dec  4 11:53:34 freenas kernel: Limiting icmp unreach response from 784 to 200 packets/sec
Dec  4 11:53:34 freenas kernel: Limiting icmp unreach response from 784 to 200 packets/sec
Dec  4 11:53:35 freenas kernel: Limiting icmp unreach response from 764 to 200 packets/sec
Dec  4 11:53:35 freenas kernel: Limiting icmp unreach response from 764 to 200 packets/sec
Dec  4 11:53:36 freenas kernel: Limiting icmp unreach response from 745 to 200 packets/sec
Dec  4 11:53:36 freenas kernel: Limiting icmp unreach response from 745 to 200 packets/sec
Dec  4 11:53:37 freenas kernel: Limiting icmp unreach response from 720 to 200 packets/sec
Dec  4 11:53:37 freenas kernel: Limiting icmp unreach response from 720 to 200 packets/sec
Dec  4 11:53:38 freenas kernel: Limiting icmp unreach response from 844 to 200 packets/sec
Dec  4 11:53:38 freenas kernel: Limiting icmp unreach response from 844 to 200 packets/sec
Dec  4 11:53:39 freenas kernel: Limiting icmp unreach response from 752 to 200 packets/sec
Dec  4 11:53:39 freenas kernel: Limiting icmp unreach response from 752 to 200 packets/sec
Dec  4 11:53:40 freenas kernel: Limiting icmp unreach response from 740 to 200 packets/sec
Dec  4 11:53:40 freenas kernel: Limiting icmp unreach response from 740 to 200 packets/sec
Dec  4 11:53:41 freenas kernel: Limiting icmp unreach response from 783 to 200 packets/sec
Dec  4 11:53:41 freenas kernel: Limiting icmp unreach response from 783 to 200 packets/sec
Dec  4 11:53:42 freenas kernel: Limiting icmp unreach response from 757 to 200 packets/sec
Dec  4 11:53:42 freenas kernel: Limiting icmp unreach response from 757 to 200 packets/sec
Dec  4 11:53:43 freenas kernel: Limiting icmp unreach response from 740 to 200 packets/sec
Dec  4 11:53:43 freenas kernel: Limiting icmp unreach response from 740 to 200 packets/sec
Dec  4 11:53:44 freenas kernel: Limiting icmp unreach response from 733 to 200 packets/sec
Dec  4 11:53:44 freenas kernel: Limiting icmp unreach response from 733 to 200 packets/sec
Dec  4 11:53:45 freenas kernel: Limiting icmp unreach response from 835 to 200 packets/sec
Dec  4 11:53:45 freenas kernel: Limiting icmp unreach response from 835 to 200 packets/sec
Dec  4 11:53:46 freenas kernel: Limiting icmp unreach response from 791 to 200 packets/sec
Dec  4 11:53:46 freenas kernel: Limiting icmp unreach response from 791 to 200 packets/sec
Dec  4 11:53:47 freenas kernel: Limiting icmp unreach response from 687 to 200 packets/sec
Dec  4 11:53:47 freenas kernel: Limiting icmp unreach response from 687 to 200 packets/sec
Dec  4 11:53:48 freenas kernel: Limiting icmp unreach response from 841 to 200 packets/sec
Dec  4 11:53:48 freenas kernel: Limiting icmp unreach response from 841 to 200 packets/sec
Dec  4 11:53:49 freenas kernel: Limiting icmp unreach response from 696 to 200 packets/sec
Dec  4 11:53:49 freenas kernel: Limiting icmp unreach response from 696 to 200 packets/sec
Dec  4 11:53:50 freenas kernel: Limiting icmp unreach response from 770 to 200 packets/sec
Dec  4 11:53:50 freenas kernel: Limiting icmp unreach response from 770 to 200 packets/sec
Dec  4 11:53:51 freenas kernel: Limiting icmp unreach response from 779 to 200 packets/sec
Dec  4 11:53:51 freenas kernel: Limiting icmp unreach response from 779 to 200 packets/sec
Dec  4 11:53:52 freenas kernel: Limiting icmp unreach response from 541 to 200 packets/sec
Dec  4 11:53:52 freenas kernel: Limiting icmp unreach response from 541 to 200 packets/sec
Dec  4 11:53:53 freenas kernel: Limiting icmp unreach response from 783 to 200 packets/sec
Dec  4 11:53:53 freenas kernel: Limiting icmp unreach response from 783 to 200 packets/sec
Dec  4 11:53:54 freenas kernel: Limiting icmp unreach response from 753 to 200 packets/sec
Dec  4 11:53:54 freenas kernel: Limiting icmp unreach response from 753 to 200 packets/sec
Dec  4 11:53:55 freenas kernel: Limiting icmp unreach response from 857 to 200 packets/sec
Dec  4 11:53:55 freenas kernel: Limiting icmp unreach response from 857 to 200 packets/sec
Dec  4 11:53:56 freenas kernel: Limiting icmp unreach response from 873 to 200 packets/sec
Dec  4 11:53:56 freenas kernel: Limiting icmp unreach response from 873 to 200 packets/sec
Dec  4 11:53:57 freenas kernel: Limiting icmp unreach response from 766 to 200 packets/sec
Dec  4 11:53:57 freenas kernel: Limiting icmp unreach response from 766 to 200 packets/sec
Dec  4 11:53:58 freenas kernel: Limiting icmp unreach response from 979 to 200 packets/sec
Dec  4 11:53:58 freenas kernel: Limiting icmp unreach response from 979 to 200 packets/sec
Dec  4 11:53:59 freenas kernel: Limiting icmp unreach response from 585 to 200 packets/sec
Dec  4 11:53:59 freenas kernel: Limiting icmp unreach response from 585 to 200 packets/sec

 

[dlavigne]

From the root messages, it looks like the Realtek NIC is being overwhelmed by packet scans from the Internet. Recommendations would be to replace the Realtek with one of the Intel NICs recommended on the forums and to place a firewall (that is configured to allow your VPN connection) in front of the NAS system.

 

[boggie1688]

From the root messages, it looks like the Realtek NIC is being overwhelmed by packet scans from the Internet. Recommendations would be to replace the Realtek with one of the Intel NICs recommended on the forums and to place a firewall (that is configured to allow your VPN connection) in front of the NAS system.

I'll look grabbing a intel nic here shortly.

The router has a firewall. I have a ATT modem/router in IP passthrough mode, connected to my router, then everything is connected to the router. I dumped the message log again, and the ICMP messages are gone as of yesterday, 11th. Some investigating is required.

I'm happy to report the jail connectivity issue is gone, but I think picking at this issue has clearly revealed other issues I need to fix in the system. This is what I get for setting up a freenas box years ago, having no clue what I was doing. Hell I still don't know what I'm doing. Problem now is I have a decent amount of data on it, which would make moving everything a PIA.

Thanks again for the help and suggestions!!