Jails nor Plugins can connect outside FreeNAS

imdos · Oct 1, 2013

I have FreeNAS 9.1.1 (x64) up and running with:
HP N36L with 8 Gb (to be replaced with N54L & 16 Gb by end of year)
6x 2 Tb disks in raidz1 ( to be replaced with 6x 3Tb this week) zfs, nfs shared
1 USB attached 2,5 inch WDC 320Gb with a ~120 Gb UFS mount and a ~120 Gb zfs mount (Jail)
2 NICs, the default bge0 and an additional pcie x1 as vge0.

I have an ADSL connection with a simple router (from the provider) which is connected to bge0.
On top of the simple router I have also an D-LINK DHP-1565 which provides the Wifi and Wifi-guest networks and some wired connections to a xbmc box, an additional Wifi AP and to the FreeNAS box, through vge0.

This is the output of netstat -rn from FreeNAS:

Code:

Destination        Gateway            Flags    Refs      Use  Netif Expire     
default            192.168.1.1        UGS        0    43878  bge0           
10.0.0.0/24        link#2            U          0      185  bge0           
10.0.0.1          link#2            UHS        0      463    lo0           
10.0.0.2          link#2            UHS        0    1151    lo0           
10.0.0.3          link#2            UHS        0      13    lo0           
127.0.0.1          link#11            UH          0    91489    lo0           
192.168.1.0/24    link#2            U          0      157  bge0           
192.168.1.11      link#2            UHS        0      447    lo0           
192.168.2.0/24    link#1            U          0      165  vge0           
192.168.2.30      link#1            UHS        0      450    lo0

I have been experiencing a lot with possible network options for my Jails, but can't get it to work to ping, pkg install, pkg_add -r or any other network command from a Plugin or Jail. This is a requisite however for SABnzbd/Couchpotato etc.

I can ping both gateways from FreeNAS, the internet, as well as the IP's of the Jails in zfs (jail).
I managed to get Jboss server running some handmade Scores through modyfing the default nginx configuration (adding my own config through an include statement in the nginx.conf and the /etc/rc.d/ix-nginx). This is available from the Internet through the reverse proxy.

From inside that Jboss Jail (and the other jails) I can't however make a connection outside the FreeNAS server. I can ping the IP's of the two interfaces as well as the other jails, but can't ping anything outside of FreeNAS, e.g. the gateway 192.168.1.1 or 192.168.2.1, do a pkg install, pkg_add -r or any other network command.

I have tried different options for a Jail by setting:
* a default IPv4 gateway
* enabling VIMAGE
* enabling NAT

But that doens't help at all.

I'm not a newbie at networking nor Linux/*BSD, but can't figure out how to properly configure my jails. The included plugins from appcafe for SAB, Couch etc, don't provide enough flexibility and control.

Whattteva · Oct 1, 2013

Post the output of netstat -rn from the jail you're trying to use.

Also, you may want to check the nameserver listed in /etc/resolv.conf (also in the jail).
Make sure it's pointing to your DNS server (probably your router or your ISP DNS servers).

Considering you can't ping anything outside of FreeNAS though, the problem most likely lies in the jail's routing table.

imdos · Oct 1, 2013

Netstat inside a jail (AFAIK doesn't/shouldn't differ).

Code:

Destination        Gateway            Flags    Refs      Use  Netif Expire     
default            192.168.1.1        UGS        0    67874  bge0           
10.0.0.0/24        link#2            U          0      209  bge0           
10.0.0.1          link#2            UHS        0      669    lo0           
10.0.0.2          link#2            UHS        0    1747    lo0           
10.0.0.3          link#2            UHS        0      13    lo0           
127.0.0.1          link#11            UH          0  105770    lo0           
192.168.1.0/24    link#2            U          0      165  bge0           
192.168.1.11      link#2            UHS        0      951    lo0           
192.168.2.0/24    link#1            U          0      189  vge0           
192.168.2.30      link#1            UHS        0      953    lo0           
                                                                               
Internet6:                                                                     
Destination                      Gateway                      Flags      Netif
Expire                                                                       
::/96                            ::1                          UGRS        lo0
::1                              link#11                      UH          lo0
::ffff:0.0.0.0/96                ::1                          UGRS        lo0
fe80::/10                        ::1                          UGRS        lo0
fe80::%lo0/64                    link#11                      U          lo0
fe80::1%lo0                      link#11                      UHS        lo0
ff01::%lo0/32                    ::1                          U          lo0
ff02::/16                        ::1                          UGRS        lo0
ff02::%lo0/32                    ::1                          U          lo0

resolv.conf from jail (but doesn't differ with the one from FreeNAS)

Code:

root@Fifa:/ # cat /etc/resolv.conf                                             
search lan                                                                     
nameserver 192.168.1.1                                                         
nameserver 192.168.2.1

Those nameservers are my ADSL gateway/router and my Wifi AP/router, which are able to route perfectly for FreeNAS itself.

One thing to note though, I have to use a routing restart from /etc/rc.local on FreeNAS; otherwise it won't listen on the open ports which are port forwarded from my ADSL gateway/router device. I geuss this has to do with my dual NIC configuration, since I had this issue on 8.3.1 as well.

Whattteva · Oct 1, 2013

That's strange cause the netstat from the jail for me is definitely quite different.

All my jails only have a loopback device and epair devices (virtual ethernet crossovers). The netstat in the jails never list the actual physical ethernet device (bge0 in your case).
Your jails must be set up differently. What options do you use for it?

imdos · Oct 1, 2013

I have tried all options to be honest :)

Vanilla/ non vanilla, VIMAGE, NAT, non seem to really help.
Pluginjail/portjail .

But I don't have any problem with trowing away my current attempts or changing the configuration. I can easily backup the changes I made to a file or so and start from scratch.

The only problem/issue is that a reboot isn't working atm due to my other post (http://forums.freenas.org/threads/9-1-1-x64-build-hangs-at-reboot.15257/), seems to have something to do with (additional attached) USB storage and causes an endless loop.

Whattteva · Oct 1, 2013

Try creating a new standard type jail with VIMAGE and vanilla checked, and NAT and 32-bit unchecked (this is the setup I personally use).
Then do netstat -rn in the jail and see if the output is different.

imdos · Oct 1, 2013

resolv.conf

Code:

Internet:                                                                     
Destination        Gateway            Flags    Refs      Use  Netif Expire     
10.0.0.0/24        link#2            U          0        0 epair0           
10.0.0.3          link#2            UHS        0        0    lo0           
127.0.0.1          link#1            UH          0        0    lo0           
                                                                               
Internet6:                                                                     
Destination                      Gateway                      Flags      Netif
Expire                                                                       
::1                              link#1                        UH          lo0
fe80::%lo0/64                    link#1                        U          lo0
fe80::1%lo0                      link#1                        UHS        lo0
fe80::%epair0b/64                link#2                        U      epair0b
fe80::99:cbff:fe00:e0b%epair0b    link#2                        UHS        lo0
ff01::%lo0/32                    ::1                          U          lo0
ff01::%epair0b/32                fe80::99:cbff:fe00:e0b%epair0b U      epair0b
ff02::%lo0/32                    ::1                          U          lo0
ff02::%epair0b/32                fe80::99:cbff:fe00:e0b%epair0b U      epair0b

ifconfig

Code:

ping www.xxx.com                                              
ping: cannot resolve www.xxx.com: Host name lookup failure                      
root@Download:/ # nslookup www.xxx.com                                          
;; connection timed out; no servers could be reached                            
                                                                                
root@Download:/ # ifconfig                                                      
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384               
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>                   
        inet6 ::1 prefixlen 128                                                 
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1                              
        inet 127.0.0.1 netmask 0xff000000                                       
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                               
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500   
        options=8<VLAN_MTU>                                                     
        ether 02:99:cb:00:0e:0b                                                 
        inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255                   
        inet6 fe80::99:cbff:fe00:e0b%epair0b prefixlen 64 scopeid 0x2           
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>                               
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)                     
        status: active

The resolv.conf remained the same.

Should I start adding a static route or is there some ipfw / other routing configuration I should set.

I'm puzzled

Whattteva · Oct 1, 2013

There it is. Your jail setup is now similar to mine, just an epair and an lo device.
The only thing missing is default gateway entry in your routing table.
I'm sure once you add that, nslookup should work.

Mine sets everything up automatically (including default gateway) without me having to do anything.
I guess it may have something to do with your dual NIC.

Here's what mine looks like, notice the default entry:

Code:

Routing tables
 
Internet:
Destination        Gateway            Flags    Refs      Use  Netif Expire
default            192.168.7.1        UGS        0  2237069 epair2
127.0.0.1          link#1            UH          0      82    lo0
192.168.7.0/24    link#2            U          0  361924 epair2
192.168.7.20      link#2            UHS        0        0    lo0

imdos · Oct 1, 2013

I geuss so. I'm trying to add your suggestion, but it doesn't seem to work.
I added this and restarted the Jail.

Code:

root@Download:/ # grep default /etc/rc.conf                                   
defaultrouter="192.168.1.1"

netstat -rn didn't show any changes.

Code:

root@Download:/ # route add default 192.168.1.1                               
route: writing to routing socket: Network is unreachable                       
add net default: gateway 192.168.1.1: Network is unreachable

This doesn't work either. How do I add a default route to my jail?

Code:

ping -c 1 192.168.2.1                                         
PING 192.168.2.1 (192.168.2.1): 56 data bytes                                   
ping: sendto: No route to host                                                  
root@Download:/ # ping -c 1 192.168.2.30                                        
PING 192.168.2.30 (192.168.2.30): 56 data bytes                                 
ping: sendto: No route to host

The same applies for 192.168.1.1 or my FreeNAS IP 192.168.1.11. I'm still able to ping 10.0.0.1 & 2, which are my other jails Fifa (Jboss, mysql & java, through nginx) and download, which currently holds my migrated SABnzbd/Couchpotato.

Whattteva · Oct 1, 2013

To add default route, use this:

Code:

route add 0.0.0.0 gateway_ip

I'm not sure if that'll be persistent or not... you may have to add it through the FreeNAS GUI to make it persistent. However, you can still do it this way for testing purposes.

imdos · Oct 1, 2013

This should be done inside the jail right? Or am I not authorized to do so?

Code:

[root@NAS ~]# jexec 6 csh                                                       
root@Download:/ # route add 0.0.0.0 192.168.1.1                                 
route: writing to routing socket: Network is unreachable                        
add net 0.0.0.0: gateway 192.168.1.1: Network is unreachable

Whattteva · Oct 1, 2013

Yeah, should be in the jail.
I think it's unreachable because the 192.168.1.0/24 network is not "attached" to your jail.
You would probably have to either configure 10.0.0.1 to route 192.168.1.0 traffic or create another bridge interface for the jail to connect to that network.

imdos · Oct 1, 2013

At the moment I can't add 10.0.0.1 since this is my first jail.

I just noticed however that I can ping only 10.0.0.1 and .2 and NOT .3.

Code:

root@Download:/ # ifconfig
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x1
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
epair0b: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=8<VLAN_MTU>
        ether 02:44:27:00:0e:0b
        inet 10.0.0.3 netmask 0xffffff00 broadcast 10.0.0.255
        inet6 fe80::44:27ff:fe00:e0b%epair0b prefixlen 64 scopeid 0x2
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
root@Download:/ # ping -c 1 10.0.0.3
PING 10.0.0.3 (10.0.0.3): 56 data bytes
1 packets transmitted, 0 packets received, 100.0% packet loss
 
PING 127.0.0.1 (127.0.0.1): 56 data bytes
64 bytes from 127.0.0.1: icmp_seq=0 ttl=64 time=0.166 ms

Isn't that strange!

Whattteva · Oct 1, 2013

Heh, that's strange indeed. That prompted me to try it on my jail and what do you know... the same thing happens.
The jail can't ping itself, but it can ping everything else.

The difference is, my jail can ping everything, including Internet servers.... just not itself

I'm guessing it's the side effect of the virtual ethernet card being implemented as a pair of cards with cross-0ver cable.

imdos · Oct 3, 2013

Whattteva said:
The difference is, my jail can ping everything, including Internet servers.... just not itself

I would be happy already if nslooukup, nntp and http(s) requests would work. I really don't care about ping. At the moment however it appears I'm unable to get a single packet to the outside (the Internet) from my jails.

If only I had a clue how to debug this properly. I don't mind to reinstall or make certain changes to warden or do whatever it takes, but this is driving me nuts

imdos · Oct 5, 2013

Here is the output of warden's list. Is there something more I could do / other information which could help

Code:

id: 6
host: Download
ipv4: 10.0.0.30/24
alias-ipv4:
bridge-ipv4:
alias-bridge-ipv4:
defaultrouter-ipv4:
ipv6:
alias-ipv6:
bridge-ipv6:
alias-bridge-ipv6:
defaultrouter-ipv6:
autostart: Disabled
vnet: Enabled
nat: Disabled
mac:
status: Stopped
type: standard
 
 
id: 4
host: Fifa
ipv4: 10.0.0.10/24
alias-ipv4:
bridge-ipv4:
alias-bridge-ipv4:
defaultrouter-ipv4:
ipv6:
alias-ipv6:
bridge-ipv6:
alias-bridge-ipv6:
defaultrouter-ipv6:
autostart: Enabled
vnet: Disabled
nat: Enabled
mac:
status: Running
type: portjail
 
 
id: 7
host: Usenet
ipv4: 10.0.0.31/24
alias-ipv4:
bridge-ipv4:
alias-bridge-ipv4:
defaultrouter-ipv4:
ipv6:
alias-ipv6:
bridge-ipv6:
alias-bridge-ipv6:
defaultrouter-ipv6:
autostart: Enabled
vnet: Disabled
nat: Enabled
mac:
status: Running
type: pluginjail
 
 
id: 5
host: download
ipv4: 10.0.0.20/24
alias-ipv4:
bridge-ipv4:
alias-bridge-ipv4:
defaultrouter-ipv4:
ipv6:
alias-ipv6:
bridge-ipv6:
alias-bridge-ipv6:
defaultrouter-ipv6:
autostart: Disabled
vnet: Disabled
nat: Enabled
mac:
status: Stopped
type: portjail

imdos · Oct 12, 2013

This is the debug info I receive when starting and stopping from the cli. This is from Couchpotato, one of the default plugins .

Code:

[root@NAS:~] # warden stop couchpotato_1
jexec 4 /usr/pbi/couchpotato-amd64/control stop
Stoppingthejail....arp: writing to routing socket: Invalid argument
arp: fe80::1b:27ff:fe00:e0b: Unknown host
Unmounting user-supplied file-systems
umount: fstab reading failure
Stopping jail with: /etc/rc.shutdown
[root@NAS:~] # warden start couchpotato_1
Mounting user-supplied file-systems
jail -c path=/mnt/jail/couchpotato_1 name=couchpotato_1 host.hostname=couchpotato_1 allow.raw_sockets=true persist vnet
Setting IP4 address: 10.0.0.32/24
route: writing to routing socket: Network is unreachable
add net default: gateway 192.168.1.1: Network is unreachable
Starting jail with: /etc/rc
jexec 5 /usr/pbi/couchpotato-amd64/control start 10.0.0.32 12346
[root@NAS:~] # warden stop couchpotato_1
jexec 5 /usr/pbi/couchpotato-amd64/control stop
Stoppingthejail....arp: writing to routing socket: Invalid argument
arp: fe80::2a:32ff:fe00:e0b: Unknown host
Unmounting user-supplied file-systems
umount: fstab reading failure
Stopping jail with: /etc/rc.shutdown

It appears that it has something to do with routing from FreeNAS itself.

imdos · Oct 16, 2013

WTF; It appears to be working.

I added a static route 10.0.0.0/24 and gateway 10.0.0.1.
Did a dhclient bridge0 (obtained 192.168.1.12 or 13).
Went into a newly created (standard) jail, and did dhclient epair0b (obtained 192.168.1.13 or 14).

As of now I'm able to install packages, ping every host on FreeNAS etc.

Now only thing left is do a deduction of what is really required and how to configure the jails to automatically setup DHCP.

marian78 · Oct 17, 2013

Hi,

i have same problem (maybe):

http://forums.freenas.org/threads/plex-after-restart-plex-jail-web-interface-not-working.15656/

....

imdos · Oct 20, 2013

@marian78;

I'm unsure if this is the same problem, but you could try this. This has worked for me.

Just append the following to /etc/rc.conf

Code:

ifconfig_epair0b="SYNCDHCP"
synchronous_dhclient="YES"      # Start dhclient directly on configured
dhclient_enable="YES"

Replace the following in /etc/rc.d/dhclient

Code:

#KEYWORD: nojail nostart
ifn="$2"

With this

Code:

########Otherwise it won't run inside a jail and won't start without help #########KEYWORD: nojail nostart
ifn="epair0b"

Then execute " service netif restart "
and voila; you should have received an IP-address from the DHCP server.

Important Announcement for the TrueNAS Community.

Jails nor Plugins can connect outside FreeNAS

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Wizard

Dabbler

Dabbler

Dabbler

Dabbler

Patron

Dabbler

Similar threads