SOLVED Lacking internet from jails on one of two subnets - Setting up a FreeNAS Dual NIC configuration

[Jimstein Perrless]

Setting up a FreeNAS Dual NIC configuration: the motherboard NIC as em0 (FreeNAS is on 192.168.0.50) and an additional pcie NIC as em1 (FreeNAS is on 192.168.1.51). I am lacking Internet from jails on inner subnet 192.168.0.0/24 (with gateway 192.168.0.1) that is a neighbor network to the outer border network 192.168.1.0/24. Gateway 192.168.0.1 has the ip 192. 168.1.251 on the outer border network. Rest of inner and outer networks and jails on the outer border network has Internet access.

Advanced options set for each Jail:
* no DHCP
* a default IPv4 gateway matching the nic
* disabling VIMAGE ( => NAT is disabled) and picked the right NIC (em0 for 192.168.0.X and em1 for 192.168.1.X)
* I set jails IPv4 default gateway for em0 jails to 192.168.0.1. But this field is blank after saving options and reopening the advanced options.
Edit: P.S After some digging I found that the value in field "IPv4 default gateway" is in fact saved in this file mnt/red/jails/.<name_of_jail>.meta/defaultrouter-ipv4. One just can't see what this value was saved as though the Freenas web interface. (But is the value ignored? :S)

This is the jails metadata:
warden list -v

Code:

id: 1
host: transmission_1
iface: em0
ipv4: 192.168.0.10/24
alias-ipv4:
bridge-ipv4:
alias-bridge-ipv4:
defaultrouter-ipv4: 192.168.0.1
ipv6:
alias-ipv6:
bridge-ipv6:
alias-bridge-ipv6:
defaultrouter-ipv6:
autostart: Enabled
vnet: Disabled
nat: Disabled
mac:
status: Running
type: pluginjail
flags: allow.raw_sockets=true
...

id: 10
host: lms_2
iface: em1
ipv4: 192.168.1.19/24
alias-ipv4:
bridge-ipv4:
alias-bridge-ipv4:
defaultrouter-ipv4:
ipv6:
alias-ipv6:
bridge-ipv6:
alias-bridge-ipv6:
defaultrouter-ipv6:
autostart: Enabled
vnet: Disabled
nat: Disabled
mac:
status: Running
type: standard
flags: allow.raw_sockets=true

This is the output of netstat -rn from FreeNAS (192.168.0.50):
[root@freenas] ~# netstat -rn

Code:

Routing tables



Internet:

Destination		Gateway			Flags	  Netif Expire

default			192.168.1.250	  UGS		 em1

127.0.0.1		  link#3			 UH		  lo0

192.168.0.0/24	 link#1			 U		   em0

192.168.0.10	   link#1			 UHS		 lo0

192.168.0.12	   link#1			 UHS		 lo0

192.168.0.13	   link#1			 UHS		 lo0

192.168.0.14	   link#1			 UHS		 lo0

192.168.0.15	   link#1			 UHS		 lo0

192.168.0.50	   link#1			 UHS		 lo0

192.168.1.0/24	 link#2			 U		   em1

192.168.1.11	   link#2			 UHS		 lo0

192.168.1.16	   link#2			 UHS		 lo0

192.168.1.18	   link#2			 UHS		 lo0

192.168.1.19	   link#2			 UHS		 lo0

192.168.1.21	   link#2			 UHS		 lo0

192.168.1.51	   link#2			 UHS		 lo0

The output of netstat -rn from the jail 192.168.0.10
root@transmission_1:/ # netstat -rn

Code:

Routing tables



Internet:

Destination		Gateway			Flags	  Netif Expire

192.168.0.10	   link#1			 UHS		 lo0

The output of netstat -rn from the jail 192.168.1.19
root@lms_2:/ # netstat -rn

Code:

Routing tables

Internet:

Destination		Gateway			Flags	  Netif Expire

192.168.1.19	   link#2			 UHS		 lo0

root@192.168.0.1 / # ping google.com
[root@freenas] ~# ping google.com

Code:

PING google.com (216.58.211.142): 56 data bytes
64 bytes from 216.58.211.142: icmp_seq=0 ttl=51 time=4.523 ms
64 bytes from 216.58.211.142: icmp_seq=1 ttl=51 time=4.575 ms
^C
--- google.com ping statistics ---
2 packets transmitted, 2 packets received, 0.0% packet loss
round-trip min/avg/max/stddev = 4.523/4.549/4.575/0.026 ms
[root@freenas] ~# traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  DD-WRT (192.168.1.250)  0.221 ms  0.204 ms  0.135 ms
 2  *^C

root@transmission_1:/ # ping google.com

Code:

PING google.com (216.58.209.142): 56 data bytes

^C

--- google.com ping statistics ---

22 packets transmitted, 0 packets received, 100.0% packet loss

root@transmission_1:/ #
root@transmission_1:/ # traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  DD-WRT (192.168.1.250)  0.640 ms  0.632 ms  0.488 ms
 2  * * *
 3  * * *
 4  * * *
^C
root@transmission_1:/ # traceroute google.com
traceroute to google.com (216.58.209.110), 64 hops max, 40 byte packets
 1  DD-WRT (192.168.1.250)  0.584 ms  0.599 ms  0.420 ms
 2  * * *
 3  * * *
^C​

root@lm_2:/ # ping google.com

Code:

PING google.com (172.217.22.174): 56 data bytes

64 bytes from 172.217.22.174: icmp_seq=0 ttl=51 time=5.198 ms

64 bytes from 172.217.22.174: icmp_seq=1 ttl=51 time=5.119 ms

^C

--- google.com ping statistics ---

13 packets transmitted, 13 packets received, 0.0% packet loss

round-trip min/avg/max/stddev = 5.038/5.095/5.198/0.046 ms
root@lms_2:/ # traceroute 8.8.8.8
traceroute to 8.8.8.8 (8.8.8.8), 64 hops max, 40 byte packets
 1  DD-WRT (192.168.1.250)  0.446 ms  0.207 ms  0.140 ms
^C

root@transmission_1:/ # route add 0.0.0.0 192.168.0.1

Code:

route: writing to routing socket: Operation not permitted

So I added in FreeNAS network settings a static route for destination network 192.168.0.0/24 to gateway 192.168.0.1. Still no Internet from jails on subnet 192.168.0.0/24


If I add a computer to the 92.168.0.0/24 it will be able to access internet. Actually I post this content from that subnet. So it appears that it has something to do with internal routing on FreeNAS itself but my understanding is lacking. Please help me do better.

I can ping other computers on both border network and inner network from inner network jail. Also able to ping the border gateway - so what is rely going on?

Code:

root@transmission_1:/ # ping 192.168.0.50
PING 192.168.0.50 (192.168.0.50): 56 data bytes
64 bytes from 192.168.0.50: icmp_seq=0 ttl=64 time=0.020 ms
64 bytes from 192.168.0.50: icmp_seq=1 ttl=64 time=0.024 ms
64 bytes from 192.168.0.50: icmp_seq=2 ttl=64 time=0.041 ms


root@transmission_1:/ # ping 192.168.1.53
PING 192.168.1.53 (192.168.1.53): 56 data bytes
64 bytes from 192.168.1.53: icmp_seq=0 ttl=255 time=0.545 ms
64 bytes from 192.168.1.53: icmp_seq=1 ttl=255 time=6.819 ms


root@transmission_1:/ # ping 192.168.0.133
PING 192.168.0.133 (192.168.0.133): 56 data bytes
64 bytes from 192.168.0.133: icmp_seq=0 ttl=128 time=0.750 ms
64 bytes from 192.168.0.133: icmp_seq=1 ttl=128 time=0.372 ms


root@transmission_1:/ # ping 192.168.1.250
PING 192.168.1.250 (192.168.1.250): 56 data bytes
PING 192.168.1.250 (192.168.1.250): 56 data bytes
64 bytes from 192.168.1.250: icmp_seq=0 ttl=63 time=0.683 ms
64 bytes from 192.168.1.250: icmp_seq=1 ttl=63 time=0.637 ms

 

[dlavigne]

Check this:

• subnet 192.168.0.0/24 (with gateway 192.168.0.1) that is a subnet to the border network 192.168.1.0/24 (that does not make sense with that mask)

 

[Jimstein Perrless]

Interesting. But why is that? If I use the Subnet Calculator it say mask 192.168.0.0/24 => 192.168.0.1 - 192.168.0.254 and 192.168.1.0/24 =>192.168.1.1 - 192.168.1.254. And that is what i expected.

Edit: Oh. I think I understand now. It is in the wording. It is not a subnet of the other but a neighbor network. I have clarified that in the original question.

 

[Jimstein Perrless]

Still had no success with accessing internet from 192.168.0.x/24 jails. Can't figure out how to configure custom default route and not use the Freenas global default route.

Freenas Network Summery show:

Name IPv4 Address

em1 192.168.1.51/24
192.168.1.18/24
192.168.1.19/24
192.168.1.21/24
192.168.1.16/24
192.168.1.11/24

em0 192.168.0.50/24
192.168.0.15/24
192.168.0.12/24
192.168.0.13/24
192.168.0.14/24
192.168.0.10/24


Nameserver (what is inside etc/resolv.conf)

192.168.1.250

192.168.0.1

208.67.222.222

8.8.8.8


Default route

192.168.1.250

 

[Jimstein Perrless]

Try set default route by editing defaultrouter variable in /path/to/jails/etc/rc.conf:

Code:

defaultrouter="192.168.0.1"

Restarted the jail and verify it with:

Code:

# netstat -r -n
Routing tables
Internet:
Destination		Gateway			Flags	  Netif Expire
192.168.0.10	   link#1			 UHS		 lo0
# ping google.com
PING google.com (172.217.22.174): 56 data bytes

Nothing indicate that jail did pick up the change. :(

 

[dlavigne]

Were you able to resolve this?

 

[Jimstein Perrless]

Were you able to resolve this?

Sorry but no. I'll will return to this problem and use this thread to log my progress. Tnx.

 

[Jimstein Perrless]

Now I am 100% sure I actually is able to set the default route correctly on jails in my inner subnet. This is what I did:
Define IP and iface in GUI and leave VIMAGE checked. This is VERY important, and this was the thing that made the difference. enable VIMAGE will nullify your choose of iface so you need to set it manually from console.

1. Create <jail_home>/.<jail_name>.meta/iface file. Put the name of the interface in the file if not correct.
2. Create <jail_home>/.<jail_name>.meta/defaultrouter-ipv4. Put the IP of the router.
3. Restart the jail from the command line. warden stop <jail_name> then warden start <jail_name>

Now I can ping google.com and bing.com. If i did not enable VIMAGE i would get IP to google.com and bing.com but still get 100.0% packet loss. With VIMAGE checked I do not get packet loss inside jails on the inner subnet.

I would say there is something very wrong with the FREENAS GUI that don't allow me to set up my jails correctly. This was on FreeNAS-9.10.2-U1 (86c7ef5) so perhaps this is fixed in later versions.