Static IP in VNET/NAT for jail?

[Robert Thomspon]

Hey guys... is it possible to set a static occurring IP address for jail? (currently my TrueNAS seems to change the assigned IP of its jails on reboots of the jail (currently occurring as 172.0.10.X where X changes) OR, is there a way for one jail to communicate with another through the NAT provided by TrueNAS? (currenly, one jail can only address another by the assigned address from TrueNAS... Im fine with the changing NAT if i can address the TrueNAS directly by port for the jail the way outside running devices do it...)

Any help is appreciated.

 

[sretalla]

I think the answer to that is you can't, but you might consider working with dns/mdns to refer to systems by name instead if they need to talk between themselves inside the NAT network.

 

[Grand_Moff_Twerkin]

Is there a solution for this?

 

[Ericloewe]

For what part of this, exactly?

 

[Grand_Moff_Twerkin]

1.) How can I either lock the IP's IOcage is using for it's NAT jails, so they don't change when I add or delete jails and reboot the host?
2.) Is there any good scripts that will keep track of running jails and update the host files in them all?
3.) Does IOcage run some sort or internal DNS so that I can just point it toward host names?
4.) Can I use port forwarding from my router? If so how?
5.) Why will pointing the jails toward the host on their running port not work? I have done this with Docker containers on Linux.

Other than trying to get DHCP or static IP's set up (which aren't working for me either (post coming up)) there has to be some way to work around this?

 

[Ericloewe]

It seems that satic IP addresses inside the NAT'ed network are easy: https://iocage.readthedocs.io/en/latest/networking.html#shared-ip

Does IOcage run some sort or internal DNS so that I can just point it toward host names?

I don't think so, no.

4.) Can I use port forwarding from my router? If so how?

Point it at the jail's IP address? There's not much more to it. With NAT networking, you obviously can't have multiple NAT'ed jails on the same port. Or just use VNET and have a proper networking stack inside the jail and handle things as you would for a physical machine.

5.) Why will pointing the jails toward the host on their running port not work? I have done this with Docker containers on Linux.

I don't see why, either, but it feels dirty.

 

[Grand_Moff_Twerkin]

None of that works. If I go dhcp it doesn't work, the jail doesn't even start. If I go static ip it doesn't work, the jail will not find the default gateway to resolve dns. I may look into port forwarding, but if the ip's change. Or I might just give up, this is making me crazy.

 

[sretalla]

OK, let's calm the silliness down... jails work, IP addresses work... what's wrong with your situation in particular is at this point unknown, so let's break it down to the real basics and go from there:

iocage create -n "test" -r 12.3-RELEASE defaultrouter="192.168.1.1" dhcp="on" vnet="on" bpf="yes"

Can you run that command and get a jail created with a DHCP address of its own? (modify the defaultrouter to your pfSense address)

 

[Grand_Moff_Twerkin]

Goes like it always does:

Code:

sudo iocage create -n "test" -r 12.3-RELEASE defaultrouter="192.168.0.1" dhcp="on" vnet="on" bpf="yes"
test successfully created!

| None | test | down | 12.3-RELEASE | DHCP |

sudo iocage start test
No default gateway found for ipv6.
* Starting test
  + Started OK
  + Using devfs_ruleset: 1010 (iocage generated default)
  + Configuring VNET OK
  + Using IP options: vnet
  + Starting services OK
  + Executing poststart OK
  + Acquiring DHCP address: FAILED, address received: 0.0.0.0/8

Stopped test due to DHCP failure

I have also tried making sure that raw sockets have been enabled, I have tried disabling hardware offloading (no difference). Like I have said in this post and another that I created recently, I cannot get dhcp or static ip jails to work. I know my dhcp server is working because I can get it to hand out addresses to a Virtualbox on a bridged adapter all day long. Not sure what else I can do here? I don't think it's my lagg0, or my vlan's.

 

[sretalla]

OK, so it's probably the bridging that needs work...

ifconfig on the host please...

 

[Grand_Moff_Twerkin]

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=e13abb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:b3:47:aa
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=e13abb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:b3:47:aa
    hwaddr 0c:c4:7a:b3:47:ab
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: lagg0
    options=e13abb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO6,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,TXCSUM_IPV6>
    ether 0c:c4:7a:b3:47:aa
    inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
    laggproto loadbalance lagghash l2,l3,l4
    laggport: igb0 flags=4<ACTIVE>
    laggport: igb1 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:15:9b:5a:8c:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 10000
    groups: bridge
    nd6 options=1<PERFORMNUD>
vnet0.96: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Jackett as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:b3:a6:26
    hwaddr 02:6e:e2:c1:a3:0a
    inet 172.16.0.1 netmask 0xfffffffc broadcast 172.16.0.3
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.97: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Prowlarr as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:e9:1c:1b
    hwaddr 02:e2:34:94:de:0a
    inet 172.16.0.5 netmask 0xfffffffc broadcast 172.16.0.7
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.98: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Radarr as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:f9:39:8e
    hwaddr 02:43:f4:2f:6c:0a
    inet 172.16.0.9 netmask 0xfffffffc broadcast 172.16.0.11
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.99: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Sonaar as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:db:8d:39
    hwaddr 02:63:1a:46:10:0a
    inet 172.16.0.13 netmask 0xfffffffc broadcast 172.16.0.15
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.100: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Syncthing as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:4a:a8:cc
    hwaddr 02:e8:fc:e1:dd:0a
    inet 172.16.0.17 netmask 0xfffffffc broadcast 172.16.0.19
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>
vnet0.101: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: Transmission as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:5d:8c:c3
    hwaddr 02:0b:c1:5c:ce:0a
    inet 172.16.0.21 netmask 0xfffffffc broadcast 172.16.0.23
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

I tried giving the bridge0 an inet address once but that didn't fix anything.

 

[sretalla]

I don't think you want the IP address to be on the LAGG... rather on the Bridge.

Can we see it with the jail running?

Also iocage get vnet_default_interface test

 

[Grand_Moff_Twerkin]

I don't think you want the IP address to be on the LAGG... rather on the Bridge.

This is where you're losing me. You think I need to take the IP off the LAGG and put it on bridge0? I don't even have the slightest idea of how to do this.

Can we see it with the jail running?

It doesn't start successfully but here's when it's starting:

Code:

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:15:9b:5a:8c:00
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.111 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 10000
    groups: bridge
    nd6 options=1<PERFORMNUD>
    
    vnet0.111: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: test as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:ae:1b:75
    hwaddr 02:d3:ac:89:62:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

iocage get vnet_default_interface test

auto

 

[sretalla]

You think I need to take the IP off the LAGG and put it on bridge0? I don't even have the slightest idea of how to do this.

In Network | Interfaces

You would need to ensure that the LAGG isn't on DHCP (I assume probably not), but do that first and save/test.

Stop all your jails.

Then go back to Network | Interfaces and unset the IP from the LAGG (but don't do the test of the changes yet).

Update the bridge0 interface to have the IP, then save and test the changes... if all goes well, configm within 60 seconds to keep the updated settings... if not, you'll revert back to the previous working settings after 60 seconds.

 

[Grand_Moff_Twerkin]

The LAGG is set up as statically assigned through the pfSense router with its MAC and as for network interfaces I only have igb0, igb1, and lagg0. There is no bridge0 it has created that all on its own.

So you want me to unclick DHCP? Assign a static IP?

 

[Grand_Moff_Twerkin]

 

[sretalla]

So you want me to unclick DHCP? Assign a static IP?

Yes, that's step 1. Later we'll remove that too.

It's probably important that you stop your jails before messing with the network settings.

You can leave the Static Lease in pfSense, it won't be hurting anything and will remind you not to assign that address to another host

FYI, we're working on this premise: https://www.truenas.com/community/t...ace-best-practice-questions.93275/post-645515

The reason for the IP address on the bridge requirement is that it is documented for FreeBSD that any bridge member interface must not have an IP address. It breaks multicast.

I think DHCP uses multicast, so that could certainly explain some of what you're seeing.

 

[Grand_Moff_Twerkin]

I stopped all my jails, assigned a static IP of 192.168.0.100 to the lagg0 and am currently looking through the ifconfig man page as to how to add 192.168.0.100 as the bridge0 IP. For some reason I can create a bridge but it won't stay in the Network / Interfaces page.

 

[Grand_Moff_Twerkin]

Used command ifconfig bridge0 inet 192.168.0.100/24, now I have two interfaces with the same IP and I can't remove the one from lagg0 through the GUI. Should I try removing it with ifconfig? Should I be removing the static IP from pfSense? Or changing the MAC to bridge0?

 

[sretalla]

Go back to what I said originally... you're working in the GUI here, not with ifconfig... changes made with ifconfig in the CLI won't survive a reboot.

You might want to set all your jails to not starting on boot and reboot the system to come up clean without the bridge in place.