Static IP in VNET/NAT for jail?

[Grand_Moff_Twerkin]

Ok I hear you, but then how do I edit the bridge0 interface when there is no listing in Network / Interface. Also I can't unset the LAGG0 IP through the GUI it won't let me.

 

[sretalla]

Ok I hear you, but then how do I edit the bridge0 interface when there is no listing in Network / Interface

You should be able to create it as a new bridge interface.

Also I can't unset the LAGG0 IP through the GUI it won't let me.

Can we get a look at how it's showing up and what's preventing you?

 

[sretalla]

You may need to completely break it down by removing the lagg interface and making it again, then adding the bridge... as long as you don't test in between steps, you'll be able to do that.

 

[Grand_Moff_Twerkin]

I ended up using ifconfig to destroy the bridge0 interface. Then I went into the GUI and edited lagg0 removing the static ip, I then created bridge0 with lagg0 as the member and assigned 192.168.0.100 (static) and now the bridge listing is staying in the gui. Now I have to find out if this is working or if I have to update the MAC in pfSense.

 

[Grand_Moff_Twerkin]

It was saying that the server's ip was offline, so I switched the MAc to that of the bridge and now it's online. Still not working though keep getting [B]Error:[/B] [EFAULT] + Acquiring DHCP address: FAILED, address received: 0.0.0.0/8 Stopped test due to DHCP failure errors when tring to start dhcp jails.

 

[sretalla]

So you're creating a new jail with the command I gave and it still doesn't get DHCP?

Can we see the ifconfig again while that jail is running?

 

[Grand_Moff_Twerkin]

Nope still no DHCP, here is ifconfig while test jail is trying to start:

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=a120b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6>
    ether 0c:c4:7a:b3:47:aa
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=a120b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6>
    ether 0c:c4:7a:b3:47:aa
    hwaddr 0c:c4:7a:b3:47:ab
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: lagg0
    options=a120b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6>
    ether 0c:c4:7a:b3:47:aa
    laggproto loadbalance lagghash l2,l3,l4
    laggport: igb0 flags=4<ACTIVE>
    laggport: igb1 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:15:9b:5a:8c:00
    inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.137 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 10000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.137: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: test as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:ae:1b:75
    hwaddr 02:91:26:70:b6:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

 

[sretalla]

OK, well that all looks as it should... have you defined the default gateway and DNS in the Network Global Configuration in the GUI?

And what about a test jail with a static IP?

iocage create -n "teststatic" -r 12.3-RELEASE defaultrouter="192.168.0.1" vnet="on" ip4_addr="192.168.0.101/24"

 

[sretalla]

Maybe a reboot for good measure also and re-test DHCP and static...

 

[Grand_Moff_Twerkin]

Yes DNS and default gateway are defined in global networking. This container behaves the same way it did before where it will start but it will have no networking, like it can't find the default gateway to resolve DNS.

Code:

igb0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether 0c:c4:7a:b3:47:aa
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: member of lagg0
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether 0c:c4:7a:b3:47:aa
    hwaddr 0c:c4:7a:b3:47:ab
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x3
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: lagg0
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether 0c:c4:7a:b3:47:aa
    laggproto loadbalance lagghash l2,l3,l4
    laggport: igb0 flags=4<ACTIVE>
    laggport: igb1 flags=4<ACTIVE>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    ether 02:15:9b:5a:8c:00
    inet 192.168.0.100 netmask 0xffffff00 broadcast 192.168.0.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.144 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 7 priority 128 path cost 2000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 5 priority 128 path cost 10000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.144: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: teststatic as nic: epair0b
    options=8<VLAN_MTU>
    ether 0e:c4:7a:fe:a1:16
    hwaddr 02:3f:2c:df:f2:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=1<PERFORMNUD>

Reboot here I go!!!

 

[sretalla]

Another thought here... what switch are you using? and do you allow impersonation/spoofing for all switch ports on it?

The fact that you have the bridge working sort-of says it must be, but it's worth a check.

 

[Grand_Moff_Twerkin]

The reboot went ok all my NAT jails started like they usually do, the DHCP jail won't start at all with the same error, and the static jail starts but doesn't get internet. It's basically the same as if I didn't create the bridge.

As for the switch it's a TP-Link TL-SG1016DE and no I don't have impersonation/spoofing or any type of port mirroring going on there.

 

[sretalla]

TP-Link TL-SG1016DE and no I don't have impersonation/spoofing or any type of port mirroring going on there.

OK, but I guess you have config in there for the LAGG NIC PORTS?

 

[sretalla]

Is IGMP snooping enabled on the switch?

 

[Grand_Moff_Twerkin]

Yes there are settings for static LAG in the switch but I haven't set it up as I didn't think I needed it or at the time I tried to use LACP and it wouldn't work so I set up TrueNAS with load balance LAGG. IGMP snooping is enabled in the switch yes.

 

[sretalla]

Yes there are settings for static LAG in the switch but I haven't set it up as I didn't think I needed it or at the time I tried to use LACP and it wouldn't work so I set up TrueNAS with load balance LAGG. IGMP snooping is enabled in the switch yes.

OK, maybe the IGMP snooping is fine if the LAGG is set up right, but I think it isn't.

Perhaps you should try setting your LAGG as failover (since that's the simplest mode that doesn't need the awareness of the switch) and see if that makes any difference.

I very much doubt that there's any benefit at all to having it in Loadbalance mode.

 

[Grand_Moff_Twerkin]

Guess what? Now it works since I switched it over to failover. It was the LAGG all along!! Now I'm wondering if there was any point creating the bridge? I'm also wondering if there is any point of even having the LAGG. PEBKAC and IDI0T errors.

Thanks for your help!!!

 

[Grand_Moff_Twerkin]

No I don't need the bridge0, works fine without it. Reverted all the changes made and everything works like it should.

 

[sretalla]

Now I'm wondering if there was any point creating the bridge?

Read that linked post from @Patrick M. Hausen ... maybe it "works" but it's not properly configured if it's like that with an IP on the LAGG.

 

[sretalla]

I'm also wondering if there is any point of even having the LAGG.

In my 25 years of experience working in this sector of the industry, I have seen plenty of problems caused by the poor configuration of LAGG/Loadbalancing NICs and their attached switching equipment, but I can't really recall any significant positives where it saved us or made a substantive positive impact (almost all the failures where we might have been saved ended up being caused further up the network chain).

In a "home/lab" environment, it might be interesting to run it for the "fun", but NICs are solid state and don't often fail... same with switch ports (although I am aware of many failures of these in my personal and professional travels, so maybe this is the one "pro" for using it in failover mode).

Almost all of the home/lab use I see is people thinking they can make their 1 client (usually file copy or crystal diskmark) go faster by running on 2 NICs... it doesn't work like that and all you can get to/from one client is the limit of a single NIC in the team/LAGG.

Anyway, happy jailing, hopefully you can get whatever you wanted to work now. (and do or don't do the bridging that's recommended... up to you).