How to use my 2nd NIC

[john.doe79]

Hello,

I've got a 2 NIC Gbit/s mainboard and I would like to make use of the 2nd NIC.

I've tried LAGG/LACP with my UniFi switch. But as soon as I started jails (selected the lagg0) the connection dropped.

I've then removed the IP from the one interface, set that to DHCP, set the Link Aggregation in TrueNAS 13.

TrueNAS came up with that correctly. I've assigned the lagg0 interface to jails then with IP addresses (in the VLAN3 Subnet, where also the LAGG was in), which caused the network to become unstable until I removed the interfaces again.

After that I tried it without LAGG. 1 port to my UI 24 Port Switch and 1 port to my UDM-Pro. Port at the switch had IP assigned, port at the UDM-Pro didn't had an IP assigned and no DHCP - shouldn't be used for anything at TrueNAS, just for Jails & VMs. But as soon as I assigned the interface to a Jail, also the network dropped until I disconnected the Port from the UDM-Pro.

Can't explain better what happens.

I would like to use the port to the UDM-Pro for the more internet facing stuff (downloads ...) and the other interface on the USW24 for the more internal stuff like SMB, TimeMachine, Plex, ...

Any suggestion to configure that best?

 

[sretalla]

If you want it to work, you should assign a bridge and have the IP address on that.

You should also disable hardware offload on the interfaces.

It should go like:

Physical Interfaces --> LAGG --> VLANS (if you want those) --> Bridge (TrueNAS IP address here) --> Jails/VMs, etc.

 

[Dice]

You should also disable hardware offload on the interfaces.

Can I ask for the reason for this?
I've never seen this be recommended except for virtualized nics.

edit:
I ....reached for the search button.
In case anyone else finds themselves here:

Disabling Hardware Offloading is discouraged as it can reduce network performance. However, disabling this option might be needed when the interface is managing jails, plugins, or virtual machines.

Setting Up a Network Bridge

Provides instructions on setting up a network bridge interface on TrueNAS CORE.

www.truenas.com

 

[john.doe79]

If you want it to work, you should assign a bridge and have the IP address on that.

You should also disable hardware offload on the interfaces.

It should go like:

Physical Interfaces --> LAGG --> VLANS (if you want those) --> Bridge (TrueNAS IP address here) --> Jails/VMs, etc.

so that would mean.

I remove the config (IP) from both NICs.
I create a LAGG with both NICs with no IP Config?
I create a bridge using the LAG and assign the IP 192.168.3.9 for my TrueNas
In the Jail/VMs I assign the bridge?

Is that correct? (should I do that locally via the console? Step 2 and 3?)

 

[sretalla]

I remove the config (IP) from both NICs.
I create a LAGG with both NICs with no IP Config?
I create a bridge using the LAG and assign the IP 192.168.3.9 for my TrueNas
In the Jail/VMs I assign the bridge?

Is that correct? (should I do that locally via the console? Step 2 and 3?)

Looks right.

All should be done in the GUI and make sure to apply/test and come back within 60 seconds to confirm and save.

 

[john.doe79]

Looks right.

All should be done in the GUI and make sure to apply/test and come back within 60 seconds to confirm and save.

right.

what is the purpose/benefit of the bridge at that point? He is doing it w/o a bridge: https://www.youtube.com/watch?v=B7Dpuu_gBmo

 

[sretalla]

If it "works" you can do it that way if you want...

A bridge is flexible and in most cases is the best option to share one or more physical adapters among jails or VMs.

The system will often do it for you as part of the jail setup if it's not already there.

 

[john.doe79]

If it "works" you can do it that way if you want...

A bridge is flexible and in most cases is the best option to share one or more physical adapters among jails or VMs.

The system will often do it for you as part of the jail setup if it's not already there.

I've now created bridge0 using the lagg0, but it is not available in the jail's NIC selection ... (settings are applied)

 

[Patrick M. Hausen]

You need to set vnet_default_interface to none and further down in the interfaces section set vnet0:bridge0.

This entire area in TrueNAS suffers from wrong default settings and a lack of proper documentation. It's all "obvious" if you know FreeBSD networking inside out and have been working with iocage a couple of years

Possibly that will improve: https://jira.ixsystems.com/browse/NAS-114755

 

[john.doe79]

You need to set vnet_default_interface to none and further down in the interfaces section set vnet0:bridge0.

This entire area in TrueNAS suffers from wrong default settings and a lack of proper documentation. It's all "obvious" if you know FreeBSD networking inside out and have been working with iocage a couple of years

Possibly that will improve: https://jira.ixsystems.com/browse/NAS-114755

It is day 4 with FreeBSD, I will get there with all your kind support and suggestions. :)

vnet0:bridge0 does not show up when I select none?

 

[sretalla]

You're still in the Basic Properties section... you need to set that in the Network Properties section, you can leave that one at vnet0.

 

[Patrick M. Hausen]

you can leave that one empty.

Really? I'd say vnet0 is the correct setting. And then further down we do the mapping vnet0 --> bridge0.

 

[sretalla]

Really? I'd say vnet0 is the correct setting. And then further down we do the mapping vnet0 --> bridge0.

Yes, I already saw that myself and updated my post, but you must have beaten me to it.

 

[john.doe79]

That worked. Another learning day. Thanks!

 

[john.doe79]

another question, to learn something more.

Is it possible to add a second NIC to a Jail? So having lets say the management GUI (website) internally and the 2nd interface (I got 6x 1 Gbit/s on that box, only 3 used so far) on the DMZ?

 

[Patrick M. Hausen]

Yes, it is. You need to create bridge1 like you did with bridge0 but pick a different physical interface. You need not necessarily assign an IP address on that bridge interface if it is only the jails communicating via that network and not the NAS itself. For the physical interface disable hardware offloading and put up in the options field.

Then in the jail settings, Network Properties, interfaces you set vnet0:bridge0,vnet1:bridge1 and last in the Basic Properties section you can then add the interface vnet1 and an IP address and prefix length.

 

[john.doe79]

Thanks for the input - haven't tested that yet.

Stumbled over the the initial config of having the LAGG with 2 Ports at my USW-24-PoE#17&18 (Aggregation, VLAN trusted tagged, network 192.168.3.0/24, defgw 192.168.3.1, the UDM-Pro) and 1 port at my UDM-Pro#6 (VLAN management tagged, network 192.168.1.0/24, defgw 192.168.1.1, the UDM-Pro). Seems like my TrueNAS act as a router/bridge - at least my Unifi setup thinks so - or I miss understood something badly. Also DHCP stopped working at the trusted VLAN at that point.

If I shut down Port#6 at the UDM-Pro it normalize and it revert back to the uplink between the #11 and #26 (both SFP).

 

[Patrick M. Hausen]

We would need a full ifconfig output of your TrueNAS host when everything (jails, VMs) is up and running, to diagnose that.

 

[john.doe79]

We would need a full ifconfig output of your TrueNAS host when everything (jails, VMs) is up and running, to diagnose that.

Here we go

Code:

igb0: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: LAN1
    options=8120b8<VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER>
    ether a0:36:9f:20:a6:04
    media: Ethernet autoselect
    status: no carrier
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether a0:36:9f:20:a6:05
    media: Ethernet autoselect
    status: no carrier
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
    ether a0:36:9f:20:a6:06
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb3: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
    ether a0:36:9f:20:a6:06
    hwaddr a0:36:9f:20:a6:07
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
igb4: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
    ether b4:2e:99:31:9b:63
    media: Ethernet autoselect (1000baseT <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
em0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: management
    options=4810099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,VLAN_HWFILTER,NOMAP>
    ether b4:2e:99:31:9b:62
    inet 192.168.1.10 netmask 0xffffff00 broadcast 192.168.1.255
    media: Ethernet autoselect
    status: no carrier
    nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
    options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
    inet6 ::1 prefixlen 128
    inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
    inet 127.0.0.1 netmask 0xff000000
    groups: lo
    nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
    groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Bridge at lagg0
    ether 58:9c:fc:10:ff:bc
    inet 192.168.3.9 netmask 0xffffff00 broadcast 192.168.3.255
    id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
    maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
    root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
    member: vnet0.26 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 14 priority 128 path cost 2000
    member: vnet0.24 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 16 priority 128 path cost 2000
    member: em0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 6 priority 128 path cost 20000
    member: vnet0.19 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 12 priority 128 path cost 2000
    member: vnet0.18 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 13 priority 128 path cost 2000
    member: vnet0.16 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 11 priority 128 path cost 2000
    member: lagg0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
            ifmaxaddr 0 port 15 priority 128 path cost 10000
    groups: bridge
    nd6 options=9<PERFORMNUD,IFDISABLED>
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: Port 17 and 18
    options=4a500b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,NOMAP>
    ether a0:36:9f:20:a6:06
    laggproto lacp lagghash l2,l3,l4
    laggport: igb2 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: igb3 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    groups: lagg
    media: Ethernet autoselect
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.16: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: mineos as nic: epair0b
    options=8<VLAN_MTU>
    ether b6:2e:99:c5:1d:5f
    hwaddr 02:1c:61:0b:a5:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.18: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: tautulli as nic: epair0b
    options=8<VLAN_MTU>
    ether b6:2e:99:46:b0:16
    hwaddr 02:01:50:35:d5:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.19: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: plexbeta as nic: epair0b
    options=8<VLAN_MTU>
    ether b6:2e:99:96:a3:59
    hwaddr 02:36:9a:7e:89:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.24: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: radarr as nic: epair0b
    options=8<VLAN_MTU>
    ether b6:2e:99:c1:df:ca
    hwaddr 02:f8:f1:90:24:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.26: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    description: associated with jail: myUsenetJail as nic: epair0b
    options=8<VLAN_MTU>
    ether b6:2e:99:ac:8f:4b
    hwaddr 02:6b:b1:a8:91:0a
    groups: epair
    media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
    status: active
    nd6 options=9<PERFORMNUD,IFDISABLED>

 

[sretalla]

I'm pretty sure you can't add em0 into the same bridge where you have a completely different subnet and the LAGG assigned.

Well... seems you can, but I don't expect it to work properly.