Issues with multiple network interfaces for TrueNAS Scale?

[ernestr]

Hey all,

Thanks for the excellent storage solution and community.

I’m trying to add a subinterface for k3s. Whenever I add a subinterface and statically assign it an IP (on a different subnet than TrueNAS Scale’s IP) TrueNAS Scale can’t sustain a TCP connection to its orginial IP. Also, it assumes that its Webserver and SSH server should be listening on that new subinterface’s IP address.

I’m not sure if I am missing something fundamental about TrueNAS Scale’s configuration or if this is a bug. I expect the former, but I haven’t found anything in the docs or forum to confirm that. Hoping someone here can help point me to the docs or threads I’ve missed.

TrueNAS Scale is running on bare metal, and I have 4 physical NICS dedicated to it. Here’s my setup:

Network:

• vlan1 – Management
• vlan10 – I want k3s on this subnet and no other TrueNAS Scale services
• vlan20 – IoT

TrueNAS Scale Interfaces:

• en02 is on vlan1. I use it for TrueNAS management: Web UI, SSH, rsync
• en01 and en03 are slaved to bond0
• eno4 is unused and not connected to anything
• bond0 has two subinterfaces. When neither have IPs assigned via DHCP or statically I don’t have issues with maintaining a TCP connection to TrueNAS Scale via its management interface (en2 on vlan1)
  • vlan10@bond0
  • vlan20@bond0

Side note: I’ve got a VM on vlan20 (macvtap10@vlan20) that gets its IP allocated from the vlan20 subnet’s DHCP and it works great and doesn’t impact my TCP connection to TrueNAS.

If I try to assign an IP to either vlan20 or vlan10, I quickly lose IP connectivity to TruenNAS Scale through the vlan1 (management) IP. Meanwhile, the new subinterface performs great for HTTPS and SSH. When I remove the IP assignment the issue goes away. I've tried reboots and other things after saving the network config. No joy.

Is there a means for me to prevent TrueNAS services from listening on the new subinterface except for k3s? I’m happy to dive into IPTables but would like to know if there is a simpler option.

 

[ziotibia81]

System Settings -> General -> GUI has "Web Interface IPv4 Address" option.
Could it come in handy?

 

[ernestr]

Thank's ziotibia81. That worked perfectly for forcing nginx to bind only to the management IP on vlan1. Still, when connected to the Web UI or SSH from my laptop (which is on the vlan10 subnet), the session was being reset every 30 or so seconds.

My only intent for the vlan10@bond0 IP is for it to be the gateway for k3s. However, even with the web server bound to the vlan1 IP, traffic from the Web server was still routed through the vlan10@bond interface because my laptop was also on the same subnet. Since TrueNAS has IPs on vlan1 and vlan10 the network stack is selecting the shortest path back to that subnet and avoiding the single hop through pfSense.

I added a static route through vlan1 for my laptop's IP. This resolved the session drops. SSH, NFS, and HTTPS are stable at least to that single host. I would still like to expose a network interface for exclusive k3s use and prevent TrueNAS Scale from routing traffic through it.

I know I need to read more about k3s networking and TrueNAS Scale's implementation. Can anyone point me in the right direction? Should I take a look at traefik and whether that container can use a static IP without assigning one to TrueNAS?

 

[allcellspot22]

hey can someone help me please i upgrade my treunas to 12.0-u5 and now im getting this when i boot can someone help me tho what do i need to mount

 

[ernestr]

Thank's ziotibia81. That worked perfectly for forcing nginx to bind only to the management IP on vlan1. Still, when connected to the Web UI or SSH from my laptop (which is on the vlan10 subnet), the session was being reset every 30 or so seconds.

My only intent for the vlan10@bond0 IP is for it to be the gateway for k3s. However, even with the web server bound to the vlan1 IP, traffic from the Web server was still routed through the vlan10@bond interface because my laptop was also on the same subnet. Since TrueNAS has IPs on vlan1 and vlan10 the network stack is selecting the shortest path back to that subnet and avoiding the single hop through pfSense.

I added a static route through vlan1 for my laptop's IP. This resolved the session drops. SSH, NFS, and HTTPS are stable at least to that single host. I would still like to expose a network interface for exclusive k3s use and prevent TrueNAS Scale from routing traffic through it.

I know I need to read more about k3s networking and TrueNAS Scale's implementation. Can anyone point me in the right direction? Should I take a look at traefik and whether that container can use a static IP without assigning one to TrueNAS?

I upgraded to TrueNAS-SCALE-21.08-BETA.1. My static route band-aid no longer resolves the session dropping issue. Back to square one. Is there documentation on Truenas Scale's networking middleware anywhere?

 

[Vertigo 7]

the only thing I can think for you to do is to create bridge interfaces for each of your vlan interfaces and put the vlans as the only members of the bridges, and assign your ip addresses there. I've had to use that config in core or I get some weird network stuff happening and I've tried the same in scale and it's worked for me there too.

 

[FreeNASftw]

There is another thread regarding some issues with inability to create new sub interfaces and weird behaviour. I ended up giving up - will try again in VM

 

[ernestr]

Thanks all! I ended up deleting and re-adding my static route. It got me back to band-aid status (see above) but I want more and I'm curious why this worked. Still would really like to be able to read some documentation about the middleware.

Vertigo 7 I'm going to try your bridge approach when I get some time and my family allows for a Plex ASI

 

[skittlebrau]

Did you ever resolve this properly? I’m getting the exact same problem and have an almost identical network layout to you.

Setting a static route for my MacBook Pro and my desktop resolved the connection reset problems though, so thanks for that. Just wondering why this happens in the first place.

 

[skittlebrau]

After further testing, I’m satisfied with setting the static routes.

I have bridge interfaces for each VLAN. So my layout is like this:

• LAGG/bond0 (eno1+eno2)
• 4x VLANs assigned to bond0 interface (vlan10, vlan20, vlan50, vlan100)
• Bridges for each VLAN (br10, br20, br50, br100)

Static routes set for each corresponding gateway (192.168.10.0/24 with 192.168.10.1 gateway)

• 192.168.10.0/24
• 192.168.20.0/24
• 192.168.50.0/24
• 192.168.100.0/24

Now I don’t get any problems with the web UI timing out or SSH connections being dropped for any devices accessing the management interface.