TrueNAS Scale Bluefin - NIC questions and problems

[browntiger]

Can I suggest: (1) remove one card and create an alias and run k8s on one alias and truenas (ssl) on another. You do not need two cards.

====

If you do run two cards, you need to understand the issues involved. Since your servers are interconnected two both networks, having two cards each, the name to ip / to ARP to MAC resolution could be really bad. You could be getting randomly different IPs (internal or external). You will be getting this message that a server transitioned from internal ip to external ip and back. W/o fixes that setup is not stable.

You can set those configs and remediate the issues by manipulating hosts file and the arp table with e.g arp -s 192.168.86.65 00:50:ba:85:85:ca
If you want to go in that direction selecting something not routable:
192.168.86.10 (Public) and 192.169.86.10 (Private).

Or distinctly different subnets.
Do like 192.168.86.10/16 Public
And 10.7.10.10/16 Private

Be aware that Linux has this “Linux ARP flux” problem. When Linux answers ARP requests on wrong and unassociated interfaces per default. This is not for weak hearted.

 

[paddy01]

If you want multiple IP's on the same subnet that can be used for multiple purposes, then you're answer is multiple aliases on the same physical interface in TrueNAS.

 

[ninthturtle1034]

Can I suggest: (1) remove one card and create an alias and run k8s on one alias and truenas (ssl) on another. You do not need two cards.

====

If you do run two cards, you need to understand the issues involved. Since your servers are interconnected two both networks, having two cards each, the name to ip / to ARP to MAC resolution could be really bad. You could be getting randomly different IPs (internal or external). You will be getting this message that a server transitioned from internal ip to external ip and back. W/o fixes that setup is not stable.

You can set those configs and remediate the issues by manipulating hosts file and the arp table with e.g arp -s 192.168.86.65 00:50:ba:85:85:ca
If you want to go in that direction selecting something not routable:
192.168.86.10 (Public) and 192.169.86.10 (Private).

Or distinctly different subnets.
Do like 192.168.86.10/16 Public
And 10.7.10.10/16 Private

Be aware that Linux has this “Linux ARP flux” problem. When Linux answers ARP requests on wrong and unassociated interfaces per default. This is not for weak hearted.

Currently both NIC's are connected to the same network (just one network), one of my ideas was to have a second network and have one NIC on each network however I don't have the switch for that just yet.
The public vs private network has always sort of confused me, the private network would be what the K8's and stuff would use (blocking their internet access)? I use CF Tunnels for my external access so as long as I put my CF Tunnel host on the same private network (and the public one) it will still be able to route CF tunnels to my TrueNAS apps right?

And the virtual link aggregation loadbalancer that TrueNAS has wouldn't really be worth it to setup I take it?