TrueNAS Scale Bluefin - NIC questions and problems

[ninthturtle1034]

Hi all, First off I just want to say that this is my first actual post so I apologize if I breach any of the rules. I attempted to ask my question in the TrueNAS Siscord and on a few other Discords however I did not receive a response.

Summary: So I was watching a YouTube video on 'Secure HTTPS Traffic for TrueNAS Scale apps (traefic)' and gave it a go, but it's not worked out how I'd expect it to.

I have 2 NIC's in my truenas scale server, I just attempted to assign an alias IP address to the first NIC, but it appears to have offlined the second NIC (with a slash through the icon) and not assigned the alias address. I'm not sure how to bring the other NIC back online, as there doesn't appear to be a button for it, and I'm not sure why the alias didn't get assigned. I did get presented with the 'Do you want to test this configuration' menu and did test it, but it didn't work, so I just assumed the test was buggy and ignored it, which, in hindsight, was a bad idea. I can still access the web GUI on my first NIC, though.
After some experimentation (and an upgrade to Bluefin), I was able to bring the second NIC back online (sort of), I just selected the NIC and it brought up a menu for me to set the NIC's gateway IP, which I did, but it's still not showing me an IP for the NIC and if I edit the NIC options and assign an alias IP address it again does not save.

Thank you for any replies.

 

[browntiger]

Go to System / Shell >
And type "ifconfig"
post the results surrounded by using code brackets

 

[ninthturtle1034]

Go to System / Shell >
And type "ifconfig"
post the results surrounded by using code brackets

Okay thanks will do when I get home in a few hours, seems my cloudflare tunnel is down so I can't get to the remotely right now

 

[paddy01]

Speculation without the additional information, but are you attempting to assign IP addresses in the same subnet to both individual NIC's?

If so then that's a no no and Truenas won't let you (the config won't stick).

 

[ninthturtle1034]

Go to System / Shell >
And type "ifconfig"
post the results surrounded by using code brackets

Hi Sorry it's been a few days, Been busy with work and other things and I'm still going to be busy till about this time next week so I will likely have more delayed responses.

Code:

enp3s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fd36:c21f:9677:b4fb:2e0:4cff:feb0:2f06  prefixlen 64  scopeid 0x0<global>
        inet6 fe80::2e0:4cff:feb0:2f06  prefixlen 64  scopeid 0x20<link>
        ether 00:e0:4c:b0:2f:06  txqueuelen 1000  (Ethernet)
        RX packets 768245  bytes 126623316 (120.7 MiB)
        RX errors 0  dropped 234237  overruns 0  frame 0
        TX packets 56  bytes 10763 (10.5 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

enp8s0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 192.168.86.5  netmask 255.255.255.0  broadcast 192.168.86.255
        inet6 fe80::da5e:d3ff:fea0:bc71  prefixlen 64  scopeid 0x20<link>
        inet6 fd36:c21f:9677:b4fb:da5e:d3ff:fea0:bc71  prefixlen 64  scopeid 0x0<global>
        ether d8:5e:d3:a0:bc:71  txqueuelen 1000  (Ethernet)
        RX packets 2309222  bytes 1165309179 (1.0 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 735938  bytes 89278160 (85.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

kube-bridge: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet 172.16.0.1  netmask 255.255.0.0  broadcast 172.16.255.255
        inet6 fe80::687f:bff:fe2b:22e9  prefixlen 64  scopeid 0x20<link>
        ether 5a:86:18:f5:5e:38  txqueuelen 1000  (Ethernet)
        RX packets 2402457  bytes 670903831 (639.8 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 2657698  bytes 597512963 (569.8 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

kube-dummy-if: flags=195<UP,BROADCAST,RUNNING,NOARP>  mtu 1500
        inet 172.17.0.10  netmask 255.255.255.255  broadcast 0.0.0.0
        inet6 fe80::d4a1:b1ff:fe61:e7b8  prefixlen 64  scopeid 0x20<link>
        ether 6a:95:a1:0d:8e:be  txqueuelen 0  (Ethernet)
        RX packets 0  bytes 0 (0.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 91  bytes 19642 (19.1 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

lo: flags=73<UP,LOOPBACK,RUNNING>  mtu 65536
        inet 127.0.0.1  netmask 255.0.0.0
        inet6 ::1  prefixlen 128  scopeid 0x10<host>
        loop  txqueuelen 1000  (Local Loopback)
        RX packets 5104319  bytes 1666427858 (1.5 GiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 5104319  bytes 1666427858 (1.5 GiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth3659962e: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::60fb:46ff:fec6:732d  prefixlen 64  scopeid 0x20<link>
        ether fe:1a:7f:2e:32:f6  txqueuelen 0  (Ethernet)
        RX packets 1  bytes 42 (42.0 B)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 22  bytes 5860 (5.7 KiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth5543ca11: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::4c03:8bff:fee3:9c41  prefixlen 64  scopeid 0x20<link>
        ether 4e:03:8b:e3:9c:41  txqueuelen 0  (Ethernet)
        RX packets 322809  bytes 371931624 (354.7 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 337021  bytes 30001125 (28.6 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth5eaed3d8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::459:68ff:febe:1d16  prefixlen 64  scopeid 0x20<link>
        ether a6:ee:96:a4:c7:09  txqueuelen 0  (Ethernet)
        RX packets 1248871  bytes 251575304 (239.9 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 1435842  bytes 467778849 (446.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth6aeab0e0: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::6467:c1ff:fe88:4357  prefixlen 64  scopeid 0x20<link>
        ether 8e:76:d5:3d:84:d7  txqueuelen 0  (Ethernet)
        RX packets 762589  bytes 72488671 (69.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 829458  bytes 72541116 (69.1 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

veth9389fff8: flags=4163<UP,BROADCAST,RUNNING,MULTICAST>  mtu 1500
        inet6 fe80::4471:37ff:feb1:4d23  prefixlen 64  scopeid 0x20<link>
        ether 46:71:37:b1:4d:23  txqueuelen 0  (Ethernet)
        RX packets 68215  bytes 8546855 (8.1 MiB)
        RX errors 0  dropped 0  overruns 0  frame 0
        TX packets 55440  bytes 27198418 (25.9 MiB)
        TX errors 0  dropped 0 overruns 0  carrier 0  collisions 0

 

[ninthturtle1034]

Speculation without the additional information, but are you attempting to assign IP addresses in the same subnet to both individual NIC's?

If so then that's a no no and Truenas won't let you (the config won't stick).

I wouldn't throw it out as an idea and I likely am without realising it. My end goal is to have one NIC connected to my main network switch and then the second NIC connected to a different switch and then all the devices that need faster uplinks connect to this second switch but atm both NIC's are connected to the same switch. You may be able to get an understanding of my network config from the reply above but essentially my system is currently on 192.168.86.5 on the first NIC (which I think is the motherboard one) with an alias for 192.168.86.12 (so I can still access the web GUI if/when I implement Trafic as this (should) get around me needing to change what port the web GUI is on) and I want the second NIC to broadcast on 192.168.86.11. The thing that confuses me is it was working prior to me setting the alias on the first NIC

 

[paddy01]

Ok so yes it looks like you're trying to set IP address on the individual NIC's which are both in the same IP subnet (i.e. 192.168.86.0/24).

Whilst some OS's will let you do that, it's basically a networking no no and breaks certainly fundamental elements of networking.

Depending on what switches and router you have then if you want different things to connect on TrueNAS on different addresses for different purposes then you'd need to put each interface in a different IP Subnet and use the router to route between those 2 networks.

You can set multiple aliases in the same IP subnet on the same NIC, but not on separate NIC's regardless of whether they're connected to the same or separate switches.

 

[Whattteva]

Ok so yes it looks like you're trying to set IP address on the individual NIC's which are both in the same IP subnet (i.e. 192.168.86.0/24).

Whilst some OS's will let you do that, it's basically a networking no no and breaks certainly fundamental elements of networking.

I don't really see anything wrong with this. I have done this on both Windows and Linux systems all the time, especially on laptops with both a wired NIC and a WiFi NIC. The OS will simply choose a route with a lesser metric number for the route that is faster. In most cases, it usually means the WiFi card will just be unused. I haven't tried this on my FreeBSD laptop since it doesn't have a native wired NIC, but I'm sure it works just the same. Maybe I'll test it out sometime today if I can get off my lazy ass.

 

[danb35]

I don't really see anything wrong with this.

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

 

[Whattteva]

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

That really doesn't say anything wrong with it either. It simply says it doesn't work the way most people think it would work in their minds (which is dedicated link) and is simply not the case due to how traffic is prioritized with the metrics system.

I'm not saying it's an optimal way of doing things especially if you're looking for more bandwidth, just that there's no reason why you can't use that configuration, even if it's quite pointless.

 

[browntiger]

The second nic is not offline, but it has no IPv4 IP.

On the server, you should have Console menu

Select option 1 to configure your Network Interfaces,
it will ask enp3s0 or enp8s0.
=== (Next depends on a version of Truenas)
It will offer you to "DELETE INTERFACE": N
Remove Current Settings: N
Configure Interface for DHCP: n
Configure IPv4: yes
IP 192.168.86.200 <- pick some number
Netmask 255.255.255.0
Configure IPv6 no
Restart network yes.

You can add alias to whatever card you want just make sure to say no DHCP.

 

[paddy01]

That really doesn't say anything wrong with it either. It simply says it doesn't work the way most people think it would work in their minds (which is dedicated link) and is simply not the case due to how traffic is prioritized with the metrics system.

I'm not saying it's an optimal way of doing things especially if you're looking for more bandwidth, just that there's no reason why you can't use that configuration, even if it's quite pointless.

If the traffic is arriving at at system from a client machine in the same IP subnet then the routing table has nothing to do with how the server will respond, it's all handled at layer 2 of the OSI model and based on ARP.

Remember, we're not talking about traffic originating from the TrueNAS server here, but how it will respond to packets arriving from another system/client.

If it's got 2 NIC's with IP's in the same L2 subnet, how's it supposed to know which interface to respond on given that L3 (routing) has nothing to do with that.

And even if routing were in play, if you've got 2 NIC's with IP's in the same subnet:

NIC 1 - 192.168.86.10/24
NIC 2 - 192.168.86.20/24

NIC 1 has a lower metric (therefore preffered) than NIC2.

Packet arrives on 192.168.86.20 (NIC2) but (absent any other factors) the reply will leave on 192.168.86.10 (asymmetric routing)..

 

[Whattteva]

If the traffic is arriving at at system from a client machine in the same IP subnet then the routing table has nothing to do with how the server will respond, it's all handled at layer 2 of the OSI model and based on ARP.

Sure, but the thing about networks is every device has autonomy in how they respond. I mean, just look at routers/firewalls. They just decide arbitrarily to drop packets nilly-willy. Just because a packet arrives, doesn't mean they're obligated to even do anything about it.

Remember, we're not talking about traffic originating from the TrueNAS server here, but how it will respond to packets arriving from another system/client.

If it's got 2 NIC's with IP's in the same L2 subnet, how's it supposed to know which interface to respond on given that L3 (routing) has nothing to do with that.

And even if routing were in play, if you've got 2 NIC's with IP's in the same subnet:

NIC 1 - 192.168.86.10/24
NIC 2 - 192.168.86.20/24

NIC 1 has a lower metric (therefore preffered) than NIC2.

Packet arrives on 192.168.86.20 (NIC2) but (absent any other factors) the reply will leave on 192.168.86.10 (asymmetric routing)..

Again, devices have autonomy. No other machine can tell it what to do. It simply sees that the packet that arrived at L2 but has the same L3 subnet with the other NIC attached to it with lower metric. Obviously, this probably will end up with that packet essentially not being replied to and that session may end up timing out, but all future negotiations will just simply switch to the other NIC.

 

[ninthturtle1034]

Ok so yes it looks like you're trying to set IP address on the individual NIC's which are both in the same IP subnet (i.e. 192.168.86.0/24).

Whilst some OS's will let you do that, it's basically a networking no no and breaks certainly fundamental elements of networking.

Depending on what switches and router you have then if you want different things to connect on TrueNAS on different addresses for different purposes then you'd need to put each interface in a different IP Subnet and use the router to route between those 2 networks.

You can set multiple aliases in the same IP subnet on the same NIC, but not on separate NIC's regardless of whether they're connected to the same or separate switches.

When I put in the second NIC originally, I thought I could have a dedicated NIC for the docker/Kubernetes that Scale has, but then I had the idea that I could link all of my servers together on dedicated 2.5GB NICs/Switches to have faster speeds and to cut out the potential for other network noise and then keep the primary NICs for connecting them to the wider network and my router.
If I did setup that isolated network for the fast speeds then yes I would give it a different IP subnet

I have a Google WiFi mesh setup as my router and a couple AP's so there are not a whole ton of options for configuring the network; I can tell it what IP range to use for the DHCP and subnet but all that really achieves is a higher limit of connected devices as it doesn't care what IP it gives to devices so even if I reserve 192.168.84 (for example) to be just for my servers I would have to input 192.168.0.0 for the DHCP pool and then there's nothing actually preventing a random device like a mobile phone from connecting to that dedicated range meaning I'd have to individually go through and reserve all 255 IP's in the .84 range and I don't have 255 devices to do that for

Okay good to know about that last point although other comments have suggested that's not the case

 

[ninthturtle1034]

I don't really see anything wrong with this. I have done this on both Windows and Linux systems all the time, especially on laptops with both a wired NIC and a WiFi NIC. The OS will simply choose a route with a lesser metric number for the route that is faster. In most cases, it usually means the WiFi card will just be unused. I haven't tried this on my FreeBSD laptop since it doesn't have a native wired NIC, but I'm sure it works just the same. Maybe I'll test it out sometime today if I can get off my lazy ass.

Well I'm running the Scale version (Debian based) instead of the Core version (FreeBDSD based) so it should be possible to do as it's almost pue Linux

 

[ninthturtle1034]

That really doesn't say anything wrong with it either. It simply says it doesn't work the way most people think it would work in their minds (which is dedicated link) and is simply not the case due to how traffic is prioritized with the metrics system.

I'm not saying it's an optimal way of doing things especially if you're looking for more bandwidth, just that there's no reason why you can't use that configuration, even if it's quite pointless.

If I was looking for more bandwidth although that is not currently the plan) I Could just create a virtual link aggregation NIC and link both NICs to it, right?

 

[ninthturtle1034]

The second nic is not offline, but it has no IPv4 IP.

On the server, you should have Console menu

Select option 1 to configure your Network Interfaces,
it will ask enp3s0 or enp8s0.
=== (Next depends on a version of Truenas)
It will offer you to "DELETE INTERFACE": N
Remove Current Settings: N
Configure Interface for DHCP: n
Configure IPv4: yes
IP 192.168.86.200 <- pick some number
Netmask 255.255.255.0
Configure IPv6 no
Restart network yes.

You can add alias to whatever card you want just make sure to say no DHCP.

So when I configure that IP I could put it on the same subnet right i/e have both NIC's on 192.168.86.0, so long as DHCP is disabled for both NICS

 

[ninthturtle1034]

If the traffic is arriving at at system from a client machine in the same IP subnet then the routing table has nothing to do with how the server will respond, it's all handled at layer 2 of the OSI model and based on ARP.

Remember, we're not talking about traffic originating from the TrueNAS server here, but how it will respond to packets arriving from another system/client.

If it's got 2 NIC's with IP's in the same L2 subnet, how's it supposed to know which interface to respond on given that L3 (routing) has nothing to do with that.

And even if routing were in play, if you've got 2 NIC's with IP's in the same subnet:

NIC 1 - 192.168.86.10/24
NIC 2 - 192.168.86.20/24

NIC 1 has a lower metric (therefore preffered) than NIC2.

Packet arrives on 192.168.86.20 (NIC2) but (absent any other factors) the reply will leave on 192.168.86.10 (asymmetric routing)..

Thank you for this explanation, A virtual Link Aggregation device would get around that and act as a sort of load balancer right? That's not really what I'm looking to do I don't think but I'm just checking I'm sort of understanding.

 

[Whattteva]

If I was looking for more bandwidth although that is not currently the plan) I Could just create a virtual link aggregation NIC and link both NICs to it, right?

Yes, however this will require a managed switch though. I'm not sure if you mentioned what kind of switch you have in your previous posts.
Also, be aware that this will only increase bandwidth on one end of the connection (the aggregated end). So if you want full speed with 1 client, the client will also have to be able to handle that.

 

[ninthturtle1034]

Yes, however this will require a managed switch though. I'm not sure if you mentioned what kind of switch you have in your previous posts.
Also, be aware that this will only increase bandwidth on one end of the connection (the aggregated end). So if you want full speed with 1 client, the client will also have to be able to handle that.

I have a zyxcel 8 port managed switch and it does support link aggregation, but I was refering more to TrueNAS scale as I saw it had a link aggregation option when I was poking around trying to get the alias IP and second NIC working.
Edit: Actually just noticed the TrueNAS system is plugged into the unmaged tp-link switch so uhhh.... might need to move that around at some point