Trouble with multiple NICs for access from different VLANs/Subnets

[felixthecat]

Hi everyone,

I'm trying to set up my TrueNAS Scale Box to be reachable for Network Shares from two separate VLANs/Subnets:
- 10.0.20.0/24
- 10.0.40.0/24

My Setup so far was as follows:
- Gigabit NIC 1: Static IP 10.0.20.21/24, used for Shares and Management
- Gigabit NIC 2: Member of br1, both no IP, Containers and VMs are connected to it.
- Default Gateway and DNS: 10.0.20.1 (my Router)

Now I'm trying to add a 10G NIC on the second VLAN/Subnet, to have 10Gbit Networking Speed for Devices on that Network.
Eventually I'd like to only use the Gigabit NIC for Management, while using the other one for Network Shares.

I tried to add the following Configuration:
- 10G NIC1: Static IP 10.0.40.21/24


But while it seems to work fine at first, issues start shortly afterwards:
- When accessing the GUI while the new configuration is set, it seems to hang every 20 seconds or so and after a while reloads itself. This Process infinitely repeats itself.
- The Containers/VMs on the Bridge don't have a Network Connection anymore
- Sometimes Network Shares don't work properly either

I've tried to do Traceroute from both TrueNAS and the Client on the 40 Subnet and have looked at Routing Tables on Router and Switch to make sure there's nothing weird going on from the Networking side of things.


I also check Routes with "ip route" and it seemed that 10.0.40.21 was correctly configured as the Gateway for the 10.0.40.0/24 Subnet by TrueNAS.

I'd highly appreciate any tips and feedback on this.

 

[sretalla]

What hardware are you using? it sounds like your driver may be dumping.

 

[felixthecat]

What hardware are you using? it sounds like your driver may be dumping.

Thanks for the quick reply!

The two NICs on the Mainboard are Intel i350 and the 10G one is an Intel X540-T2.
So I'd assume that the Hardware shouldn't be the problem.

 

[sretalla]

Maybe it's a fake? https://www.truenas.com/community/threads/10gb-card-truenas-macos-hackintosh.90643/post-631535

The forum rules (and common sense) require that you share hardware details so we can help you better... up to you.

 

[felixthecat]

Maybe it's a fake? https://www.truenas.com/community/threads/10gb-card-truenas-macos-hackintosh.90643/post-631535

The forum rules (and common sense) require that you share hardware details so we can help you better... up to you.

Sorry for not posting more hardware details, I'd assumed it was a configuration problem, since it works fine, just not in TrueNAS.

The System consists of:
- Supermicro X10srm-f
- Xeon E5 2620 V4
- 4x16GB ECC RDIMMs
- Intel X540-T2
- NVME SSDs and SATA HDDs for tiered Storage


About whether it's a fake...I don't think so for the following three reasons:
- It comes from a well-known reseller of used server hardware
- I verified the Yottamark online
- iperf3 shows symmetrical 10Gbit Performance consistenly and there's no issues when not using TrueNAS.

 

[sretalla]

- I verified the Yottamark online

Seems fine then.

Are you disabling hardware offload on the NICs? (apparently not good to have it when bridging them)


Since you mention it's fine with another OS, these shouldn't have any impact, but maybe there's something there to work from:

If you look at the schematic on page 18 here: https://www.supermicro.com/manuals/motherboard/C600/MNL-1845.pdf

It seems there's some strange stuff mentioned about shared LAN on port 1... also the IPMI/BMC seems to have hooks into both lan ports (maybe for some kind of boot options... maybe makes sense to disable those).

The manual seems to indicate some kind of troubleshooting steps which include checking power supply adequacy and heat conditions... just another shot.

 

[felixthecat]

Seems fine then.

Are you disabling hardware offload on the NICs? (apparently not good to have it when bridging them)


Since you mention it's fine with another OS, these shouldn't have any impact, but maybe there's something there to work from:

If you look at the schematic on page 18 here: https://www.supermicro.com/manuals/motherboard/C600/MNL-1845.pdf

It seems there's some strange stuff mentioned about shared LAN on port 1... also the IPMI/BMC seems to have hooks into both lan ports (maybe for some kind of boot options... maybe makes sense to disable those).

The manual seems to indicate some kind of troubleshooting steps which include checking power supply adequacy and heat conditions... just another shot.

The 10Gbit NIC is an Add-in-Card (my board is the -f version, not the -tf which has one on-board), just for clarification.

My IPMI/BMC uses a dedicated port as it's currently set up.
The Onboard Ports, as mentioned, are the 2 Gigabit ones I'm successfully using.

I don't really think it's a Hardware issue since the system is completely fine otherwise.

There's two more things I just noticed:
- It seems that I can still ping / traceroute / iperf while the TrueNAS Scale GUI goes unreachable.
- As soon as I apply the Network Settings, the Bridge that has Containers and VMs attached also goes offline for a bit - is this expected behavior?

I attached a few Screenshots from the Remote Console below (note I also tried via the UI, it just seems easier to show the settings this way).

Edit: eno1 and eno2 are the onboard 1Gbit NICs, ens4f0 and ens4f1 are the 10G NICs from the Add-in-Card. Currently only one of them (f1) is connected to the switch.

1. Configure Network Interfaces

-> Note that the Bridge doesn't have an IP, but Member eno2. Only Members of the Bridge (virtual interfaces for VMs, CTs) have one.

2. Configure Network Settins

3. Output of "ip route"

 

[sretalla]

As soon as I apply the Network Settings, the Bridge that has Containers and VMs attached also goes offline for a bit - is this expected behavior?

At least a short cut of network is expected on settings application, so yes.

Note that the Bridge doesn't have an IP, but Member eno2. Only Members of the Bridge (virtual interfaces for VMs, CTs) have one.

I thought you had said you assigned the IP to the bridge... (which you should).

Perhaps an indication that you're not making it through to save the tested changes?

 

[sretalla]

As soon as I apply the Network Settings, the Bridge that has Containers and VMs attached also goes offline for a bit

Wait... Are you making network changes with apps and VMs still running?

You should stop all VMs and Apps before doing that.

 

[felixthecat]

I thought you had said you assigned the IP to the bridge... (which you should).

Perhaps an indication that you're not making it through to save the tested changes?

The Bridge isn't used to access the Host. It's only there to bridge inbetween my Network and the Containers IP Aliases. This is usual practice in Linux and had worked fine for me with TrueNAS

Definitely making it through to save the changes since I tried from the TrueNAS Shell and confirmed that they'd been set through the Linux Shell.

As I'd noted the following works fine. Unless it impacts the 10Gbit NIC somehow I'm not concerned about it:

My Setup so far was as follows:
- Gigabit NIC 1: Static IP 10.0.20.21/24, used for Shares and Management
- Gigabit NIC 2: Member of br1, both no IP, Containers and VMs are connected to it.

-> Gigabit NIC 1 and 2 are eno1 and eno2 respectively

What I'm trying to do now is to add another NIC (ens4f1 @ X540-T2 Card) on a different Subnet/VLAN so that the Clients there have access to TrueNAS without going through the Router to traverse VLANs.

You should stop all VMs and Apps before doing that.

There's only custom Containers via "Launch Docker" running which have their own IP Alias on the Bridge.
I've tried to shut all down when making the change and even restarting the system, it didn't help.

 

[xzibit]

The Bridge isn't used to access the Host. It's only there to bridge inbetween my Network and the Containers IP Aliases. This is usual practice in Linux and had worked fine for me with TrueNAS

Definitely making it through to save the changes since I tried from the TrueNAS Shell and confirmed that they'd been set through the Linux Shell.

As I'd noted the following works fine. Unless it impacts the 10Gbit NIC somehow I'm not concerned about it:

-> Gigabit NIC 1 and 2 are eno1 and eno2 respectively

What I'm trying to do now is to add another NIC (ens4f1 @ X540-T2 Card) on a different Subnet/VLAN so that the Clients there have access to TrueNAS without going through the Router to traverse VLANs.


There's only custom Containers via "Launch Docker" running which have their own IP Alias on the Bridge.
I've tried to shut all down when making the change and even restarting the system, it didn't help.

Did you ever resolve this issue? I'm trying to do a similar setup and nothing seems to work.