Juggling multiple NICs

[romanpj]

Searched and read round the various posts but still not sure.

Got a couple of 1GB NICs built-in to the mobo (X10SL7-F). Decided a couple of years ago to migrate over to 10GB networking (at least between my virtual host and the 2 TrueNAS boxes) when the CRS305-1G-4S+ appeared. So all 3 boxes have 10GB NIC's.

Normally only one of the 10GB interfaces are configured with an IP. But occasionally I might want to power off a machine and move from my data centre (the loft ) to my work bench to work on it where there is only 1GB provision.

So given that only one NIC will be physically connected at a time what would be the recommended configuration?

1. Set same IP on 2 of the NIC's? (Convenient - links to web GUI stay same.)

2. Set different IP's (same subnet) on the 2 required NIC's?

3. Some other config?

Thanks,

P.

 

[danb35]

So given that only one NIC will be physically connected at a time what would be the recommended configuration?

Leave them (ordinarily) both plugged in, and set up failover--which I'm certain I saw a thread by @jgreco on how to do it, but I now can't find it. But it's a form of link aggregation, so poking around there may find it for you. You'll need to set it up on both the NAS and the switch.

 

[Patrick M. Hausen]

Configure a bridge interface with both NICs as members. Move IP address configuration to bridge interface. Configure the phys. NICs only "up" in the options field and disable hardware offloading.

Continues to work as it always did but you can plug in any port you like.

Careful: never plug in two ports. That will bring down your network.

 

[romanpj]

Configure a bridge interface with both NICs as members. Move IP address configuration to bridge interface. Configure the phys. NICs only "up" in the options field and disable hardware offloading.

Continues to work as it always did but you can plug in any port you like.

Careful: never plug in two ports. That will bring down your network.

The network 192.168.7.0/24 is already in use by another interface.

 

[Patrick M. Hausen]

Remove IP address from NIC first, then assign to bridge.

 

[romanpj]

Remove IP address from NIC first, then assign to bridge.

OK, I seem to be missing something that is thought of as obvious so its not mentioned in documentation or the various discussions.

[On web GUI]
I remove static IP from igb0 by 'reset configuration' button.
Result = DHCP becomes enabled on all interfaces and currently connected interface gets an address.
Only way to continue config is to connect to new IP.
Press ahead and create bridge with static IP (type, add members, set IP). Click apply.
Result = DHCP set to 'no' on all interfaces. Bridge listed with required IP:-

When 'Test Changes' clicked (and confirmed) I get this:-

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 139, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1235, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 842, in commit
await self.sync()
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 1933, in sync
await self.middleware.call('interface.bridge_setup', bridge)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1278, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1246, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1151, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 57, in bridge_setup
iface.add_member(member)
File "netif.pyx", line 1211, in netif.BridgeInterface.add_member
File "netif.pyx", line 1229, in netif.BridgeInterface.bridge_cmd
OSError: [Errno 16] Device busy

A second go at testing changes appears to work and GUI shows desired config.
Then I save changes.
But at this point I would expect to lose connection to GUI as bridged IP becomes active.
A refresh of browser window displays "Network / Interfaces" with bridge gone and all interfaces set to DHCP.

I did look at the local console (because the machine is still on the bench with a monitor available). Doesn't seem to be an option to create a bridge from that interface (unless I have to use the shell).

FYI
Version: TrueNAS-13.0-U1

Now that I think of it I struggled to configure my other FreeNAS machine when I added the 10GB NIC and only got it working more by luck than design. So I'd really like to understand where I'm going wrong.

Thanks,

P.

 

[Patrick M. Hausen]

You do not need to reset the configuration. Simply remove the IP clicking into the field and using the backspace key or triple click and Ctrl-X or whatever. Just empty the IP address field and click "apply". Before applying check the "disable hardware offloading" box and put "up" into the options field.

 

[romanpj]

You do not need to reset the configuration. Simply remove the IP clicking into the field and using the backspace key or triple click and Ctrl-X or whatever. Just empty the IP address field and click "apply". Before applying check the "disable hardware offloading" box and put "up" into the options field.

Its still not playing ball.

1. Edit igb0 to remove IP.
2. Tick "disable hardware offloading" box.
3. Add "up" to Options field.
4. Click "Apply".
Result:-

Should I test changes?
or
Modify sgfxge0 with "disable hardware offloading" and "up" settings?
or
Go ahead and add bridge?

5. Modified sgfxge0 with "disable hardware offloading" and "up" settings but no IP (I assume it needs to be consistent with other interface).
6. Created Bridge:-

And clicked Apply.

[BTW if I try to set "disable hardware offloading" in the Bridge settings I get:-

So I assume that is only set on physical device.]

Result:-

7. Click Test Changes
Result:-

8. Try Test Changes a second time. Appears to work.
9. Click to make changes permanent.

Result:-

At this point I would expect to lose connection to the web page as the address should have changed (that's how its worked with lots of other devices that I've managed thru a web interface where I change the device IP).
If I open another tab to the bridge IP it times out.
If I refresh the last screen above I get prompted to log back in to TrueNAS and Network/Interfaces show this:-

Where are my changes? Where am I going wrong?

P.

 

[Patrick M. Hausen]

You possibly already have a bridge interface and igb0 is a member of that. Are you running any jails and/or VMs? In that case TrueNAS auto-creates a bridge for you that is not visible in the UI. Yes, that sucks. And they create it in an unsupported manner.

Could you post the output of ifconfig, please?

 

[romanpj]

You possibly already have a bridge interface and igb0 is a member of that. Are you running any jails and/or VMs? In that case TrueNAS auto-creates a bridge for you that is not visible in the UI. Yes, that sucks. And they create it in an unsupported manner.

Could you post the output of ifconf [/QUOTE]

No jails or VMs that I know of (maybe something left over from previous fiddling).

Here is ifconfig:-

root@freenasN54L-01[~]# ifconfig
sfxge0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ec07ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0f:53:26:47:10
media: Ethernet autoselect
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>
sfxge1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=ec07ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
ether 00:0f:53:26:47:11
media: Ethernet autoselect
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>
igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4e503bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 0c:c4:7a:6c:82:40
inet 192.168.7.41 netmask 0xffffff00 broadcast 192.168.7.255
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=4e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
ether 0c:c4:7a:6c:82:41
media: Ethernet autoselect
status: no carrier
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x5
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
groups: pflog
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 58:9c:fc:10:ff:c5
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 3 priority 128 path cost 2000000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>
root@freenasN54L-01[~]#​

I think I captured it all. C'n'P is much easier in Powershell.

Dunno where that bridge0 is from!

Thanks for your patience.

P.

 

[Patrick M. Hausen]

I think I captured it all. C'n'P is much easier in Powershell

What SSH client are you using? The shell in the web UI is useless.

Could you enclose the output in CODE tags, please? It's really hard to decipher at the moment.

That bridge0 needs to go. I'm about to go to sleep, back tomorrow.

 

[romanpj]

What SSH client are you using? The shell in the web UI is useless.

Could you enclose the output in CODE tags, please? It's really hard to decipher at the moment.

That bridge0 needs to go. I'm about to go to sleep, back tomorrow.

Thank you, thank you, thank you Patrick.
I lost too much of my weekend to this problem.

Don't have a regular SSH client. You're right about the web UI.

Already taken care of the unwanted bridge:-

ifconfig bridge0 destroy

Required bridge setup and working.

Thanks again,

P.