SOLVED Network - replace switch with internal NIC & routing - howto?

[flashdrive]

tldr:

solution / proper howto:

SOLVED - Network - replace switch with internal NIC & routing - howto?

tldr: solution / proper howto: https://www.truenas.com/community/threads/network-replace-switch-with-internal-nic-routing-howto.96040/post-664230 Hello, I want to remove a LAN switch out of the setup. Situation: TrueNAS host: 1 onboard NIC with 1x LAN port 1 PCIe NIC with 2x LAN port ->...

www.truenas.com

Hello,

I want to remove a LAN switch out of the setup.

Situation:

TrueNAS host:
1 onboard NIC with 1x LAN port
1 PCIe NIC with 2x LAN port -> connected to Switch 1 and Switch 2

client PC:
1 onboard NIC with 1x LAN port
1 PCIe NIC with 2x LAN port -> connected to Switch 2



goal:

Only TrueNAS host will connect to 1 Switch with WAN connection.

client PC will connect to TrueNAS host and use the WAN connection from there.

How would I set this up in TrueNAS?

Chapter 34. Advanced Networking

Advanced networking in FreeBSD: basics of gateways and routes, CARP, how to configure multiple VLANs on FreeBSD, etc

docs.freebsd.org

 

[Samuel Tai]

This is not supported. TrueNAS isn't meant to be a switch or a router.

 

[flashdrive]

@Samuel Tai
Thank you for your very fast answer.

Even if this is not supported via GUI - is there no way to set this up in FreeBSD fashion?

 

[Patrick M. Hausen]

Well you can create a bridge across all 3 LAN ports and use the TrueNAS as a small switch. Depending on your rooms and the cabling that might solve a particular constraint somebody experiences. And it can be done in the UI and is supported, IMHO.

Using it as a router is a much more complex matter altogether. So if it's all just about the cabling and you stick to a single IP network, "bridge" is the keyword.

Be aware that the performance might be worse than without the bridge. Which will be fixed in the next major release, probably, but not sooner. Major release meaning something starting with "13." ...

 

[flashdrive]

@Patrick M. Hausen

Thank you for your insight.

Yes, it will be a single IP range network.

All I would need is the switch functionality in TrueNAS itself. I will have a look into the "bridge" option:

Chapter 34. Advanced Networking

Advanced networking in FreeBSD: basics of gateways and routes, CARP, how to configure multiple VLANs on FreeBSD, etc

docs.freebsd.org

 

[Patrick M. Hausen]

The most important things:

• disable hardware offloading in the physical interface settings for all member interfaces
• don't put any IP address on any member interface - put the IP address(es) on the bridge interface instead
• name the bridge interface "bridge0"
• if you use jails and/or VMs please come back for advice on network settings

Edit: and no, don't go for the FreeBSD handbook and try to edit config files or set tunables. It's all in the UI.

Network > Interfaces > Add --> type "Bridge", ... etc.

 

[ChrisRJ]

The primary reason for me to not do this is security. @flashdrive , can you elaborate why you want to do this?

 

[Patrick M. Hausen]

@ChrisRJ What difference does it make if you turn your NAS into a layer 2 bridge which is connected to the same internal network with the same IP configuration? In the ongoing discussion we already ruled out turning it into an internet facing router.

I have run FreeNAS/TrueNAS as a "free" 4-port bridge for years so save on separate devices.

Maybe your TrueNAS is in the same room as your desktop system and you have only one network cable leading somewhere else to your ISP router two stories down ... or some such.

 

[flashdrive]

@ChrisRJ
@Patrick M. Hausen

The reason for me to try this approach is indeed to get rid of 1 switch (energy consumption is not a real issue with that one).

The TrueNAS host will run 24/7 anyway.

"Maybe your TrueNAS is in the same room as your desktop system and you have only one network cable leading somewhere else to your ISP router two stories down ... or some such."

> jackpot - with the current solution being"throw money at the problem, run 2 switches"

 

[flashdrive]

more to read

7. Network — TrueNAS®11.3-U5 User Guide Table of Contents

www.ixsystems.com

/core/network/interfaces/

Bridge between network interfaces

Hi, I have a Freenas system with 4 x 10GB NIC's that are connected point to point to to 2 servers, each server has 2 x 10GB NIC's. Reason for this is that I don't have a 10GB switch at the moment. The two servers are VMware hosts and the Freenas acts as a SAN via iSCSI protocol over the 10GB...

www.truenas.com

 

[Patrick M. Hausen]

Oh no! Don't use that old thread from 2016 even if I contributed to it. All of the setup can easily be done in the UI. I repeat: everything necessary can be done in the UI. Implying it is perfectly well supported.

• Edit each of your physical interfaces and disable hardware offloading.
• Save ("Test" on the button) and apply ("Save") on the button.
• Create a bridge interface named bridge0 and make all three physical interfaces a member of it.
• Save/test and apply/save.
• Remove the IP address from your first interface. Make sure hardware offloading stays disabled.
• Add the same IP address to the bridge interface.
• Save/test and apply/save.

Done. If you don't run any jails or VMs at the moment.

 

[flashdrive]

@Patrick M. Hausen

I got your point and thanks to your original valueable hints it is up and running, albeit I am only running the 2 NIC ports in bridged mode as of now. But it works according to my goal !

2016 the non GUI approach was the way to go, no need to be ashamed for that old guide

Thank you again for taking the time.

I did read it until the very end, the hint from 2020 was a good point:

Bridge between network interfaces

Yes 10GB Links for iSCSI to VMware. I setup the Bridge because I was too cheap to buy a Switch, this allowed me to hook 2 separate VMware hosts to Freenas Via 10GB links and keep them on same 10GB Subnet. My pool is all SSD drives for VMWare my vm's boot up in seconds. combination of Chelsio and...

www.truenas.com

"I took the plunge and updated to 11.3. Took out my bridge script and relied on the new bridge mechanism to set it up. I had to avoid getting the 1Gb port that was on DHCP to not fight with the bridge. I want the bridge on my main network 192.168.0.x and the 1Gb port was already there. The system set them both on the 192,168.0.x subnet so I could not access the GUI after I set it up. I used the commend console to change the address of the 1Gb port to a different subnet and I was off sailing."

 

[flashdrive]

There are a couple more questions which come to mind:

1)
"disable hardware offloading"
Is there a specific reason why this is necessary?

2)
What would I have to edit in order to run jails?

3)

bridge(4)

www.freebsd.org

BUGS
The if_bridge driver currently supports only Ethernet and Ethernet-like
(e.g., 802.11) network devices, with exactly the same interface MTU size
as the bridge device.

> So the whole network needs to run on the same MTU size? I can therefor not select Jumbo frames on a per client base?


4)
Rapid Spanning Tree Protocol

Does one have to enable / care about this in TrueNAS setup?

 

[Patrick M. Hausen]

"disable hardware offloading"
Is there a specific reason why this is necessary?

Hardware offloading of e.g. TCP checksums only makes sense if the TrueNAS host is the destination of that particular TCP stream.
Currently the driver is not clever enough to use optimized hardware supported processing for local packets while not doing the same to packets destined for another host. I don't know enough to give you a well-founded judgement but my gut feeling says, that this is more or less impossible.
Summary: the system providing the bridge should not mess with packets that only pass through it.

What would I have to edit in order to run jails?

You need to set some specific values for the jail, not configure anything else differently. And these specific settings are not the TrueNAS default. Again they are valid and perfectly supported, you just need to be aware that there are at least two specific settings you need to change when creating a jail.

• vnet_default_interface: "none" instead of "auto"
• interfaces: "vnet0:bridge0" instead of whatever you might find there

> So the whole network needs to run on the same MTU size? I can therefor not select Jumbo frames on a per client base?

Yes. Jumbo frames are set on a per-interface basis anyway. So if you ever connect a hypervisor host for iSCSI - which is about the only scenario where you would benefit from jumbo frames - best use a dedicated interface just for that.

Rapid Spanning Tree Protocol
Does one have to enable / care about this in TrueNAS setup?

Only if you intend to connect more switches, possibly creating redundant paths (loops). A loop in your bridging/switching topology will bring down your network unless the bridges/switches can figure it out and disable at least one link on their own. This is achieved by STP.

If your TrueNAS is at the end of a connection to your switch and you only want to bridge to your PC and a printer or similar - don't worry about it.

HTH,
Patrick

 

[flashdrive]

SOLVED - Network - replace switch with internal NIC & routing - howto?

tldr: solution / proper howto: https://www.truenas.com/community/threads/network-replace-switch-with-internal-nic-routing-howto.96040/post-664230 Hello, I want to remove a LAN switch out of the setup. Situation: TrueNAS host: 1 onboard NIC with 1x LAN port 1 PCIe NIC with 2x LAN port ->...

www.truenas.com

> This worked great after a reboot of TrueNAS

Initially I got this error message when I wanted to test and save the changes made after

• Create a bridge interface named bridge0 and make all three physical interfaces a member of it.
• Save/test and apply/save.

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 138, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1213, in _call
return await methodobj(*prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 975, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 757, in commit
await self.sync()
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/network.py", line 1843, in sync
await self.middleware.call('interface.bridge_setup', bridge)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1256, in call
return await self._call(
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1224, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1128, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/interface/bridge.py", line 57, in bridge_setup
iface.add_member(member)
File "netif.pyx", line 1211, in netif.BridgeInterface.add_member
File "netif.pyx", line 1229, in netif.BridgeInterface.bridge_cmd
OSError: [Errno 22] Invalid argument

 

[Patrick M. Hausen]

OK, difficult to diagnose with only that information. You have it up and running now or not?

 

[flashdrive]

Ok, I got it up and running with all 3 NIC interfaces in bridge0.

I do not use DHCP, only 1 IP address for the bridge.

For TrueNAS host internet connection access is not possible this way. Which is what I generally want.

However:

When I want to provide internet access to the TN host, how would I do so from within the bridge? I tried to use DHCP, which worked. But this is not ideal since the LAN connection will be shortly disabled.

Is there another way to set this up?

I guess no, because in my LAN setup only 1 cable from TN host can connect to Switch 1 with the WAN Connection.

My initial idea was to set up the onboard NIC for Internet access, and disconnect this on demand either by disabling this interface or disconnecting the LAN cable.


edit:

I have figured it out myself:

TN
/ui/network/configuration

enter Nameserver and Default Gateway...

 

[Patrick M. Hausen]

You need to set up the default gateway and the DNS server in Network > Global Configuration. This is always necessary with a static IP address, independent of bridge vs. physical.

 

[flashdrive]

@Patrick M. Hausen

Sir, let me at least buy you a coffee for your ongoing support.

You have saved me from a lot of trial and error.

To wrap it up in terms of performance:

• There might be a network throughput penalty due to the bridge setup > to be tested
• Jumbo Frames: if the bridge is set to 9014 all the other devices / interfaces in that same network also do have to be set up to 9014?

 

[Patrick M. Hausen]

There might be a network throughput penalty due to the bridge setup > to be tested

Depends on your hardware. As of FreeBSD 12 the bridge interface runs on only a single core. Which might be a bottleneck or might not.
In FreeBSD 13 this subsystem is completely reworked by Kristof Provost sponsored by the FreeBSD Foundation to use all cores available.

Jumbo Frames: if the bridge is set to 9014 all the other devices / interfaces in that same network also do have to be set up to 9014?

Of course. If the interface of your TrueNAS is set to an MTU that size, the TrueNAS will send out frames that size if there is enough data to fill them. If the receiving system does not support jumbo frames, it will not get these. All participating systems have to agree on a maximum frame size, always.