Hello,
I posted the below message on OPNsense forum but I didn't get any replies. I hope someone here may shed some light on the issue I have.
At the bottom of this message is the network diagram of my home network. Some details which may help in identifying a solution to the problem I am experiencing:
- the focus is on "opnsense" - "switch" - "truenas"
- the VLANs and interfaces of interest are:
VLAN 1 (MGMT) tagged on em0 (opnsense), port 10 (switch), lagg0 (truenas), BMC (truenas)
VLAN 2 (USERS) tagged on same ports as VLAN 1 less BMC, and untagged on port 6 (switch)
- truenas: VLAN 1 (MGMT) IP address:
192.168.1.2 tagged on BMC (IPMI interface)
192.168.1.3 tagged on lagg0 (igb0 and igb1 LACP link aggregation)
- truenas: VLAN 2 (USERS) IP address:
192.168.2.3 tagged on lagg0 (igb0 and igb1 link aggregation)
- opnsense: VLAN 1 (MGMT) IP address: 192.168.1.1 tagged on em0
- opnsense: VLAN 2 (USERS) IP address: 192.168.2.1 tagged on em0
- rpi4: IP address: 192.168.2.21
- switch: VLAN 1 (MGMT) IP address: 192.168.1.4 tagged on ports 10, Link Aggregation 1 (ports 3 and 4, LACP), and 5
- switch: VLAN 2 (USERS) tagged on ports 10 and Link Aggregation 1, and untagged on port 6
The only firewall rules configured on opnsense are:
[MGMT] Pass | Protocol IPV4 * | Source: MGMT Net | Source Port * | Destination: * | Dest. Port * | Gateway * | Description: Allow all
[USERS] Pass | Protocol IPV4 * | Source: USERS Net | Source Port * | Destination: * | Dest. Port * | Gateway * | Description: Allow all
The problem:
- access the web UI of truenas from rpi4 web browser on 192.168.1.3 and
- truenas SSH access from rpi4: $ ssh root@192.168.1.3
HTTP / HTTPS connection drops after less then a minute, then restore, drops again and so on; SSH connection drops and, obviously, doesn't restore without me entering the command again.
I do not experience this issues when using the 192.168.2.3 IP address. Even more: no lost connectivity when assigning a static IP to rpi4 in VLAN 1 [MGMT] (ex. 192.168.1.10). So everything works fine when both truenas and rpi4 are in the same network.
And, no issues when accessing the web UI for the IPMI interface on 192.168.1.2 (VLAN 1) from rpi4 with an IP address on VLAN 2 - so this time, no inter-VLAN routing issues.
Would this be an opnsense routing issue or truenas link aggregation one? The next step in troubleshooting will be to "break" the link aggregation and see if the problem persists when using a standard link, but I would like to have the community's feedback first. Just to add that everything was working fine when I had another VLAN configured on all devices (VLAN 3), but then I decided to get rid of it and simplify the design by bringing those devices in VLAN 2.
Your input will be appreciated.
I posted the below message on OPNsense forum but I didn't get any replies. I hope someone here may shed some light on the issue I have.
At the bottom of this message is the network diagram of my home network. Some details which may help in identifying a solution to the problem I am experiencing:
- the focus is on "opnsense" - "switch" - "truenas"
- the VLANs and interfaces of interest are:
VLAN 1 (MGMT) tagged on em0 (opnsense), port 10 (switch), lagg0 (truenas), BMC (truenas)
VLAN 2 (USERS) tagged on same ports as VLAN 1 less BMC, and untagged on port 6 (switch)
- truenas: VLAN 1 (MGMT) IP address:
192.168.1.2 tagged on BMC (IPMI interface)
192.168.1.3 tagged on lagg0 (igb0 and igb1 LACP link aggregation)
- truenas: VLAN 2 (USERS) IP address:
192.168.2.3 tagged on lagg0 (igb0 and igb1 link aggregation)
- opnsense: VLAN 1 (MGMT) IP address: 192.168.1.1 tagged on em0
- opnsense: VLAN 2 (USERS) IP address: 192.168.2.1 tagged on em0
- rpi4: IP address: 192.168.2.21
- switch: VLAN 1 (MGMT) IP address: 192.168.1.4 tagged on ports 10, Link Aggregation 1 (ports 3 and 4, LACP), and 5
- switch: VLAN 2 (USERS) tagged on ports 10 and Link Aggregation 1, and untagged on port 6
The only firewall rules configured on opnsense are:
[MGMT] Pass | Protocol IPV4 * | Source: MGMT Net | Source Port * | Destination: * | Dest. Port * | Gateway * | Description: Allow all
[USERS] Pass | Protocol IPV4 * | Source: USERS Net | Source Port * | Destination: * | Dest. Port * | Gateway * | Description: Allow all
The problem:
- access the web UI of truenas from rpi4 web browser on 192.168.1.3 and
- truenas SSH access from rpi4: $ ssh root@192.168.1.3
HTTP / HTTPS connection drops after less then a minute, then restore, drops again and so on; SSH connection drops and, obviously, doesn't restore without me entering the command again.
I do not experience this issues when using the 192.168.2.3 IP address. Even more: no lost connectivity when assigning a static IP to rpi4 in VLAN 1 [MGMT] (ex. 192.168.1.10). So everything works fine when both truenas and rpi4 are in the same network.
And, no issues when accessing the web UI for the IPMI interface on 192.168.1.2 (VLAN 1) from rpi4 with an IP address on VLAN 2 - so this time, no inter-VLAN routing issues.
Would this be an opnsense routing issue or truenas link aggregation one? The next step in troubleshooting will be to "break" the link aggregation and see if the problem persists when using a standard link, but I would like to have the community's feedback first. Just to add that everything was working fine when I had another VLAN configured on all devices (VLAN 3), but then I decided to get rid of it and simplify the design by bringing those devices in VLAN 2.
Your input will be appreciated.
