Can someone tell me what I am doing wrong regarding Bridges

[Redemption]

Hi all, I know that Truenas natively does not allow VMs to access applications on the host. To get access to the host operating system, one must create a bridge. I have created my bridges and assigned VLANs to the bridges. I have Truenas on Vlan 10 and assigned dedicated ip address. I installed an AD server on vlan100. The ad server is able to reach the internet and other computers on the network. I can resolve dns request except when it comes to applications on the host OS. I am not able to log into Truenas from the AD server. What am I doing wrong?

 

[browntiger]

> Hi all, I know that Truenas natively does not allow VMs to access applications on the host.
Pretty sure it is a bad practice and they want us to use Kubernetes, not the VMs. But I can see how sometimes it is needed.

>To get access to the host operating system, one must create a bridge.
To network access the host, bridge is needed. VM can access any other network resources (Internet).

>I have created my bridges and assigned VLANs to the bridges. What am I doing wrong?
Do not have a clue, sorry. Not sure what you posted. Not enough to even guess.

Kindly, show us ifconfig

 

[Patrick M. Hausen]

Physical Interface --> VLAN(s) --> Bridge(s)

You need one bridge per VLAN if necessary, not multiple VLANs on a single bridge. Also the IP address (if present) goes on the bridge interface, not the physical or the VLAN.

 

[Redemption]

Thanks for the responses. Patrick, I believe I have it configured per your instructions. If I enable DHCP on the br100, the system goes haywire and does not come back up after reboot. If I configure an alias via GUI, I am unable to save changes. If Ido not set DHCP or alias on br100my domain controller can reach the internet and other regular computers; however, it will not resolve any of the truenas applications.

Code:

2: enp23s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc mq state UP group default qlen 1000

3: enp23s0f1: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

4: enp23s0f2: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000

5: enp23s0f3: <BROADCAST,MULTICAST> mtu 1500 qdisc noop state DOWN group default qlen 1000

6: eno1: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 1500 qdisc mq state DOWN group default qlen 1000

    altname enp26s0f0
7: eno2: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br101 state UP group default qlen 1000
 altname enp26s0f1
       valid_lft forever preferred_lft forever
      
8: vlan100@enp23s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br100 state UP group default qlen 1000

9: vlan10@enp23s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br10 state UP group default qlen 1000

10: vlan20@enp23s0f0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br20 state UP group default qlen 1000

11: br100: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000

12: br10: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP group default qlen 1000
    inet 192.168.10.2/24 brd 192.168.10.255 scope global br10
       valid_lft forever preferred_lft forever
      
13: br20: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

14: br101: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue state UP group default qlen 1000

 

[Patrick M. Hausen]

DHCP is discouraged for servers, anyway. Configure static addresses and leave it at that. My take, YMMV.

 

[Redemption]

Hi Patrick, I was referring to assigning a DHCP IP address to the bridge interface. My server has a static IP address. If I add a DHCP address to the bridge interface and reboot the server, It will be assigned an IP address, and for some reason, if I log into Truenas thereafter, I will get logged off the Truenas GUI frequently, like every minute. If I try to assign a static IP address to the bridge interface, I cannot save the change as stated above.

 

[Patrick M. Hausen]

1. Don't use DHCP for any interface of the TrueNAS server. The bridge interfaces are also parts of your server.
2. Is the DHCP address part of the same network as the static "main" one?

 

[Redemption]

Oh, thanks for the clarification. The subnet is different; the static main one is on br10.

Static IP interface br10
enp23s0f0 -> vlan10 -> br10 192.168.10.2/24


Second Bridge Interface where I tried DHCP and STATIC
enp23s0f0 -> vlan100 -> br100 192.168.1.0/24

To make sure I understand Patrick, I only need to set up one management interface static IP address in one of my bridge interfaces, such as I have done on br10. On the other bridge interfaces assigned to other VLANs, I should not configure either a Static or DHCP IP address?

Thanks for the help in advance.

 

[Patrick M. Hausen]

If they are only to connect VMs then do not configure any IP address at all. If the host is to communicate in any way in that VLAN, do set an IP address. A bridge interface is a virtual switch. An unmanaged switch does not have an IP address, either. It's all layer 2.

 

[Redemption]

That makes a lot of sense; thank you for the explanation. How can I allow a VM to connect to the host?

Example:

Host;
enp23s0f0 -> vlan10 -> br10 192.168.10.2/24

VM installed in Truenas scale
enp23s0f0 -> vlan100 -> br100 No Ip on the bridge interface. The IP on the VM connected to the Bridge interface is 192.168.1.2/24

As configured, if I type the Truenas IP address on the VM, the request times out.

 

[Patrick M. Hausen]

Whatever router/firewall/layer-3-switch connects all those VLANs needs to take care of that. Also you need a proper default gateway in your VM guest OS and also on your TN host. These are separate networks, routers connect those.

 

[browntiger]

This stuff when it doesn't work was always giving me a headache. [Also did not think vlans are that complicated...]
Nothing particularly wrong with your setup. Before you attempt DHCP always use static IPs IMHO.

You can ping your VM / or reach some known service on Your VM from some other computer?
Your Truenas UI (or some service e.g. SMB/nfs) are not bound to that VM IP (only 192.168.10.2/24)?
VM can ping other computers?

 

[Patrick M. Hausen]

This stuff when it doesn't work was always giving me a headache. [Also did not think vlans are that complicated...]

They are not complicated. They are separate virtual switches. You need to have a good understanding about what goes on on which layer of the network stack (layer 2 vs. layer 3). That's all, really. And that is 100% identical for a router with 5 interfaces connected to 5 switches or a router with 5 VLANs connected to a switch with 5 VLANs.

If you lack that understanding, why do you want to use VLANs in the first place? Over and over again I see people mess with enterprise technology and concepts for a small home network. How large is your setup? What do you gain by using VLANs?

@Redemption, @browntiger's question is valid and helpful. What other systems are connected to that same VLAN and can your VM ping them?

 

[Redemption]

Vlans for me are indispensable. I run /25 and have a ton of VMs that I use for labing for my job. I have Vmware servers and a few Cisco-managed switches on a fiber network. I have a good understanding of how to configure rules on my router and switches for VLAN access. Everything is working as it should on my network, except for the Truenas server and trying to run Vm's from the Truenas server. I could always throw the truenas server or VMWare or even Proxmox. But I do not want to do that unless I have to, I rather keep it barebones. However if I am unable to run a domain controller on the barebone system, then I will have to go with option B :(

This stuff when it doesn't work was always giving me a headache. [Also did not think vlans are that complicated...]
Nothing particularly wrong with your setup. Before you attempt DHCP always use static IPs IMHO.

You can ping your VM / or reach some known service on Your VM from some other computer?
Your Truenas UI (or some service e.g. SMB/nfs) are not bound to that VM IP (only 192.168.10.2/24)?
VM can ping other computers?

I can Ping the Truenas server IP address from my Domain controller VM installed in truenas. I can resolve the IP address from my Domain controller VM. I can also ping other network devices, and the Domain Controller is in sync with another Domain controller on the network. The problem is that any time I type the hostname of an app installed in Truenas it does not resolve. If I type the IP address of the tTruenas server and the port of the application, it times out. So for some reason, Truenas is not accepting traffic from the VM.

Thanks for the questions and the help.

 

[browntiger]

Just $.02... I do not trust pings. You don't know who is really replying...
Does this work from your VM:

Code:

 curl http://192.168.10.2/ui/shell

?

 

[Patrick M. Hausen]

What type of device is doing the routing from your DC VM to the TrueNAS host IP?

 

[Redemption]

What type of device is doing the routing from your DC VM to the TrueNAS host IP?

I am using PFsense on a barebones computer. I have no policies blocking traffic from one VLAN to the other at this time. Once I get this sorted out, I will put some policies to block traffic except for DNS, DHCP and access to the Domain Controler.

Just $.02... I do not trust pings. You don't know who is really replying...
Does this work from your VM:

Code:

 curl http://192.168.10.2/ui/shell

?

I am using port 81 and port 444. So I changed the curl command.
curl http://192.168.10.2:81/ui/shell
<html>
<head><title>307 Temporary Redirect</title></head>
<body>
<center><h1>307 Temporary Redirect</h1></center>
<hr><center>nginx</center>
</body>
</html>

On port 81 I get a response, however, if I try an application such as Jackett I get no response.

curl http://192.168.10.2:9117/UI/Dashboard
curl: (28) Failed to connect to 192.168.10.2 port 9117 after 21049 ms: Timed out

 

[Patrick M. Hausen]

Might be specific to the application setup in SCALE. Did you configure interface and default gateway in the global application settings?

Apps > Settings > Advanced settings


The particular settings are for my environment, of course, adapt as necessary.

If all else fails do a tcpdump on your pfsense on the (VLAN) interface connected to the TrueNAS host and apps. If you see "SYN" packets from your VM and no answers, try a tcpdump on the TrueNAS host and the management/app interface. If again you see "SYN" packets coming in but no answers, then there'S definitely something with the default gateway and/or other routing issue with respect to your app setup.

If you see answers on the TN host but not on the pfsense then we need to investigate further.

 

[browntiger]

Sounds like on the 192.168.10.2 you are reaching out to UI shell, but NOT Kubernetes? This is a totally different issue...
So show us the Kubelets settings - it should be 192.168.10.2+ vlan10 + the default gateway and not some 0.0.0.0 bs.

 

[Redemption]

Thank you both for your patience and assistance. The picture of the Kubernetes is after I changed the node IP to the actual IP address; it was set to 0.0.0.0. It did not solve my problem, but I am glad we fixed that.

Patrick, I will follow your advice and do some dumps a little later. Thanks for the suggestion.