TrueNAS apps can't access physical devices on local network

[rs_taylor]

Wondering if anyone can help me.

I have a simple home network (192.168.31.0/24). It has a typical Home Wifi Router that provides DHCP. TrueNAS Scale is running on PC (192.168.31.225).
TrueNAS has many drives & shares, all working great. All devices (PC's, IP Camera's, Phones, Tablets etc) on the 192.168.31.0 network can access them.

TrueNAS also has many apps (Plex, Jelllyfin, QBitTorrerent, SyncThing, Uptime Kuma, etc) installed and for the most part the apps work well (accessed via ports like 192.168.31.225:8095, no domains/reverse proxyies), they can access the Internet, via the gateway router (192.168.31.1) as well as other apps/resources running on the TrueNAS box.
HOWEVER, i have noticed that the Apps running on the TrueNas Scale do NOT have access to other physical devices running on the main network (192.168.31.0/24), apart from the Truenas box itself (192.168.31.225) and the Home Wifi Router (Gateway)

For example none of the TrueNAS Apps can see my IP Cameras (192.168.31.250 + 251). If i open a Shell on TrueNAS' it can ping the camera's, but the Apps running on TrusNas cannot access the Camera's.
I have no idea why, im sure there is someonthing simple i'm missing but its driving me mad.

 

[davistw]

It aint easy... It took me about a week and lots and lots of searching and experimenting to figure it out. Basicly you have to install a bridge to get Apps from the kubernetes cluster out to your local network. Look into Metalb. From there you can assign your APPs an IP address on your local address space and from there it can accesses from your local net. Do some research on this. Again it is not easy by any means but the path is out there.

 

[Patrick M. Hausen]

HOWEVER, i have noticed that the Apps running on the TrueNas Scale do NOT have access to other physical devices running on the main network (192.168.1.0), apart from the Truenas box itself (192.168.31.225) and the Home Wifi Router (Gateway)

I just noticed the same yesterday after I found that Uptime Kuma can monitor everything but systems on the same local network. It seems the app/k3s networking routes these packets to the default gateway instead of using ARP on the local net.

I can work around this by a firewall rule on my OPNsense that explicitly permits e.g. "src: TrueNAS, dst: local network, dst port: 443" and OPNsense will reroute the packets, but it's a hack and very annoying.

No idea what is wrong with TrueNAS here.

Kind regards,
Patrick

 

[rs_taylor]

I just noticed the same yesterday after I found that Uptime Kuma can monitor everything but systems on the same local network. It seems the app/k3s networking routes these packets to the default gateway instead of using ARP on the local net.

I can work around this by a firewall rule on my OPNsense that explicitly permits e.g. "src: TrueNAS, dst: local network, dst port: 443" and OPNsense will reroute the packets, but it's a hack and very annoying.

No idea what is wrong with TrueNAS here.

Kind regards,
Patrick

Thanks for the quick response Patrick, not sure if it's a recent thing for me or just that i hadn't noticed, Uptime Kuma stating stuff was down that was actually working fine.

I dont use OPNSense or PFSense but maybe i should be anyway, especially if it helps me work around this app weirdness.

Regards,

Rob

 

[crownrai]

HOWEVER, i have noticed that the Apps running on the TrueNas Scale do NOT have access to other physical devices running on the main network (192.168.1.0), apart from the Truenas box itself (192.168.31.225) and the Home Wifi Router (Gateway)

What sub-net mask are you using? Typically, 192.168.x.x addresses use 255.255.255.0 and are split into smaller sub-nets that include for example, IP's 192.168.1.1-254. or 192.168.31.1-254 exclusively. If everything on your network is using 255.255.224.0 (or smaller like 255.255.0.0), then that would include everything from 192.168.1.1 - 192.168.31.254 and I would suspect it could work fine. This is assuming you are running all your networks on the same physical network segment/VLAN.

However, if you are using 255.255.255.0 and relying on ARP broadcasts to allow your 192.168.1.x network devices to see your 192.168.31.x devices, then this will cause problems with Scale apps as K3S (the app service) uses it own internal subnet/network and likely won't be able to use ARP broadcasts to see outside the same subnet assigned to Scale.

Your typical home WiFi routers don't usually have settings for multiple sub-nets or VLANs. You would need a layers 3 router to allow the different sub-nets to talk to each other.

 

[rs_taylor]

Sorry for confusion all physical devices are on 192.168.31.x 255.255.255.0.

 

[Patrick M. Hausen]

What sub-net mask are you using?
[...]

This is not the issue in my case. 192.168.2.0/24. All apps are sending all their packets to the default gateway even if the destination is on the local network.

I started this thread on the OPNsense forum, because I found a feature not working as expected in OPNsense, too

Traffic on same interface blocked?

Traffic on same interface blocked?

forum.opnsense.org

 

[Patrick M. Hausen]

I just had an idea that solves the problem for me at least. My entire home network runs on VLANs, all servers (OPNsense, CORE, SCALE) are connected with LACP and VLANs on top.

Just give the apps their own VLAN - case closed.

 

[Ericloewe]

Also SR-IOV VFs, automatic management of those would be cool to see.

 

[rs_taylor]

Also SR-IOV VFs, automatic management of those would be cool to see.

What's that?

 

[NugentS]

I just noticed the same yesterday after I found that Uptime Kuma can monitor everything but systems on the same local network. It seems the app/k3s networking routes these packets to the default gateway instead of using ARP on the local net.

I can work around this by a firewall rule on my OPNsense that explicitly permits e.g. "src: TrueNAS, dst: local network, dst port: 443" and OPNsense will reroute the packets, but it's a hack and very annoying.

No idea what is wrong with TrueNAS here.

Kind regards,
Patrick

I have been waffling about this for a while. IX have refuxed to accept this as a bug. I spotted this because I have PBR on my router that intercepted the traffic and sent local traffic off to the internet. Basically the kube-router is sending all local traffic straight at the gateway.

Its a bug

IX won't fix it saying its an upstream issue - which seems unlikley to me. There is / was a long discussion on JIRA which they closed as working as designed / not a bug - thats the correct way a router should work. I disagreed.

Possible solutions:
1. Use a firewall rule to allow the firewall to redirect traffic back to the LAN
2. Does the app have a host or advanced networking check box - if so check that and the routing seems to work correctly (where I have tested)
3. Give every app its own VLAN - which neatly gets around the issue - but IMO - something of a ballache

 

[Ericloewe]

What's that?

SR-IOV allows for PCIe devices to expost multiple Virtual Functions (VFs), which can be used semi-independently and even passed through to VMs. E.g. a NIC can present multiple VFs so that VMs or other resources can pretend they have a real, physical NIC attached directly to the same network as the physical NIC, with minimal overhead.

 

[Patrick M. Hausen]

1. Use a firewall rule to allow the firewall to redirect traffic back to the LAN

That was my first "solution" with OPNsense but I very much prefer a clean design so that triggers me quite a bit.

2. Does the app have a host or advanced networking check box - if so check that and the routing seems to work correctly (where I have tested)

The one app that needs to reach out to various systems in my setup is Uptime Kuma and that did not work with "host networking". Then maybe I do not yet understand what that does in the end

3. Give every app its own VLAN - which neatly gets around the issue - but IMO - something of a ballache

No, of course not! Give the entire app/k3s thingy one private VLAN for all apps. That's at least working for me and what I now settled on.

 

[NugentS]

I have no idea what host networking actually does. I do know it has "fixed" the issue in some cases

 

[harsh]

That was my first "solution" with OPNsense but I very much prefer a clean design so that triggers me quite a bit.

Such is the nature of containers. If you're going to isolate something in a container, there's going to have to be a path established to expose it to the LAN.

 

[Patrick M. Hausen]

That's why I prefer jails with their no-nonsense virtual network interfaces that you can bridge, route, NAT or not ... to your heart's content.

 

[harsh]

That's why I prefer jails with their no-nonsense virtual network interfaces that you can bridge, route, NAT or not ... to your heart's content.

If only jails were as easy to come by as Docker containers.

What you suggest is like preferring a Holly carburetor in a world of fuel injection and smog testing

 

[Ericloewe]

That's not a good analogy at all. More like someone figured out a cheaper machine to make screws, but it only makes left-handed threads, so everyone who was using right-handed threads was sort of left hanging.

 

[Patrick M. Hausen]

If only jails were as easy to come by as Docker containers.

You mean preconfigured jails for dedicated applications? For me it's: what could be easier than pkg install <application>?

 

[harsh]

For me it's: what could be easier than pkg install <application>?

How complex does that process get when there is no preconfigured jail package for the application you want to install?

Jails are very much a FreeBSD thing.