How to reach apps from host via external interface IP?

[jhk62]

Use case: writing TrueNAS reporting data to influxdb app (graphite, port 2003) on same host

I use the official 1.8 influxdb docker image from docker hub. As TrueNAS SCALE does not provide an option to use a custom port to send reporting data to influxdb, I'm stuck to 2003. Thus I defined an external interface in the influxdb app settings. After changing the reporting settings to this IP I noticed, that no data arrives in influxdb. But if I point the reporting data from another TrueNAS system to this influxdb app, data comes in. So, the app works via external interface IP. If I point the reporting from the TrueNAS SCALE system to the other TrueNAS system's influxdb, data arrives there too. So sending reporting data from TrueNAS SCALE works too. That puzzled me.

These are the systems I used for basic networking checks. Both apps are deployed on the TrueNAS SCALE host.

1	TrueNAS SCALE host (TrueNAS-SCALE-22.02.4)	192.168.178.190
2	influxdb app(from docker hub)	192.168.178.193
3	pihole app (from docker hub)	192.168.178.192
4	TrueNAS CORE host	192.168.178.115

Result from basic network check with ping:

4 -> 1 works
4 -> 2 works
4 -> 3 works

1 -> 2 doesn't work (From 192.168.178.190 icmp_seq=1 Destination Host Unreachable)
1 -> 3 doesn't work (From 192.168.178.190 icmp_seq=1 Destination Host Unreachable)
1 -> 4 works

3 -> 1 doesn't work (From 192.168.178.192 icmp_seq=1 Destination Host Unreachable)
3 -> 4 works
3 -> 2 works

Could not check from 2 because the influxdb image lacks even basic troubleshooting programs like ping

So it seems that no packets are routed between host and apps via the external interface.

Routing table (1)

Code:

root@truenas-scale[~]# ip route list
default via 192.168.178.1 dev eno1 proto static
172.16.0.0/16 dev kube-bridge proto kernel scope link src 172.16.0.1
192.168.178.0/24 dev eno1 proto kernel scope link src 192.168.178.190
root@truenas-scale[~]#

Routing table (3)

Code:

root@pihole-ix-chart-84c4898bd6-kxlgf:/# ip route list
default via 172.16.0.1 dev eth0
172.16.0.0/16 dev eth0 proto kernel scope link src 172.16.0.64
192.168.178.0/24 dev net1 proto kernel scope link src 192.168.178.192
root@pihole-ix-chart-84c4898bd6-kxlgf:/# 

So it seems that no packets are routed between host and apps via the external interface.

Is there a way to fix this (besides setting up a TIG stack in a VM instead of apps )?

 

[sgtkurtz]

Did you figure this out? It sounds like a problem I had - I was using bonded NICs and had to setup a bridge.
( ( Eth1 + Eth2 ) = Bond1 ) = Bridge1

Once I assigned that interface to things, traffic started flowing.

They do need to add a port config for the graphite data export feature. I'm running graphite on a VM as a workaround.

 

[jhk62]

Yes, I figured it out and set up a bridge too. I do use the influxdb version 1.8 docker image from DockerHub and activated the graphite plugin. The network config is set to "add external interface" -> "br0" (my bridge) -> static ip in the app setup. As I do use an additional, dedicated IP for influxdb here, I don't need another port.

My TIG stack is now

telegraf: installed within a VM because I do need IPMI tools to gather data like power consumption, temperatures, fan speeds
influxdb: image from DockerHub, dedicated IP via bridge
grafana: image from Docker hub, no dedicated IP

Works like a charm now.

 

[sgtkurtz]

Very nice! That's pretty much the setup I want to work towards as well. I'm curious about telegraf + IPMI, I wonder if you could script the setup and running both both in a script, and run it with init scripts to accomplish the same thing. I know someone had done so with just telegraf, but I haven't had time to look into it further. I also want to collect IPMI info from my iDrac.

 

[jhk62]

A part of the data can be gathered by enabling the ipmitools section in telegraf's config like this

Code:

# # Read metrics from the bare metal servers via IPMI
[[inputs.ipmi_sensor]]
#   ## optionally specify the path to the ipmitool executable
    path = "/usr/bin/ipmitool"
#   ##
#   ## Setting 'use_sudo' to true will make use of sudo to run ipmitool.
#   ## Sudo must be configured to allow the telegraf user to run ipmitool
#   ## without a password.
    use_sudo = false
#   ##
#   ## optionally force session privilege level. Can be CALLBACK, USER, OPERATOR, ADMINISTRATOR
#   # privilege = "ADMINISTRATOR"
#   ##
#   ## optionally specify one or more servers via a url matching
#   ##  [username[:password]@][protocol[(address)]]
#   ##  e.g.
#   ##    root:passwd@lan(127.0.0.1)
#   ##
#   ## if no servers are specified, local machine sensor stats will be queried
#   ##
  servers = [ "user:password@lan(<ip addr>)" ]

#
#   ## Recommended: use metric 'interval' that is a multiple of 'timeout' to avoid
#   ## gaps or overlap in pulled data
    interval = "30s"
#
#   ## Timeout for the ipmitool command to complete. Default is 20 seconds.
    timeout = "10s"

Because my Supermicro's IPMI functionionality lacks some data, I installed and call Supermicro's own IPMI tools by a shell script. This is enabled in telegraf's config like this

Code:

 [[inputs.exec]]
   ## Commands array
   commands = [ "/opt/telegraf-scripts/supermicro-power.sh truenas-main <ip addr> user password" ]

The script gathers the data and then writes it to stdout like this

Code:

/usr/bin/echo "power,server=$1 InputVoltage="$InputVoltage
/usr/bin/echo "power,server=$1 Fan1="$Fan1
/usr/bin/echo "power,server=$1 InputPower="$InputPower
/usr/bin/echo "power,server=$1 MainOutputPower="$MainOutputPower
/usr/bin/echo "power,server=$1 Temperature1="$Temperature1
/usr/bin/echo "power,server=$1 Temperature2="$Temperature2