TrueCharts Logging Server - Loki + syslog-ng + Grafana?

[mobrien118]

Hello,

Pre-warning - this is a WIP that eventually could be a howto if I figure it out. I'm asking for community support to solve the problem and have already done quite a bit of research. I'll post my current challenges as sub-posts and update the main post as I move along.

I'm having an issue with a router crashing and, due to not logging to persistent storage, it has been difficult to research the issue. I have experience using syslog-ng along with LogAnalyzer on Linux in the past for aggregating disparate logs for rudimentary research capabilities, but don't really want to build a VM just for that right now.

So, I thought "I'm enjoying TrueNAS Scale and have found it useful - let's see if TrueCharts has a solution!"

I went looking and found that Loki is a logging server that could be a solution. It then integrates into Grafana, which can provide some aggregation information that may be helpful. I don't know, though, as I haven't used either yet, but it sounds like a reasonable proposition.

I installed Loki and then discovered that it can only directly accept a newer log format (RFC5424) than my router outputs (RFC3164). I discovered in this blog that syslog-ng can forward logs received in RFC3164 format to an RFC5424 based listener - bingo!

Now I have installed 3 TrueCharts containers - Loki (+Promtail), Grafana and syslog-ng. All good - now just need to integrate them.

This is where I am now... I'm posting current challenges below.

 

[mobrien118]

My current challenge is configuring the Syslog-ng config file. Logging into the container shell, I can read the syslog-ng config file, and I'm sure I can either edit it directly or mount the container and edit it from the host, but I think a better solution would be to store the modified config file on host storage and then inject it into the container from the chart config, so that if I need to rebuild it, I just need to make sure I have the updated file and the config should persist.

I'm having trouble finding exactly how to do this - I'm pretty new to containers, but my initial research indicates this can be done through configuration, just not sure exactly how to pass the setting using TrueCharts. This has to be trivial, I just haven't been able to find the solution yet.

 

[FindingFilene]

Thank you for working on this, @mobrien118. I have the feeling you might need to mount external storage to the syslog-ng app, and then execute whatever external storage config file you've created--or just copy paste some of the configs there. Let me know because, as far as I guess, what you're doing is what TrueCharts is thinking for their users. I haven't seen Graylog like I did on TrueNAS Core.

 

[mobrien118]

Yep - got the config file in. I created a folder in the "ix-applications" filesystem called "custom-config/" and then put a folder for the specific app in there "syslog-ng/" and created a "syslog-ng.conf" file in there with the same contents as the default config in the container.

I, then, in the app config, added a "Configure additional app storage" and had it mount "/mnt/MainStorage/ix-applications/custom-config/syslog-ng/syslog-ng.conf" from the host filesystem to the mount path "/etc/syslog-ng/syslog-ng.conf"

Brilliant that you can mount individual files! As a bonus, since it is mounted, not copied at runtime, changes that I make to the host file are immediately reflected in the container - of ocurse the service has to be restarted anyway for them to actually take effect, but still a very nice solution.

At this point, I don't understand the upside/downside of having the file in the "ix-applications" folder - the point is to have this config persist host destruction (but filesystem survival) and be able to re-animate the solution by just reinstalling TrueNas SCALE and importing a backed-up config file. I'm feeling pretty good about it, though.

I also now understand inter-app communication a little better with the use of the "cluster.local" domain and the individual app networking. I'll update the main post once I have this all sorted out a little bit better with the config elements.

 

[mobrien118]

Newest challenge: Selecting which pod to mount the file into.

So, I have to do the same as above for Loki. I need to mount a custom "promtail.yaml" file. I tried the same method that I used in the "syslog-ng" case, however, Loki uses 2 different images/pods - one for Promtail and one for Loki proper. When I specify to mount the custom file, it is mounting it in the "loki" container rather than the "loki-promtail" one. I don't see a way to control that in the setup interface... Hopefully I figure it out.

 

[mobrien118]

I realize I never followed up on this. I spent weeks trying to do what seemed like it should be a simple task and eventually gave up.

TrueCharts (or, TrueNas, in general) does not currently seem to have an app that is suitable for the simple purpose of functioning as a network logging server.

After *much* trial, I got to the point where I have everything working except Promtail. All that is needed is a simple addition to a config file, but the container is read only with no capability, that I could find, to mount it as r/w. It took me quite a while just to be able to load an editor in the Promtail container. In other apps, I have been able to mount a modified config file through the chart, but there is no way to do this for "dependency" containers in Truechart's architecture. They made it clear they do not intend to implement this feature.

The Truecharts guy suggested that I run the Promatail "app" by itself. However, on multiple systems, it continuously says "deploying" on every system and never runs. This is before even attempting to load the custom config file, which seems to work, but, since the app will never actually load when not attached to a Loki deployment, it doesn't really matter.

I also tried getting help from other TrueCharts users/developers/admins/etc. in multiple forums, first their Discord because I saw many other people directed their while I was trying to research this. It felt like asking for help in a crowded room when you don't know anyone. I admit, I'm not a regular Discord user, so I'm not sure I'm using it correctly, but picking the most probable channel and posting in it seems correct - never heard back from multiple posts.

Looking for help in other channels, like Reddit, got me redirected back to the Discord, which, on multiple tries, yielded no help. Posting a bug report or feature request on the git just got closed with "we're not doing that."

I rarely come to a point in a battle to implement technology where I "throw in the towel", but I feel I have exhausted all avenues available to me, short of a pull request and writing code (with tools and in languages I'm unfamiliar with). It's not worth that much to me - I'll just suck it up and spin up a VM with Graylog on it, or syslog-ng+LogAnalyzer (although Graylog is superior).

Sorry to anyone who was hoping for more from this adventure.

 

[thecrownguy]

I realize I never followed up on this. I spent weeks trying to do what seemed like it should be a simple task and eventually gave up.

TrueCharts (or, TrueNas, in general) does not currently seem to have an app that is suitable for the simple purpose of functioning as a network logging server.

After *much* trial, I got to the point where I have everything working except Promtail. All that is needed is a simple addition to a config file, but the container is read only with no capability, that I could find, to mount it as r/w. It took me quite a while just to be able to load an editor in the Promtail container. In other apps, I have been able to mount a modified config file through the chart, but there is no way to do this for "dependency" containers in Truechart's architecture. They made it clear they do not intend to implement this feature.

The Truecharts guy suggested that I run the Promatail "app" by itself. However, on multiple systems, it continuously says "deploying" on every system and never runs. This is before even attempting to load the custom config file, which seems to work, but, since the app will never actually load when not attached to a Loki deployment, it doesn't really matter.

I also tried getting help from other TrueCharts users/developers/admins/etc. in multiple forums, first their Discord because I saw many other people directed their while I was trying to research this. It felt like asking for help in a crowded room when you don't know anyone. I admit, I'm not a regular Discord user, so I'm not sure I'm using it correctly, but picking the most probable channel and posting in it seems correct - never heard back from multiple posts.

Looking for help in other channels, like Reddit, got me redirected back to the Discord, which, on multiple tries, yielded no help. Posting a bug report or feature request on the git just got closed with "we're not doing that."

I rarely come to a point in a battle to implement technology where I "throw in the towel", but I feel I have exhausted all avenues available to me, short of a pull request and writing code (with tools and in languages I'm unfamiliar with). It's not worth that much to me - I'll just suck it up and spin up a VM with Graylog on it, or syslog-ng+LogAnalyzer (although Graylog is superior).

Sorry to anyone who was hoping for more from this adventure.

This just takes the wind out of my sails. I see all these tutorials on youtube using grafana only to find it doesnt work with anything I use. We use PRTG and love it but the plugin for grafana is long sense defunct and all of the truecharts options just dopnt work or you spend way to many hours getting it to work only to have very few functions. I dont want to give up on this either as we love PRTG and truenas but PRTG has an aweful interface.

 

[danb35]

TrueCharts (or, TrueNas, in general) does not currently seem to have an app that is suitable for the simple purpose of functioning as a network logging server.

IIRC, CORE has a plugin (which are bad news in general) for Graylog, but there isn't an app (either from iX or from Truecharts) for SCALE. Not sure why; there's definitely an official Docker image for it.

 

[mobrien118]

IIRC, CORE has a plugin (which are bad news in general) for Graylog, but there isn't an app (either from iX or from Truecharts) for SCALE. Not sure why; there's definitely an official Docker image for it.

After many poor experiences with TrueCharts (more than 3 strikes, this post's contents just being ONE of them), even if they had one, now, I wouldn't use it. I'm only going to use TrueNAS official plugins for the foreseeable future. There are more and more showing up every day - maybe Graylog will be added soon!

 

[vampirebyte]

IIRC, CORE has a plugin (which are bad news in general) for Graylog, but there isn't an app (either from iX or from Truecharts) for SCALE. Not sure why; there's definitely an official Docker image for it.

Probably because there is no good upstream chart for it? Seems like this chart is still managed, tried it and it works very well, deployed a cluster with HelmRepository in fluxcd:

Code:

apiVersion: helm.toolkit.fluxcd.io/v2beta1
kind: HelmRelease
metadata:
  name: graylog
  namespace: graylog
spec:
  releaseName: graylog
  chart:
    spec:
      chart: graylog
      version: ^2.3.4
      sourceRef:
        kind: HelmRepository
        name: graylog
        namespace: flux-system
  interval: 1h
  values:
    graylog:
      image:
        tag: "5.2"
        pullPolicy: "Always"
      replicas: 2
      input:
        tcp:
          service:
            name: graylog-tcp
            type: LoadBalancer
          ports:
            - name: gelf
              port: 12222
        udp:
          service:
            name: graylog-udp
            type: LoadBalancer
          ports:
            - name: gelf
              port: 12222
      ingress:
        enabled: true
        hosts:
        - graylog.domain.dev
        tls:
        - secretName: graylog-tls-secret
          hosts:
          - graylog.domain.dev
        annotations:
          cert-manager.io/cluster-issuer: aws-dev
          cert-manager.io/private-key-rotation-policy: Always
          traefik.ingress.kubernetes.io/router.entrypoints: websecure
          traefik.ingress.kubernetes.io/router.tls: 'true'
      init:
        kubectlVersion: v1.29.1
      metrics:
        enabled: true
        serviceMonitor:
          enabled: false
    opensearch:
      enabled: true
      sysctl:
        enabled: false
      volumeClaimTemplate:
        resources:
          requests:
            storage: 100Gi

I had a good experience at work with graylog, but with their new licence and original creator leaving the company, not sure what to say.

BTW, I also have grafana, prometheus+operator+traefik+cert-manager+issuer, all from truecharts, working beautifully.