MinIO and Prometheus

[KirklandIceCream]

I finally have MinIO installed, however there is a message that reads:

The MinIO Dashboard is displaying basic metrics only due to missing the necessary settings for displaying extended metrics.

See Collect MinIO Metrics Using Prometheus for a complete tutorial on scraping and visualizing MinIO metrics with Prometheus."

I have spent the last 2 days trying to figure it out. I have even reached out to MinIO support, but I still can't figure this out.

How do I setup Prometheus on TrueNas to get MiniIO to work? I just want the extended metrics, and the easiest setup to achieve this. Any help is appreciated, or if there is a truenas guide that does this, or a youtube video? I tried searching but didn't see any..

thank you!

 

[Samuel Tai]

Are you using the built-in S3 service, or the MinIO community plugin?

Either way, a high-level procedure to accomplish this is:

1. As there's no plugin for Prometheus, you'll have to create a generic FreeBSD jail, and then install and configure Prometheus in it.
   • See https://www.truenas.com/docs/core/uireference/jailspluginsvms/jails/jailsscreens/ on how to create a generic jail using the Wizard
   • Once the jail is up, enter the jail shell:
     
     
   • Inside the jail, run pkg bootstrap to initialize the pkg software manager.
   • Then run pkg install prometheus to install Prometheus in the jail.
   • After installation, enable Prometheus on startup via sysrc prometheus_enable=YES, and then configure /usr/local/etc/prometheus.yml as desired for the scrape_configs. (Sorry, I don't run Prometheus myself, so I have no clue how to configure it.)
   • Start Prometheus via service prometheus start.
   • Finally, run ifconfig epair0b to get the IP address of the Prometheus jail.
   • Exit out of the jail shell
2. Next modify /usr/local/etc/rc.d/minio to add the required environment variables. If you're running the system S3 service, this is the direct path. If you're running the MinIO plugin, you'll need to access the jail shell for the plugin and then find this path
   • Edit the file to include:
     
     Code:

     export MINIO_SERVER_URL="<IP of Prometheus jail>:<scrape port>"
     export MINIO_SERVER_JOB_ID="<whatever's configured in your prometheus.yml/default:minio-job>"

   • Restart MinIO to pick up the environment variables

 

[KirklandIceCream]

Thank you very much! @Samuel Tai

I have a couple questions/issues:
1. "then configure /usr/local/etc/prometheus.yml" how? Is there a command for this somewhere? and "Next modify /usr/local/etc/rc.d/minio" I don't know how to even open these files etc

I have it up to here so far:

 

[Samuel Tai]

Thank you very much! @Samuel Tai

I have a couple questions/issues:
1. "then configure /usr/local/etc/prometheus.yml" how? Is there a command for this somewhere? and "Next modify /usr/local/etc/rc.d/minio"

Use nano text editor to modify these files. If nano isn't available, install it with pkg: e.g., pkg install nano.

 

[KirklandIceCream]

Thank you!

I got the config from: https://min.io/docs/minio/linux/ope...direct#minio-metrics-collect-using-prometheus

scrape_configs:
- job_name: minio-job
bearer_token: TOKEN
metrics_path: /minio/v2/metrics/cluster
scheme: https
static_configs:
- targets: [minio.example.net]

Mine is:

and I did the ifconfig epair0b command and got:

I'm not sure what needs to be changed? I'm using the MinIO plugin, as well as the S3 service enabled. Do I just copy and paste the scrape config verbatim? and I'm not sure about the IP, I don't see any port?

maybe related?

s3 service:

minio:
http://192.168.1.210:9002 - console
http://192.168.1.210:9001 - API url?

 

[Samuel Tai]

Sorry, this is Prometheus-specific stuff, which I don't know. I do know the initial config you set up needs to be customized

Code:

scrape_configs:
- job_name: minio-job
  bearer_token: TOKEN <- replace with your MinIO bearer token
  metrics_path: /minio/v2/metrics/cluster
  scheme: https
  static_configs:
  - targets: [minio.example.net] <- replace with your MinIO IP

And I can't tell you what the scrape port is.

Also, you don't need to run both the S3 service AND the MinIO plugin. Choose one or the other. Only use the MinIO plugin if the built-in S3 service has some default setting that doesn't work for you, as you'll be able to configure it off-default in the plugin.

 

[Samuel Tai]

Note, if using the S3 service, use minio-client NOT mc to generate MinIO client commands. mc is the Midnight Commander file manager.

 

[Samuel Tai]

I was able to get this working on my installation after futzing with it on and off over a couple of days. Much of the difficulty was due to certificates.

1. First, on the host, run minio-client to initialize the /root/.minio-client directory.
   • Next, delete the built-in aliases
     minio-client alias rm gcs
minio-client alias rm s3
minio-client alias rm local
   • If you're using a certificate for the S3 service, copy the CA cert which issued the S3 cert to /root/.minio-client/certs/CAs. If this is a self-signed cert, copy the self-signed cert. If this isn't done, the next step won't complete, and minio-client will complain about invalid certs.
   • Create an alias for the S3 service
     minio-client alias set TrueNAS-S3 <URL of S3 service> <S3 service access key> <S3 service secret key>
     This will create an alias entry in /root/.minio-client/config.json. Note, the URL will usually end in :9000. If you set up the certs correctly, this will complete without complaint.
   • Now, run minio-client admin prometheus generate TrueNAS-S3. Note the term after the generate has to match an existing alias in /root/.minio-client/config.json. This will generate a Prometheus scrape config with the correct bearer token, and the correct target URL. Copy that off to a safe place for the next step.
2. Now, in your Prometheus jail
   • Create directory /usr/local/etc/ssl/certs
   • Copy the CA cert which issued the S3 cert into this directory. Note the name of the cert file.
   • Make sure the cert files have permission 644.
   • Edit /usr/local/etc/prometheus.yml
     • Replace the default scrape_configs: section with the minio-client-generated scrape config
     • Insert these lines in the scrape config under scheme: https
       enable_http2: false
       tls_config:
       ca_file: /usr/local/etc/ssl/certs/<name of your CA file> <- indent this 2 spaces
   • Restart Prometheus via service prometheus restart
   • Make sure Prometheus can see the target as up, by browsing to http://<IP of your Prometheus jail>:9090/targets?search=
     • If the target isn't UP, you most likely typoed the ca_file line in the prometheus.yml. Fix the typo and restart the Prometheus service.
     • If Prometheus itself isn't up, there's another syntax error in the prometheus.yml. Make sure you pasted in the minio-client config correctly, and inserted the 2 additional lines correctly
3. Back on the host, modify /usr/local/etc/rc.d/minio to include the required environment variables
   • Scroll down until you find the line load_rc_config ${name}
   • Insert the following lines underneath
     export MINIO_PROMETHEUS_URL="http://<IP of your Prometheus jail>:9090"
     export MINIO_PROMETHEUS_JOB_ID=minio-job
   • Restart the S3 service either via the UI or via service minio restart
4. Login to the MinIO console, and you should now have Prometheus graphs

 

[KirklandIceCream]

Thank you so much for taking the time!

>If you're using a certificate for the S3 service, copy the CA cert which issued the S3 cert to /root/.minio-client/certs/CAs. If this is a self-signed cert, copy the self-signed cert. If this isn't done, the next step won't complete, and minio-client will complain about invalid certs.

Where/how do I do this part?

 

[Samuel Tai]

Thank you so much for taking the time!

>If you're using a certificate for the S3 service, copy the CA cert which issued the S3 cert to /root/.minio-client/certs/CAs. If this is a self-signed cert, copy the self-signed cert. If this isn't done, the next step won't complete, and minio-client will complain about invalid certs.

Where/how do I do this part?

Are you using certificates on your S3 service?

 

[KirklandIceCream]

Are you using certificates on your S3 service?

Yes

 

[Samuel Tai]

OK, that's the default self-signed certificate. Go to System->Certificates and export the certificate. You can open the downloaded certificate in Notepad, and paste it into nano, and then save into the CAs folder as freenas_default.pem.

 

[KirklandIceCream]

Thank you for that. I have the notepad open with the certificate text.

I'm a bit confused now as to what to do with this?

I go to MinIo Jails > shell > "nano /root/.minio-client/certs/CAs" > paste > save?


Honestly is it possible if I pay you to get on a call with me and guide me through this, or if you have real-time chat (discord etc)?

Setting this up has really been the hardest thing for me lately, and I have routinely worked on machine learning, python, etc. I don't know why I can't figure this out for the life of me

 

[Samuel Tai]

You're confused because you don't understand the difference between the host and the jail.

• The host is the one running the S3 service and minio-client. Use the web Shell or a SSH client, like PuTTY, to access the host.
• The jail is the one running Prometheus. Use the jail shell or an SSH client combined with iocage console prometheusto access the jail.

Unfortunately, I was mistaken about cutting & pasting in nano, so that's part of your confusion. Neither the web Shell nor the jail shell can access the Windows paste buffer. PuTTY will allow pasting via right-click.

My employer also doesn't allow me to accept payment as a volunteer moderator nor participate in support chats as tech support.

 

[KirklandIceCream]

Is it possible for you to make a video guide or something on this? or dumb down your step by step guide?

Like open command prompt, then type "mc.exe", then type "blah blah" etc...

i honestly really appreciate your help so far, this is such a complicated endeavour. i'm surprised this isn't an east setup like the other stuff in truenas

 

[Samuel Tai]

I learned something that may ease your troubles. It turns out there's no cutting & pasting required. The cert is already on-box in the required format at /etc/certificates/freenas_default.crt. You can copy that into the required folders via cp /etc/certificates/*.crt /root/.minio-client/certs/CAs and cp /etc/certificates/*.crt /mnt/<name of your pool>/iocage/jails/prometheus/root/usr/local/etc/ssl/certs.

 

[KirklandIceCream]

I learned something that may ease your troubles. It turns out there's no cutting & pasting required. The cert is already on-box in the required format at /etc/certificates/freenas_default.crt. You can copy that into the required folders via cp /etc/certificates/*.crt /root/.minio-client/certs/CAs and cp /etc/certificates/*.crt /mnt/<name of your pool>/iocage/jails/prometheus/root/usr/local/etc/ssl/certs.

where do i put this command? a shell? the mc.exe?

 

[Samuel Tai]

Yes, this is in the web Shell.

 

[KirklandIceCream]

Yes, this is in the web Shell.

the command didnt work in either 3 shells

 

[Samuel Tai]

Try cp /usr/local/etc/minio/certs/*.crt /root/.minio-client/certs/CAs and cp /usr/local/etc/minio/certs/*.crt /mnt/<your pool name>/iocage/jails/prometheus/root/usr/local/etc/ssl/certs.