<div class="bbWrapper">I have Minio S3 Service working. It is configured to expose metrics endpoint and Prometheus is scraping it. I can see metrics within Prometheus UI.<br />
<br />
I'm trying to get the S3 Service build in dashboard to fetch metrics from an external Prometheus. The TrueNAS Core S3 service doesn't have native support it seems for providing the two environment variables needed to enable this <b>MINIO_PROMETHEUS_URL</b> and <b>MINIO_PROMETHEUS_JOB_ID</b>.<br />
<br />
<b>MINIO_PROMETHEUS_URL=https://k3s.[REDACTED].com/prometheus/<br />
MINIO_PROMETHEUS_JOB_ID=minio-job-truenas</b><br />
<br />
So I'm poking around...<br />
<br />
First I can CURL from TrueNAS to the remote Prometheus endpoint:<br />
<blockquote data-attributes="" data-quote="" data-source=""
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
root@truenas[~]# curl -XGET -G 'https://k3s.[REDACTED].com/prometheus/api/v1/label/__name__/values' --data-urlencode 'match[]={__name__=~".+", job="minio-job-truenas"}'<br />
<br />
{"status":"success","data":["ALERTS","ALERTS_FOR_STATE","count:up0","count:up1","minio_bucket_objects_size_distribution","minio_bucket_replication_received_bytes","minio_bucket_usage_object_total","minio_bucket_usage_total_bytes","minio_cluster_disk_free_inodes","minio_cluster_nodes_offline_total","minio_cluster_nodes_online_total","minio_inter_node_traffic_errors_total","minio_inter_node_traffic_received_bytes","minio_inter_node_traffic_sent_bytes","minio_node_disk_free_bytes","minio_node_disk_total_bytes","minio_node_disk_used_bytes","minio_node_go_routine_total","minio_node_ilm_expiry_pending_tasks","minio_node_ilm_transition_active_tasks","minio_node_ilm_transition_pending_tasks","minio_node_ilm_versions_scanned","minio_node_scanner_bucket_scans_finished","minio_node_scanner_bucket_scans_started","minio_node_scanner_directories_scanned","minio_node_scanner_objects_scanned","minio_node_scanner_versions_scanned","minio_s3_requests_errors_total","minio_s3_requests_inflight_total","minio_s3_requests_rejected_auth_total","minio_s3_requests_rejected_header_total","minio_s3_requests_rejected_invalid_total","minio_s3_requests_rejected_timestamp_total","minio_s3_requests_total","minio_s3_requests_waiting_total","minio_s3_time_ttfb_seconds_distribution","minio_s3_traffic_received_bytes","minio_s3_traffic_sent_bytes","minio_software_commit_info","minio_software_version_info","minio_usage_last_activity_nano_seconds","promhttp_metric_handler_requests_in_flight","promhttp_metric_handler_requests_total","scrape_duration_seconds","scrape_samples_post_metric_relabeling","scrape_samples_scraped","scrape_series_added","up"]}#
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
I tried exporting the variables within <b>/usr/local/etc/rc.d/minio</b> such as:<br />
<br />
<blockquote data-attributes="" data-quote="" data-source=""
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
: ${minio_prometheus_url="https://k3s.[REDACTED].com/prometheus/"}<br />
: ${minio_prometheus_job="minio-job-truenas"}<br />
...<br />
# Insert these for Minio<br />
export MINIO_PROMETHEUS_URL=${minio_prometheus_url}<br />
export MINIO_PROMETHEUS_JOB_ID=${minio_prometheus_job}<br />
<br />
pidfile="/var/run/${name}.pid"<br />
procname="/usr/local/bin/minio"<br />
command="/usr/sbin/daemon"<br />
command_args="-f -t ${name} ${minio_syslog_output_flags} -p ${pidfile} /usr/bin/env ${minio_env} ${procname} ${minio_config_flag} -S \"${mi<br />
nio_certs}\" --quiet server --address=\"${minio_address}\" ${minio_console_address_arg} ${minio_disks} ${minio_args}"<br />
...
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
Restarted the S3 service and the Minio Dashboard detects the Environment variables are set, and now displays:<br />
<blockquote data-attributes="" data-quote="" data-source=""
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
We can't retrieve advanced metrics at this time<br />
MinIO Dashboard will display basic metrics as we couldn't connect to Prometheus successfully.<br />
<br />
Please try again in a few minutes. If the problem persists, you can review your configuration and confirm that Prometheus server is up and running.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
The MinIO / Tools / Logs:<br />
<blockquote data-attributes="" data-quote="" data-source=""
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
API: SYSTEM<br />
Time: 19:12:06:0<br />
DeploymentID: 2a488c37-8e1b-401a-beda-b22910187c88<br />
Error: readlink /proc/self: no such file or directory (*fs.PathError)<br />
internal/logger/logonce.go:52:logger.(*logOnceType).logOnceIf()<br />
internal/logger/logonce.go:92:logger.LogOnceIf()<br />
cmd/metrics-v2.go:891:cmd.getMinioProcMetrics.func1()<br />
cmd/metrics-v2.go:224:cmd.(*MetricsGroup).RegisterRead.func1.1()<br />
cmd/metrics-v2.go:236:cmd.(*MetricsGroup).Get()<br />
cmd/metrics-v2.go:1737:cmd.populateAndPublish()<br />
cmd/metrics-v2.go:1702:cmd.ReportMetrics.func1()
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
Not sure what else to try, don't see any options for more verbose logging.</div>
