How to investigate system reboot after deploying prometheus or grafana on k3s?

Joined
Apr 6, 2023
Messages
1
First I use TrueNAS UI to deploy using the default configuration and the system will reboot. When they are in the stop state it is back to normal. The system has been running steadily for more than a year, no reboot is encountered only when prometheus or grafana is deployed. Now I have no idea how to troubleshoot other than reinstalling the system.

Can provide additional information as needed, just let me know.

What I've done
  • Checked the kernel logs and found no abnormalities, went into IPMI management to check the hardware status is also normal.
  • Upgraded the system from TrueNAS-SCALE-22.12.0 to TrueNAS-SCALE-22.12.1
  • Same result with prometheus-helm-charts and grafana-helm-charts with the helm default configuration.
    Code:
    helm install grafana grafana/grafana --namespace metric

Environment:
TrueNAS-SCALE-22.12.1
k3s version v1.25.3+k3s-9afcd6b9-dirty (9afcd6b9)
prometheus:2.42.0
CPU: 20C40T
Mem: 128GiB
 
Top