System shut down "truenas.local shutdown 15502 - - power-down by root:"

[C8Z06]

I was in the middle of navigating some folders via an SMB share from a windows machine, and suddenly TrueNAS became unresponsive. I could no longer access the UI or even ping it. I went to the server (Supermicro) and it was OFF.

I dug around in the logs and found this:

Jan 4 00:00:00 truenas syslog-ng[3026]: Configuration reload request received,reloading configuration;
Jan 4 00:00:00 truenas syslog-ng[3026]: Configuration reload finished;
Jan 4 07:39:15 truenas 1 2023-01-04T07:39:15.847420-08:00 truenas.local shutdown 15502 - - power-down by root:
Jan 4 07:39:17 truenas 1 2023-01-04T07:39:17.383075-08:00 truenas.local ntpd 3111 - - ntpd exiting on signal 15 (Terminated)
Jan 4 07:39:19 truenas syslog-ng[3026]: syslog-ng shutting down; version='3.35.1'
Jan 4 08:01:28 truenas syslog-ng[3027]: syslog-ng starting up; version='3.35.1'
Jan 4 08:01:28 truenas ---<<BOOT>>---
Jan 4 08:01:28 truenas Copyright (c) 1992-2021 The FreeBSD Project.

I am very stumped. Any thoughts or suggestions?

 

[artlessknave]

root powered down the system. clearly!
/sarcasm

do you have any automated scripts that might power it down? like a cron job?
otherwise, you might want to check your network for security if you have unknown roots running around...

 

[C8Z06]

root powered down the system. clearly!
/sarcasm

do you have any automated scripts that might power it down? like a cron job?
otherwise, you might want to check your network for security if you have unknown roots running around...

lol ... yea! solved! lol

So I looked in Tasks -> Cron Jobs and there is nothing, and I cannot seem to find where crons are stored on the file system with a quick pass.

What do you mean about unknown roots running around? My mind is blown - can you explain this scenario to me? (ALL my root passwords on all systems are the same password by the way. I did that in purpose when I was getting started to keep things simple. I planned to go back and change them all once I had things up and going)

 

[jgreco]

So I looked in Tasks -> Cron Jobs and there is nothing, and I cannot seem to find where crons are stored on the file system with a quick pass.

They're stored in the "GUI and middleware". There's no reason to expect that they're stored in the traditional system crontabs.

Do you have anything like a UPS monitoring daemon set up?

 

[C8Z06]

No, not yet. I will but haven't even started. Who knows. I guess if it happens again I can go back at it.

 

[Dunuin]

Got the same problem. Running a TrueNAS Core 13.0-U4 bare metal where everything is fine. Exported config file. Shutdown bare metal TrueNAS server.I want to move that TrueNAS to a VM on Proxmox VE 7.4. So I installed a new TrueNAS Core 13.0-U4 inside a VM with the same amount of CPU cores, disk space and RAM. Didn't moved any physical HBAs + disks yet, because I first want to see if it runs stable. Then I imported config file, edited the network settings. Looks like TrueNAS runs fine except that it always shutsdown itself after 4-5 minutes.
In the log I see the same message:

Code:

May  4 21:56:15 MainNAS Starting collectd_daemon.
May  4 21:56:15 MainNAS Configuring vt: keymap blanktime.
May  4 21:56:16 MainNAS net.inet.carp.allow: 0 -> 1
May  4 21:56:16 MainNAS
May  4 21:56:16 MainNAS Thu May  4 21:56:16 CEST 2023
May  4 21:58:54 MainNAS 1 2023-05-04T21:58:54.363122+02:00 MainNAS.local smartd 1496 - - Configuration file /usr/local/etc/smartd.conf parsed but has no entries
May  4 22:00:37 MainNAS 1 2023-05-04T22:00:37.752818+02:00 MainNAS.local shutdown 1552 - - power-down by root:
May  4 22:00:37 MainNAS net.inet.carp.allow: 1 -> 0
May  4 22:00:37 MainNAS Stopping cron.
May  4 22:00:37 MainNAS Waiting for PIDS: 1245.
May  4 22:00:37 MainNAS Stopping collectd_daemon.
May  4 22:00:37 MainNAS Waiting for PIDS: 1271.
May  4 22:00:37 MainNAS Waiting for collectd to terminate.
May  4 22:00:37 MainNAS Waiting for PIDS: 1272.
May  4 22:00:37 MainNAS Stopping nginx.
May  4 22:00:37 MainNAS Waiting for PIDS: 1262.
May  4 22:00:38 MainNAS Stopping sshd.
May  4 22:00:38 MainNAS Waiting for PIDS: 1205

According to the Proxmox VE logs there was no shutdown inititialized.

There are no custom init/shutdown scripts, cron jobs enabled. On the bare metal installation I was using NUT as a slave. But on the VM the NUT service and its autostart are disabled.

So no idea what is shutting down the server. Any hints to find out what is going on?

 

[winnielinnie]

Could be related to this (soon-to-be-fixed) bug:

[NAS-120602] - iXsystems TrueNAS Jira

ixsystems.atlassian.net

 

[Dunuin]

Thanks. I think you are right. Closed the old tab I had still open in Chorme (from shutting down the bare metal TrueNAS server) and now it is running for 30 minutes instead of the usual shutdown after 4-5 minutes.

 

[artlessknave]

What do you mean about unknown roots running around?

that was a /joke

Could be related to this (soon-to-be-fixed) bug:

holy wow wtf did they do to make that!

 

[artlessknave]

Proxmox VE

all orifices pucker right up.