Is there a way to keep the system log between reboots without setting up a separate syslog server?

U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
Hi, my server just turns unresponsive and all fans are maxed out. I forced it to shutdown, then rebooted it and everything is back to normal. I would like to see what happened in the system log but there is only events since the current boot in
Code:
/var/log/messages
 
winnielinnie

winnielinnie

MVP
Joined
Oct 22, 2019
Messages
3,641
This is the setting you need to enable in Core. It's probably similar with SCALE. (The menu navigation might be different.)

syslog-in-system-dataset.png
 
U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
winnielinnie said:
This is the setting you need to enable in Core. It's probably similar with SCALE. (The menu navigation might be different.)

View attachment 69773
Click to expand...
I do have that checked (it's in System > Advanced > System Log > Use system dataset for Scale) but still the syslog is cleared every time the system reboots. Am I misunderstanding where to find the syslog?
 
winnielinnie

winnielinnie

MVP
Joined
Oct 22, 2019
Messages
3,641
My logfile /var/log/messages goes back to July17th. Then there's also the compressed rotated logs (for even further back in time.)

Perhaps your log rotated, and you can find earlier messages from the most recently compressed logfile? Likely to be named messages.0.bz2

You can use "bzcat" to streamline the process, so you don't have to uncompress the file and copy it elsewhere:
Code:
bzcat /var/log/messages.0.bz2 | less
 
U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
winnielinnie said:
My logfile /var/log/messages goes back to July17th. Then there's also the compressed rotated logs (for even further back in time.)

Perhaps your log rotated, and you can find earlier messages from the most recently compressed logfile? Likely to be named messages.0.bz2

You can use "bzcat" to streamline the process, so you don't have to uncompress the file and copy it elsewhere:
Code:
bzcat /var/log/messages.0.bz2 | less
Click to expand...
I do remember seeing logs that survived between reboots before but maybe after I moved the system dataset to a different pool or since I installed Cobia, the log starts getting cleared. I just disabled "Use system dataset" and reenabled it. I will see if that makes a change.

By the way, if the system dataset is being used for logs, does that mean <Pool>/.system/<path to logs> should be mounted to /var/log?

Code:
SSD/.system                                                          623G  256K  623G   1% /var/db/system
SSD/.system/cores                                                    1.0G  133M  892M  13% /var/db/system/cores
SSD/.system/samba4                                                   623G  384K  623G   1% /var/db/system/samba4
SSD/.system/rrd-0a2b9b8243a747409c8014877982836b                     623G  128K  623G   1% /var/db/system/rrd-0a2b9b8243a747409c8014877982836b
SSD/.system/configs-0a2b9b8243a747409c8014877982836b                 623G  768K  623G   1% /var/db/system/configs-0a2b9b8243a747409c8014877982836b
SSD/.system/webui                                                    623G  128K  623G   1% /var/db/system/webui
SSD/.system/services                                                 623G  128K  623G   1% /var/db/system/services
SSD/.system/glusterd                                                 623G  128K  623G   1% /var/db/system/glusterd
SSD/.system/ctdb_shared_vol                                          623G  128K  623G   1% /var/db/system/ctdb_shared_vol
SSD/.system/netdata-0a2b9b8243a747409c8014877982836b                 623G  263M  623G   1% /var/db/system/netdata-0a2b9b8243a747409c8014877982836b


I don't see such an entry if that is the case.
 
winnielinnie

winnielinnie

MVP
Joined
Oct 22, 2019
Messages
3,641
You're missing /var/db/system/syslog-0a2b9b8243a747409c8014877982836b

But this could be the difference between SCALE (Debian Linux) and Core (FreeBSD).

Perhaps you're meant to use journalctl to read your logs in SCALE?

Worth a shot:
Code:
journalctl

* The default view will have the most recent entries sorted at the top, and the oldest towards the bottom.
 
U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
I tried journalctl, but it gives the same results with only events since the current boot starting with
Code:
Aug 28 19:56:40 TrueNAS kernel: microcode: microcode updated early to revision 0x2c, date = 2023-01-04
Aug 28 19:56:40 TrueNAS kernel: Linux version 6.1.42-production+truenas (root@tnsbuilds01.tn.ixsystems.net) (gcc (Debian 12.2.0-14) 12.2.0, GNU ld (GNU Binutils for Debian) 2.40) #2 SMP PREEMPT_DYNAMIC Mon Aug 14 23:21:26 UTC 2023
Aug 28 19:56:40 TrueNAS kernel: Command line: BOOT_IMAGE=/ROOT/23.10-BETA.1@/boot/vmlinuz-6.1.42-production+truenas root=ZFS=boot-pool/ROOT/23.10-BETA.1 ro ...

Thank you for the help anyway. I guess I will wait and see if someone else encounters the same issue. After all I am using a beta version.
 
winnielinnie

winnielinnie

MVP
Joined
Oct 22, 2019
Messages
3,641
So there are no compressed rotated logs?
 
U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
winnielinnie said:
So there are no compressed rotated logs?
Click to expand...
No. (Actually I never saw compressed rotated logs even back when logs were able to be preserved between reboots. Maybe it's just that I haven't built this server for long enough to trigger a rotation.)
 
winnielinnie

winnielinnie

MVP
Joined
Oct 22, 2019
Messages
3,641
Something is odd about the missing "syslog" dataset.

Can you list all datasets under SSD/.system? Not just what is currently mounted:
Code:
zfs list -t filesystem -r SSD/.system
 
Last edited:
U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
winnielinnie said:
Something is odd about the missing "syslog" dataset.

Can you list all datasets under SSD/.system? Not just what is currently mounted:
Code:
zfs list -t dataset -r SSD/.system
Click to expand...
Here are all the datasets under SSD/.system and indeed there is nothing seemingly related to syslogs.
Code:
sudo zfs list -t all -r SSD/.system
NAME                                                   USED  AVAIL  REFER  MOUNTPOINT
SSD/.system                                            377M   622G   136K  legacy
SSD/.system/configs-0a2b9b8243a747409c8014877982836b   756K   622G   756K  legacy
SSD/.system/cores                                      118M   906M   118M  legacy
SSD/.system/ctdb_shared_vol                             96K   622G    96K  legacy
SSD/.system/glusterd                                   104K   622G   104K  legacy
SSD/.system/netdata-0a2b9b8243a747409c8014877982836b   257M   622G   257M  legacy
SSD/.system/rrd-0a2b9b8243a747409c8014877982836b        96K   622G    96K  legacy
SSD/.system/samba4                                     312K   622G   312K  legacy
SSD/.system/services                                    96K   622G    96K  legacy
SSD/.system/webui                                       96K   622G    96K  legacy


If there is supposed to be a dataset for logs, I guess when I disable using system dataset for logs and re-enable it, it will attempt to create that dataset. I will try to look for relating logs to see if there is a failure creating that dataset later.
 
winnielinnie

winnielinnie

MVP
Joined
Oct 22, 2019
Messages
3,641
What version of SCALE is this? (There's going to be a change to how the syslog is saved going forward.)

If you are willing to try this:
  1. In the settings, move the System Dataset to another storage pool (or even to your boot-pool)
  2. After it completes the migration (shouldn't take more than a minute), check the datasets again:
    • zfs list -t filesystem -r <poolname>/.system
  3. If it looks correct, move the System Dataset back to your SSD pool, and check again:
    • zfs list -t filesystem -r SSD/.system

Now hopefully the log will persist across reboots.
 
S

sfatula

Guru
Joined
Jul 5, 2022
Messages
608
In Scale, System Settings -> Advanced, you will see syslog there and you can point the syslog to another machine. Just click on configure to the right of syslog heading.
 
U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
winnielinnie said:
What version of SCALE is this? (There's going to be a change to how the syslog is saved going forward.)

If you are willing to try this:
  1. In the settings, move the System Dataset to another storage pool (or even to your boot-pool)
  2. After it completes the migration (shouldn't take more than a minute), check the datasets again:
    • zfs list -t filesystem -r <poolname>/.system
  3. If it looks correct, move the System Dataset back to your SSD pool, and check again:
    • zfs list -t filesystem -r SSD/.system

Now hopefully the log will persist across reboots.
Click to expand...
It’s the current beta version of Cobia (there is something I need that is provided only in Linux kernel 6.1). I have made sure to use system dataset for syslog as shown below
1693324654194.png

I have then moved my system dataset from SSD pool to boot-pool, but still there is no dataset for logs
Code:
sudo zfs list -t all -r boot-pool/.system
NAME                                                         USED  AVAIL  REFER  MOUNTPOINT
boot-pool/.system                                            382M  25.2G   136K  legacy
boot-pool/.system/configs-0a2b9b8243a747409c8014877982836b   748K  25.2G   748K  legacy
boot-pool/.system/cores                                      118M   906M   118M  legacy
boot-pool/.system/ctdb_shared_vol                             96K  25.2G    96K  legacy
boot-pool/.system/glusterd                                   104K  25.2G   104K  legacy
boot-pool/.system/netdata-0a2b9b8243a747409c8014877982836b   263M  25.2G   263M  legacy
boot-pool/.system/rrd-0a2b9b8243a747409c8014877982836b        96K  25.2G    96K  legacy
boot-pool/.system/samba4                                     256K  25.2G   256K  legacy
boot-pool/.system/services                                    96K  25.2G    96K  legacy
boot-pool/.system/webui                                       96K  25.2G    96K  legacy

Here is everything in journalctl's output during the move
Code:
Aug 29 12:05:24 TrueNAS systemd[1]: run-containerd-runc-k8s.io-aac81f590ce7763db58cd3ca49ce83f74f844ee8dee6cea093adbd25472ef8c2-runc.Qw8UKo.mount: Deactivated successfully.
Aug 29 12:05:30 TrueNAS systemd[1]: Stopping smbd.service - Samba SMB Daemon...
Aug 29 12:05:30 TrueNAS systemd[1]: smbd.service: Deactivated successfully.
Aug 29 12:05:30 TrueNAS systemd[1]: Stopped smbd.service - Samba SMB Daemon.
Aug 29 12:05:30 TrueNAS systemd[1]: Stopping netdata.service - netdata - Real-time performance monitoring...
Aug 29 12:05:31 TrueNAS systemd[1]: netdata.service: Killing process 564201 (netdata) with signal SIGKILL.
Aug 29 12:05:31 TrueNAS systemd[1]: netdata.service: Deactivated successfully.
Aug 29 12:05:31 TrueNAS systemd[1]: Stopped netdata.service - netdata - Real-time performance monitoring.
Aug 29 12:05:31 TrueNAS systemd[1]: netdata.service: Consumed 2.769s CPU time.
Aug 29 12:05:31 TrueNAS systemd[1]: Stopping winbind.service - Samba Winbind Daemon...
Aug 29 12:05:31 TrueNAS systemd[1]: winbind.service: Deactivated successfully.
Aug 29 12:05:31 TrueNAS systemd[1]: Stopped winbind.service - Samba Winbind Daemon.
Aug 29 12:05:31 TrueNAS systemd[1]: Stopping nmbd.service - Samba NMB Daemon...
Aug 29 12:05:31 TrueNAS systemd[1]: nmbd.service: Deactivated successfully.
Aug 29 12:05:31 TrueNAS systemd[1]: Stopped nmbd.service - Samba NMB Daemon.
Aug 29 12:05:31 TrueNAS systemd[1]: Stopping wsdd.service - Web Services Dynamic Discovery host daemon...
Aug 29 12:05:32 TrueNAS systemd[1]: wsdd.service: Deactivated successfully.
Aug 29 12:05:32 TrueNAS systemd[1]: Stopped wsdd.service - Web Services Dynamic Discovery host daemon.
Aug 29 12:05:33 TrueNAS systemd[1]: var-lib-systemd-coredump.mount: Deactivated successfully.
Aug 29 12:05:33 TrueNAS middlewared[565871]: umount: /var/lib/systemd/coredump: not mounted.
Aug 29 12:05:33 TrueNAS systemd[1]: var-db-system-netdata\x2d0a2b9b8243a747409c8014877982836b.mount: Deactivated successfully.
Aug 29 12:05:33 TrueNAS systemd[1]: var-db-system-ctdb_shared_vol.mount: Deactivated successfully.
Aug 29 12:05:33 TrueNAS systemd[1]: var-db-system-glusterd.mount: Deactivated successfully.
Aug 29 12:05:33 TrueNAS systemd[1]: var-db-system-services.mount: Deactivated successfully.
Aug 29 12:05:34 TrueNAS middlewared[565883]: umount: /var/lib/systemd/coredump: not mounted.
Aug 29 12:05:34 TrueNAS systemd[1]: run-middleware-system.new-netdata\x2d0a2b9b8243a747409c8014877982836b.mount: Deactivated successfully.
Aug 29 12:05:34 TrueNAS systemd[1]: var-db-system-webui.mount: Deactivated successfully.
Aug 29 12:05:34 TrueNAS systemd[1]: var-db-system-configs\x2d0a2b9b8243a747409c8014877982836b.mount: Deactivated successfully.
Aug 29 12:05:34 TrueNAS systemd[1]: var-db-system-rrd\x2d0a2b9b8243a747409c8014877982836b.mount: Deactivated successfully.
Aug 29 12:05:34 TrueNAS systemd[1]: var-db-system-samba4.mount: Deactivated successfully.
Aug 29 12:05:34 TrueNAS systemd[1]: var-db-system-cores.mount: Deactivated successfully.
Aug 29 12:05:34 TrueNAS systemd[1]: var-db-system.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-ctdb_shared_vol.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-glusterd.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-services.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-webui.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: Started wsdd.service - Web Services Dynamic Discovery host daemon.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-configs\x2d0a2b9b8243a747409c8014877982836b.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-rrd\x2d0a2b9b8243a747409c8014877982836b.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-samba4.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new-cores.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: run-middleware-system.new.mount: Deactivated successfully.
Aug 29 12:05:35 TrueNAS systemd[1]: Starting nmbd.service - Samba NMB Daemon...
Aug 29 12:05:35 TrueNAS systemd[1]: Started nmbd.service - Samba NMB Daemon.
Aug 29 12:05:35 TrueNAS systemd[1]: Starting winbind.service - Samba Winbind Daemon...
Aug 29 12:05:36 TrueNAS systemd[1]: Started winbind.service - Samba Winbind Daemon.
Aug 29 12:05:36 TrueNAS systemd[1]: Started netdata.service - netdata - Real-time performance monitoring.
Aug 29 12:05:36 TrueNAS netdata[566028]: CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Aug 29 12:05:36 TrueNAS netdata[566028]: 2023-08-29 12:05:36: netdata INFO  : MAIN : CONFIG: cannot load cloud config '/var/lib/netdata/cloud.d/cloud.conf'. Running with internal defaults.
Aug 29 12:05:36 TrueNAS systemd[1]: Starting smbd.service - Samba SMB Daemon...
Aug 29 12:05:36 TrueNAS systemd[1]: Started smbd.service - Samba SMB Daemon.
Aug 29 12:05:36 TrueNAS middlewared[566064]: umount: /var/lib/systemd/coredump: not mounted.
Aug 29 12:05:39 TrueNAS k3s[5828]: {"level":"warn","ts":"2023-08-29T12:05:39.391-0400","logger":"etcd-client","caller":"v3@v3.5.7-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000d8ea80/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}

It doesn't seem to keep notice of the which datasets are moved or created though.
 
winnielinnie

winnielinnie

MVP
Joined
Oct 22, 2019
Messages
3,641
ULTIoNT said:
It’s the current beta version of Cobia
Click to expand...
I think this might have something to do with this:

Constant writes to boot device despite moving system dataset

My current setup is using a HPE Proliant Microserver Gen 10+ with 2 16 TB HDDs in raid0 (I know it has no redundancy, but it is just a small homeserver to start with), 16 GB ECC memory, 1 TB NVME SSD to test things like L2ARC/SLOG and now moving the system dataset to it so the boot pool doesn't...
www.truenas.com www.truenas.com

You might have to disable the option to store the logs to the System Dataset. It could be an "incomplete" feature in Cobia. Maybe they forgot to remove the option...
 
anodos

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
9,554
Regarding Cobia BETA - the webui option is currently a no-op. The systemd journal is on tmpfs / volatile, syslog-ng handles writes to /var/log (and other places as-needed). There is no code path to remove files from /var/log and it is on ZFS (not tmpfs).

If you can reliably reproduce cases where logs on /var/log aren't being preserved on reboot, please file bug ticket for developers to investigate what is going on.
 
U

ULTIoNT

Dabbler
Joined
Jul 14, 2023
Messages
24
winnielinnie said:
I think this might have something to do with this:

Constant writes to boot device despite moving system dataset

My current setup is using a HPE Proliant Microserver Gen 10+ with 2 16 TB HDDs in raid0 (I know it has no redundancy, but it is just a small homeserver to start with), 16 GB ECC memory, 1 TB NVME SSD to test things like L2ARC/SLOG and now moving the system dataset to it so the boot pool doesn't...
www.truenas.com www.truenas.com

You might have to disable the option to store the logs to the System Dataset. It could be an "incomplete" feature in Cobia. Maybe they forgot to remove the option...
Click to expand...
That does seem to be related. I do observe that there is constant write to the boot drive although that's not where the system dataset resides. As anodos said the option is a no-op, so I guess disabling it won't make a difference. I will go on and file a bug ticket.
 
You must log in or register to reply here.

Similar threads

W
  • Locked
FreeNAS spontaneous reboot without panic / dump / log
Replies
4
Views
2K
cyberjock
C
C
Investigating a complete hang
Replies
0
Views
513
Computer8080
C
indivision
  • Locked
Is there a way to have FreeNAS reboot on panic crashes?
Replies
6
Views
5K
indivision
indivision
M
Multiple ungraceful reboots and a temporarily unhealthy pool on "scrub-day"
2 3
Replies
43
Views
7K
marcevan
marcevan
P
System dataset is using to much space
Replies
3
Views
769
anodos
anodos
Top