System dataset broken and not mounted on pool

[viniciusferrao]

Hello. It seems that the system dataset in one of my systems is broken. RRD graphics aren't showing up so after debugging the system dataset is the one to blame.

I don't know how to reenable/recover it, but here's the situation:

Code:

root@truenas-repos[/tmp]# zfs list -t filesystem | grep -F ".system"
boot-pool/.system                                            698M  24.3G      645M  legacy
boot-pool/.system/configs-40cc91dacae0491e84781ab81ded8ba4  1.89M  24.3G     1.89M  legacy
boot-pool/.system/cores                                       96K  1024M       96K  legacy
boot-pool/.system/ctdb_shared_vol                             96K  24.3G       96K  legacy
boot-pool/.system/glusterd                                    96K  24.3G       96K  legacy
boot-pool/.system/rrd-40cc91dacae0491e84781ab81ded8ba4      41.8M  24.3G     41.8M  legacy
boot-pool/.system/samba4                                     676K  24.3G      676K  legacy
boot-pool/.system/services                                    96K  24.3G       96K  legacy
boot-pool/.system/syslog-40cc91dacae0491e84781ab81ded8ba4   8.26M  24.3G     8.26M  legacy
boot-pool/.system/webui                                       96K  24.3G       96K  legacy
repos0/.system                                               709M  4.02T      645M  legacy
repos0/.system/configs-40cc91dacae0491e84781ab81ded8ba4     2.64M  4.02T     2.64M  legacy
repos0/.system/cores                                         162K  1024M      162K  legacy
repos0/.system/ctdb_shared_vol                               162K  4.02T      162K  legacy
repos0/.system/glusterd                                      162K  4.02T      162K  legacy
repos0/.system/rrd-40cc91dacae0491e84781ab81ded8ba4         49.6M  4.02T     49.6M  legacy
repos0/.system/samba4                                        875K  4.02T      875K  legacy
repos0/.system/services                                      162K  4.02T      162K  legacy
repos0/.system/syslog-40cc91dacae0491e84781ab81ded8ba4      10.1M  4.02T     10.1M  legacy
repos0/.system/webui                                         162K  4.02T      162K  legacy

Code:

root@truenas-repos[/tmp]# mount | grep -i var   
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
boot-pool/.system on /var/db/system type zfs (rw,relatime,xattr,noacl,casesensitive)
boot-pool/.system/cores on /var/db/system/cores type zfs (rw,relatime,xattr,noacl,casesensitive)
boot-pool/.system/samba4 on /var/db/system/samba4 type zfs (rw,relatime,xattr,noacl,casesensitive)
boot-pool/.system/syslog-40cc91dacae0491e84781ab81ded8ba4 on /var/db/system/syslog-40cc91dacae0491e84781ab81ded8ba4 type zfs (rw,relatime,xattr,noacl,casesensitive)

collectd seems to be crashed:

Code:

May 21 21:23:08 truenas-repos systemd[1]: Starting Statistics collection and monitoring daemon...
░░ Subject: A start job for unit collectd.service has begun execution
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit collectd.service has begun execution.
░░ 
░░ The job identifier is 2550240.
May 21 21:23:08 truenas-repos collectd[2834667]: Configuration file /etc/collectd/collectd.conf is empty.
May 21 21:23:08 truenas-repos collectd[2834667]: Error: Parsing the config file failed!
May 21 21:23:08 truenas-repos systemd[1]: collectd.service: Main process exited, code=exited, status=1/FAILURE
░░ Subject: Unit process exited
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ An ExecStart= process belonging to unit collectd.service has exited.
░░ 
░░ The process' exit code is 'exited' and its exit status is 1.
May 21 21:23:08 truenas-repos systemd[1]: collectd.service: Failed with result 'exit-code'.
░░ Subject: Unit failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ The unit collectd.service has entered the 'failed' state with result 'exit-code'.
May 21 21:23:08 truenas-repos systemd[1]: Failed to start Statistics collection and monitoring daemon.
░░ Subject: A start job for unit collectd.service has failed
░░ Defined-By: systemd
░░ Support: https://www.debian.org/support
░░ 
░░ A start job for unit collectd.service has finished with a failure.
░░ 
░░ The job identifier is 2550240 and the job result is failed.

Any help is appreciated.

Thanks.

 

[sretalla]

A couple of things you can try...

First, move the system dataset to a different pool (and then back again).

Then restart the collectd service:

service collectd restart

 

[viniciusferrao]

A couple of things you can try...

First, move the system dataset to a different pool (and then back again).

Then restart the collectd service:

service collectd restart

How can I do that? I'm running SCALE and I can't see where are the settings for the system dataset.

 

[sretalla]

System Settings | Advanced | Storage ... click Configure...

 

[viniciusferrao]

System Settings | Advanced | Storage ... click Configure...

How could I missed that :|

Thank you, I moved to the boot-pool and the moved back to the repos0 pool and collectd is working again, graphics on /reportsdashboard are back but only the ZFS ones, others just show nothing. At least there's no longer an error.

Is there any health check that I can run?

Also the system dataset seems to be correct now:

Code:

root@truenas-repos[~]# mount | grep -i var
efivarfs on /sys/firmware/efi/efivars type efivarfs (rw,nosuid,nodev,noexec,relatime)
repos0/.system on /var/db/system type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/cores on /var/db/system/cores type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/samba4 on /var/db/system/samba4 type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/syslog-40cc91dacae0491e84781ab81ded8ba4 on /var/db/system/syslog-40cc91dacae0491e84781ab81ded8ba4 type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/rrd-40cc91dacae0491e84781ab81ded8ba4 on /var/db/system/rrd-40cc91dacae0491e84781ab81ded8ba4 type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/configs-40cc91dacae0491e84781ab81ded8ba4 on /var/db/system/configs-40cc91dacae0491e84781ab81ded8ba4 type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/webui on /var/db/system/webui type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/services on /var/db/system/services type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/glusterd on /var/db/system/glusterd type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/ctdb_shared_vol on /var/db/system/ctdb_shared_vol type zfs (rw,relatime,xattr,noacl,casesensitive)
repos0/.system/cores on /var/lib/systemd/coredump type zfs (rw,relatime,xattr,noacl,casesensitive)

Thanks.

 

[sretalla]

It may take a restart to put everything back... even clearing out the rrd data might be needed, but I would try the reboot first.

 

[viniciusferrao]

It may take a restart to put everything back... even clearing out the rrd data might be needed, but I would try the reboot first.

It was just my affliction. Graphics are gradatively coming back. So it may not even need a reboot.

I think it's fixed.

Thank you so much @sretalla