Truenas Scale GUI Unreachable

F

FabrizioR8

Dabbler
Joined
Jul 13, 2022
Messages
17
wondering if its not a networking problem, but a services issue. if sshd and middlewared crash, you’d lose access… but the k3s rancher/docker services stay up and networking is actually fine.

Just a suspicion.

take a close look at /var/log/* log files and see if middlewared.log and sshd.log have anything to say.
 
J

js4616

Dabbler
Joined
Dec 27, 2022
Messages
14
I see errors in middwared.log but don't know how to interpret them, I cannot find sshd.log in /var/db/system/syslog-a79f32b3a852420ca0f9433c4fd6884f/log.

I used winscp to pull the following. 01/08 was my last reboot before the one today to use ssh and get these


[2023/01/08 20:06:49] (INFO) middlewared.__init__():811 - Starting TrueNAS-SCALE-22.12.0 middleware
[2023/01/08 20:06:52] (DEBUG) middlewared.setup():57 - Timezone set to US/Eastern
[2023/01/08 20:06:52] (DEBUG) middlewared.generate_keys():213 - Generating SSH host keys
[2023/01/08 20:06:54] (DEBUG) middlewared.setup():25 - Certificate setup for System complete
[2023/01/08 20:06:54] (DEBUG) middlewared.__plugins_setup():926 - All plugins loaded
[2023/01/08 20:06:54] (DEBUG) middlewared.__initialize():1739 - Accepting connections
[2023/01/08 20:06:54] (INFO) DiskService.log_disk_info():72 - Found disks: {'sdg': {'serial': '7SHJU5KW', 'lunid': '5000cca252d5bbf6'}, 'sdc': {'serial': '2YJJ7E0D', 'lunid': '5000cca273e390cc'}, 'sdf': {'serial': 'PNY381822539701018D7', 'lunid': '5f8db4c3818018d7'}, 'sdd': {'serial': '7HK12RZN', 'lunid': '5000cca257eac52c'}, 'sdh': {'serial': '012345678950', 'lunid': None}, 'sda': {'serial': '7SHJTRYW', 'lunid': '5000cca252d5ba50'}, 'sdb': {'serial': 'WD-WMC4M1678086', 'lunid': '50014ee209708652'}, 'sde': {'serial': '7HK0WA9N', 'lunid': '5000cca257eaad1d'}, 'nvme0n1': {'serial': 'S29NNXBG912888', 'lunid': '00253841591cb8f7'}, 'nvme1n1': {'serial': '2030AD801802', 'lunid': 'e8238fa6bf530001001b448b498a4256'}}
[2023/01/08 20:06:54] (DEBUG) EtcService.generate():476 - No new changes for /etc/group
[2023/01/08 20:06:54] (DEBUG) EtcService.generate():476 - No new changes for /etc/passwd
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/shadow
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/sudoers
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/aliases
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/fstab
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/keyboard
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/vconsole.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/openldap/ldap.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/nslcd.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/dhcp/dhclient.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-common
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-kernel-server
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rpcbind
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/idmapd.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-account
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-auth
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-password
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-session-noninteractive
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-session
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/proftpd/proftpd.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/kdump-tools
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/smartmontools
[2023/01/08 20:06:55] (DEBUG) smartd.ensure_smart_enabled():39 - Unable to enable smart on ['/dev/sdh', '-d', 'sat']
[2023/01/08 20:06:55] (DEBUG) smartd.ensure_smart_enabled():39 - Unable to enable smart on ['/dev/sdh', '-d', 'sat']
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/apache2.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/keepalived/keepalived.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/motd
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/nscd.conf
[2023/01/08 20:06:55] (DEBUG) middlewared.render_body():34 - UPSMON: monpwd field empty, upsmon will fail to start.
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/snmp/snmpd.conf
[2023/01/08 20:06:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/sshd
[2023/01/08 20:06:56] (DEBUG) EtcService.generate():476 - No new changes for /etc/inadyn.conf
[2023/01/08 20:06:56] (DEBUG) EtcService.generate():476 - No new changes for /etc/pykmip/pykmip.conf
[2023/01/08 20:06:56] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/tftpd-hpa
[2023/01/08 20:06:56] (DEBUG) EtcService.generate():476 - No new changes for /etc/hosts
[2023/01/08 20:06:56] (INFO) InterfaceService.configure():21 - Configuring interface 'enp3s0'
[2023/01/08 20:06:56] (DEBUG) InterfaceService.configure():102 - enp3s0: adding 192.168.1.198/255.255.255.0
[2023/01/08 20:06:56] (INFO) InterfaceService.sync():1638 - Interfaces in database: enp3s0
[2023/01/08 20:06:56] (DEBUG) EtcService.generate():476 - No new changes for /etc/dhcp/dhclient.conf
[2023/01/08 20:06:56] (INFO) RouteService.sync():91 - Adding IPv4 default route to 192.168.1.1
[2023/01/08 20:06:56] (DEBUG) EtcService.generate():476 - No new changes for /etc/nginx/nginx.conf
[2023/01/08 20:07:00] (WARNING) SystemDatasetService.sysdataset_path():119 - /var/db/system: mountpoint not found
[2023/01/08 20:07:01] (DEBUG) EtcService.generate():476 - No new changes for /etc/glusterfs/glusterd.vol
[2023/01/08 20:07:01] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/08 20:07:01] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/syslog-ng
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/Includes/webdav.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/glusterfs/glusterd.vol
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/collectd/collectd.conf
[2023/01/08 20:07:02] (ERROR) ServiceService._restart():259 - Service 'idmap' not running after restart
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rrdcached
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/rsyncd.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/ctdb/ctdb.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/syslog-ng
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/group
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/passwd
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/shadow
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/sudoers
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/aliases
[2023/01/08 20:07:02] (DEBUG) SMBService.update_passdb_user():101 - User [jeremi] does not exist in the passdb.tdb file. Creating entry with rid [20066].
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-common
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-kernel-server
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rpcbind
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/idmapd.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/proftpd/proftpd.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/apache2.conf
[2023/01/08 20:07:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/Includes/webdav.conf
[2023/01/08 20:07:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/08 20:07:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/08 20:07:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/collectd/collectd.conf
[2023/01/08 20:07:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rrdcached
[2023/01/08 20:07:10] (ERROR) ServiceService.start():163 - Service 'collectd' not running after start
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task clusterjob.process_queue to run every 3600 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds
[2023/01/08 20:07:13] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2023/01/08 20:07:13] (DEBUG) EtcService.generate():476 - No new changes for /etc/avahi/avahi-daemon.conf
[2023/01/08 20:07:14] (DEBUG) EtcService.generate():476 - No new changes for /etc/wsdd.conf
/usr/lib/python3.9/multiprocessing/resource_tracker.py:96: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some resources might leak.
warnings.warn('resource_tracker: process died unexpectedly, '
[2023/01/08 20:07:33] (DEBUG) EtcService.generate():476 - No new changes for /etc/wsdd.conf
[2023/01/08 20:12:49] (ERROR) DockerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=k8s.gcr.io image=sig-storage/csi-provisioner tag=v3.0.0: Timed out waiting for a response
[2023/01/08 22:00:38] (DEBUG) UsageService.start():64 - Scheduled next run in 158262 seconds
[2023/01/09 00:00:01] (DEBUG) middlewared.plugins.pool.__run():4601 - Pool 'apps' last scrub datetime.datetime(2022, 12, 24, 17, 37, 37)
[2023/01/09 03:45:02] (DEBUG) middlewared.plugins.pool.__run():4601 - Pool 'boot-pool' last scrub datetime.datetime(2023, 1, 5, 3, 45, 3)
[2023/01/12 13:29:09] (INFO) middlewared.__init__():811 - Starting TrueNAS-SCALE-22.12.0 middleware
[2023/01/12 13:29:12] (DEBUG) middlewared.setup():57 - Timezone set to US/Eastern
[2023/01/12 13:29:12] (DEBUG) middlewared.generate_keys():213 - Generating SSH host keys
[2023/01/12 13:29:14] (DEBUG) middlewared.setup():25 - Certificate setup for System complete
[2023/01/12 13:29:14] (DEBUG) middlewared.__plugins_setup():926 - All plugins loaded
[2023/01/12 13:29:14] (DEBUG) middlewared.__initialize():1739 - Accepting connections
[2023/01/12 13:29:15] (INFO) DiskService.log_disk_info():72 - Found disks: {'sdg': {'serial': '7SHJU5KW', 'lunid': '5000cca252d5bbf6'}, 'sde': {'serial': '2YJJ7E0D', 'lunid': '5000cca273e390cc'}, 'sdd': {'serial': 'PNY381822539701018D7', 'lunid': '5f8db4c3818018d7'}, 'sdf': {'serial': '7HK12RZN', 'lunid': '5000cca257eac52c'}, 'sdh': {'serial': '012345678950', 'lunid': None}, 'sda': {'serial': '7SHJTRYW', 'lunid': '5000cca252d5ba50'}, 'sdb': {'serial': 'WD-WMC4M1678086', 'lunid': '50014ee209708652'}, 'sdc': {'serial': '7HK0WA9N', 'lunid': '5000cca257eaad1d'}, 'nvme0n1': {'serial': 'S29NNXBG912888', 'lunid': '00253841591cb8f7'}, 'nvme1n1': {'serial': '2030AD801802', 'lunid': 'e8238fa6bf530001001b448b498a4256'}}
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/group
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/passwd
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/shadow
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/sudoers
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/aliases
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/fstab
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/keyboard
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/vconsole.conf
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/openldap/ldap.conf
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/nslcd.conf
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/dhcp/dhclient.conf
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-common
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-kernel-server
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rpcbind
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/idmapd.conf
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-account
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-auth
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-password
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-session-noninteractive
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-session
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/proftpd/proftpd.conf
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/kdump-tools
[2023/01/12 13:29:15] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/smartmontools
[2023/01/12 13:29:15] (DEBUG) smartd.ensure_smart_enabled():39 - Unable to enable smart on ['/dev/sdh', '-d', 'sat']
[2023/01/12 13:29:15] (DEBUG) smartd.ensure_smart_enabled():39 - Unable to enable smart on ['/dev/sdh', '-d', 'sat']
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/apache2.conf
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/keepalived/keepalived.conf
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/motd
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/nscd.conf
[2023/01/12 13:29:16] (DEBUG) middlewared.render_body():34 - UPSMON: monpwd field empty, upsmon will fail to start.
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/snmp/snmpd.conf
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/sshd
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/inadyn.conf
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/pykmip/pykmip.conf
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/tftpd-hpa
[2023/01/12 13:29:16] (DEBUG) EtcService.generate():476 - No new changes for /etc/hosts
[2023/01/12 13:29:16] (INFO) InterfaceService.configure():21 - Configuring interface 'enp3s0'
[2023/01/12 13:29:16] (DEBUG) InterfaceService.configure():102 - enp3s0: adding 192.168.1.198/255.255.255.0
[2023/01/12 13:29:17] (INFO) InterfaceService.sync():1638 - Interfaces in database: enp3s0
[2023/01/12 13:29:17] (DEBUG) EtcService.generate():476 - No new changes for /etc/dhcp/dhclient.conf
[2023/01/12 13:29:17] (INFO) RouteService.sync():91 - Adding IPv4 default route to 192.168.1.1
[2023/01/12 13:29:17] (DEBUG) EtcService.generate():476 - No new changes for /etc/nginx/nginx.conf
[2023/01/12 13:29:17] (DEBUG) EtcService.generate():476 - No new changes for /etc/haproxy/haproxy.cfg
[2023/01/12 13:29:17] (DEBUG) EtcService.generate():476 - No new changes for /etc/ssh/sshd_config
[2023/01/12 13:29:21] (WARNING) SystemDatasetService.sysdataset_path():119 - /var/db/system: mountpoint not found
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/glusterfs/glusterd.vol
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/syslog-ng
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/collectd/collectd.conf
[2023/01/12 13:29:22] (ERROR) ServiceService._restart():259 - Service 'idmap' not running after restart
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rrdcached
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/rsyncd.conf
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/ctdb/ctdb.conf
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/syslog-ng
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/group
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/passwd
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/shadow
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/sudoers
[2023/01/12 13:29:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/aliases
[2023/01/12 13:29:23] (DEBUG) SMBService.update_passdb_user():101 - User [jeremi] does not exist in the passdb.tdb file. Creating entry with rid [20066].
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-common
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-kernel-server
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rpcbind
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/idmapd.conf
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/proftpd/proftpd.conf
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/apache2.conf
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/Includes/webdav.conf
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/collectd/collectd.conf
[2023/01/12 13:29:23] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rrdcached
[2023/01/12 13:29:29] (ERROR) ServiceService.start():163 - Service 'collectd' not running after start
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task clusterjob.process_queue to run every 3600 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds
[2023/01/12 13:29:34] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2023/01/12 13:29:34] (DEBUG) EtcService.generate():476 - No new changes for /etc/avahi/avahi-daemon.conf
[2023/01/12 13:29:35] (DEBUG) EtcService.generate():476 - No new changes for /etc/wsdd.conf
/usr/lib/python3.9/multiprocessing/resource_tracker.py:96: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some resources might leak.
warnings.warn('resource_tracker: process died unexpectedly, '
[2023/01/12 13:29:49] (ERROR) KubernetesService.post_start_internal():158 - Failed to remove 'svclb-ombi-6a518a21' daemonset
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 82, in do_delete
await self.KUBERNETES_RESOURCE.delete(name, **options)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 118, in delete
return await cls.call(cls.uri(
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 84, in call
return await cls.api_call(uri, mode, body, headers, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 45, in api_call
async with cls.request(endpoint, mode, body, headers, timeout) as resp:
File "/usr/lib/python3.9/contextlib.py", line 175, in __aenter__
return await self.gen.__anext__()
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 33, in request
raise ApiException(f'Received {resp.status!r} response code from {endpoint!r}')
middlewared.plugins.kubernetes_linux.k8s.exceptions.ApiException: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-ombi-6a518a21'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/lifecycle.py", line 153, in post_start_internal
await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1306, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/service.py", line 944, in delete
rv = await self.middleware._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 84, in do_delete
raise CallError(f'Unable to delete {self.KUBERNETES_RESOURCE.OBJECT_HUMAN_NAME}: {e}')
middlewared.service_exception.CallError: [EFAULT] Unable to delete Daemonset: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-ombi-6a518a21'
[2023/01/12 13:29:55] (DEBUG) EtcService.generate():476 - No new changes for /etc/wsdd.conf
[2023/01/12 13:35:09] (ERROR) DockerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=k8s.gcr.io image=sig-storage/csi-node-driver-registrar tag=v2.3.0: Timed out waiting for a response
[2023/01/12 13:39:17] (ERROR) middlewared.job.run():438 - Job <bound method returns.<locals>.returns_internal.<locals>.nf of <middlewared.plugins.kubernetes_linux.update.KubernetesService object at 0x7f3289877940>> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 434, in do_update
await self.validate_data(config, 'kubernetes_update', old_config)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 350, in validate_data
verrors.check()
File "/usr/lib/python3/dist-packages/middlewared/service_exception.py", line 70, in check
raise self
middlewared.service_exception.ValidationErrors: [EINVAL] kubernetes_update.route_v4_gateway: route_v4_gateway and route_v4_interface must be specified together.
[EINVAL] kubernetes_update.route_v4_interface: route_v4_gateway and route_v4_interface must be specified together.

[2023/01/12 13:39:39] (ERROR) middlewared.job.run():438 - Job <bound method returns.<locals>.returns_internal.<locals>.nf of <middlewared.plugins.kubernetes_linux.update.KubernetesService object at 0x7f3289877940>> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 434, in do_update
await self.validate_data(config, 'kubernetes_update', old_config)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 350, in validate_data
verrors.check()
File "/usr/lib/python3/dist-packages/middlewared/service_exception.py", line 70, in check
raise self
middlewared.service_exception.ValidationErrors: [EINVAL] kubernetes_update.route_v4_gateway: route_v4_gateway and route_v4_interface must be specified together.
[EINVAL] kubernetes_update.route_v4_interface: route_v4_gateway and route_v4_interface must be specified together.

[2023/01/12 13:40:00] (ERROR) middlewared.job.run():438 - Job <bound method returns.<locals>.returns_internal.<locals>.nf of <middlewared.plugins.kubernetes_linux.update.KubernetesService object at 0x7f3289877940>> failed
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 426, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 461, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 434, in do_update
await self.validate_data(config, 'kubernetes_update', old_config)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/update.py", line 350, in validate_data
verrors.check()
File "/usr/lib/python3/dist-packages/middlewared/service_exception.py", line 70, in check
raise self
middlewared.service_exception.ValidationErrors: [EINVAL] kubernetes_update.route_v4_gateway: route_v4_gateway and route_v4_interface must be specified together.
[EINVAL] kubernetes_update.route_v4_interface: route_v4_gateway and route_v4_interface must be specified together.

[2023/01/12 13:41:02] (INFO) InterfaceService.unconfigure():141 - Unconfiguring interface 'kube-bridge'
[2023/01/12 13:41:11] (ERROR) KubernetesService.post_start_internal():158 - Failed to remove 'svclb-radarr-e8c67eee' daemonset
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 82, in do_delete
await self.KUBERNETES_RESOURCE.delete(name, **options)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 118, in delete
return await cls.call(cls.uri(
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 84, in call
return await cls.api_call(uri, mode, body, headers, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 45, in api_call
async with cls.request(endpoint, mode, body, headers, timeout) as resp:
File "/usr/lib/python3.9/contextlib.py", line 175, in __aenter__
return await self.gen.__anext__()
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 33, in request
raise ApiException(f'Received {resp.status!r} response code from {endpoint!r}')
middlewared.plugins.kubernetes_linux.k8s.exceptions.ApiException: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-radarr-e8c67eee'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/lifecycle.py", line 153, in post_start_internal
await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1306, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/service.py", line 944, in delete
rv = await self.middleware._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 84, in do_delete
raise CallError(f'Unable to delete {self.KUBERNETES_RESOURCE.OBJECT_HUMAN_NAME}: {e}')
middlewared.service_exception.CallError: [EFAULT] Unable to delete Daemonset: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-radarr-e8c67eee'
[2023/01/12 13:46:13] (ERROR) DockerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=k8s.gcr.io image=sig-storage/csi-node-driver-registrar tag=v2.3.0: Timed out waiting for a response
 
F

FabrizioR8

Dabbler
Joined
Jul 13, 2022
Messages
17
Looks like you have an issue with your kubernetes cluster config:
middlewared.service_exception.ValidationErrors: [EINVAL] kubernetes_update.route_v4_gateway: route_v4_gateway and route_v4_interface must be specified together.

What do you have configured in the Apps > Settings > Advanced Settings view?


Anything in /var/log/daemon.log and /var/log/error that look like faults/errors?

Don't worry about the following error in middlewared.log:
/usr/lib/python3.9/multiprocessing/resource_tracker.py:96: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some resources might leak.
warnings.warn('resource_tracker: process died unexpectedly, '
 
J

js4616

Dabbler
Joined
Dec 27, 2022
Messages
14
I updated it after you posted this to show a node ip that matches the system ip, route v4 interface is the nic in the system, and route v4 gateway is my udm pro.

I restarted this morning and pulled the logs you mentioned and the logs you mentioned before. See below, still have this issue and its driving me crazy.

I have all kinds of python errors but not sure how to troubleshoot them.

2023/01/13 10:14:56] (INFO) middlewared.__init__():811 - Starting TrueNAS-SCALE-22.12.0 middleware
[2023/01/13 10:14:59] (DEBUG) middlewared.setup():57 - Timezone set to US/Eastern
[2023/01/13 10:14:59] (DEBUG) middlewared.generate_keys():213 - Generating SSH host keys
[2023/01/13 10:15:01] (DEBUG) middlewared.setup():25 - Certificate setup for System complete
[2023/01/13 10:15:01] (DEBUG) middlewared.__plugins_setup():926 - All plugins loaded
[2023/01/13 10:15:01] (DEBUG) middlewared.__initialize():1739 - Accepting connections
[2023/01/13 10:15:01] (INFO) DiskService.log_disk_info():72 - Found disks: {'sdg': {'serial': '7SHJU5KW', 'lunid': '5000cca252d5bbf6'}, 'sdd': {'serial': '2YJJ7E0D', 'lunid': '5000cca273e390cc'}, 'sde': {'serial': 'PNY381822539701018D7', 'lunid': '5f8db4c3818018d7'}, 'sdf': {'serial': '7HK12RZN', 'lunid': '5000cca257eac52c'}, 'sdh': {'serial': '012345678950', 'lunid': None}, 'sda': {'serial': '7SHJTRYW', 'lunid': '5000cca252d5ba50'}, 'sdb': {'serial': 'WD-WMC4M1678086', 'lunid': '50014ee209708652'}, 'sdc': {'serial': '7HK0WA9N', 'lunid': '5000cca257eaad1d'}, 'nvme0n1': {'serial': 'S29NNXBG912888', 'lunid': '00253841591cb8f7'}, 'nvme1n1': {'serial': '2030AD801802', 'lunid': 'e8238fa6bf530001001b448b498a4256'}}
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/group
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/passwd
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/shadow
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/sudoers
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/aliases
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/fstab
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/keyboard
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/vconsole.conf
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/openldap/ldap.conf
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/nslcd.conf
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/dhcp/dhclient.conf
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-common
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-kernel-server
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rpcbind
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/idmapd.conf
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-account
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-auth
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-password
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-session-noninteractive
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/common-session
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/proftpd/proftpd.conf
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/kdump-tools
[2023/01/13 10:15:02] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/smartmontools
[2023/01/13 10:15:02] (DEBUG) smartd.ensure_smart_enabled():39 - Unable to enable smart on ['/dev/sdh', '-d', 'sat']
[2023/01/13 10:15:02] (DEBUG) smartd.ensure_smart_enabled():39 - Unable to enable smart on ['/dev/sdh', '-d', 'sat']
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/apache2.conf
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/keepalived/keepalived.conf
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/motd
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/nscd.conf
[2023/01/13 10:15:03] (DEBUG) middlewared.render_body():34 - UPSMON: monpwd field empty, upsmon will fail to start.
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/snmp/snmpd.conf
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/pam.d/sshd
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/inadyn.conf
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/pykmip/pykmip.conf
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/tftpd-hpa
[2023/01/13 10:15:03] (DEBUG) EtcService.generate():476 - No new changes for /etc/hosts
[2023/01/13 10:15:03] (INFO) InterfaceService.configure():21 - Configuring interface 'enp3s0'
[2023/01/13 10:15:03] (DEBUG) InterfaceService.configure():102 - enp3s0: adding 192.168.1.198/255.255.255.0
[2023/01/13 10:15:04] (INFO) InterfaceService.sync():1638 - Interfaces in database: enp3s0
[2023/01/13 10:15:04] (DEBUG) EtcService.generate():476 - No new changes for /etc/dhcp/dhclient.conf
[2023/01/13 10:15:04] (INFO) RouteService.sync():91 - Adding IPv4 default route to 192.168.1.1
[2023/01/13 10:15:04] (DEBUG) EtcService.generate():476 - No new changes for /etc/nginx/nginx.conf
[2023/01/13 10:15:04] (DEBUG) EtcService.generate():476 - No new changes for /etc/haproxy/haproxy.cfg
[2023/01/13 10:15:04] (DEBUG) EtcService.generate():476 - No new changes for /etc/ssh/sshd_config
[2023/01/13 10:15:08] (WARNING) SystemDatasetService.sysdataset_path():119 - /var/db/system: mountpoint not found
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/glusterfs/glusterd.vol
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/syslog-ng
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/Includes/webdav.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/glusterfs/glusterd.vol
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/collectd/collectd.conf
[2023/01/13 10:15:09] (ERROR) ServiceService._restart():259 - Service 'idmap' not running after restart
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rrdcached
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/rsyncd.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/ctdb/ctdb.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/syslog-ng
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/group
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/passwd
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/shadow
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/sudoers
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/aliases
[2023/01/13 10:15:09] (DEBUG) SMBService.update_passdb_user():101 - User [jeremi] does not exist in the passdb.tdb file. Creating entry with rid [20066].
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-common
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/nfs-kernel-server
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rpcbind
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/idmapd.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/proftpd/proftpd.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/apache2.conf
[2023/01/13 10:15:09] (DEBUG) EtcService.generate():476 - No new changes for /etc/apache2/Includes/webdav.conf
[2023/01/13 10:15:10] (DEBUG) EtcService.generate():476 - No new changes for /etc/smb4.conf
[2023/01/13 10:15:10] (DEBUG) EtcService.generate():476 - No new changes for /etc/security/pam_winbind.conf
[2023/01/13 10:15:10] (DEBUG) EtcService.generate():476 - No new changes for /etc/collectd/collectd.conf
[2023/01/13 10:15:10] (DEBUG) EtcService.generate():476 - No new changes for /etc/default/rrdcached
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task certificateauthority.crl_generation to run every 86400 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run every 86400 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task clusterjob.process_queue to run every 3600 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task mail.send_mail_queue to run every 600 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task pool.dataset.sync_db_keys to run every 86400 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task truecommand.health_check to run every 1800 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds
[2023/01/13 10:15:20] (DEBUG) middlewared._setup_periodic_tasks():939 - Setting up periodic task zettarepl.flush_state to run every 3600 seconds
[2023/01/13 10:15:21] (DEBUG) EtcService.generate():476 - No new changes for /etc/avahi/avahi-daemon.conf
[2023/01/13 10:15:22] (DEBUG) EtcService.generate():476 - No new changes for /etc/wsdd.conf
[2023/01/13 10:15:33] (ERROR) KubernetesService.post_start_internal():158 - Failed to remove 'svclb-ombi-403fee05' daemonset
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 82, in do_delete
await self.KUBERNETES_RESOURCE.delete(name, **options)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 118, in delete
return await cls.call(cls.uri(
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 84, in call
return await cls.api_call(uri, mode, body, headers, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 45, in api_call
async with cls.request(endpoint, mode, body, headers, timeout) as resp:
File "/usr/lib/python3.9/contextlib.py", line 175, in __aenter__
return await self.gen.__anext__()
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 33, in request
raise ApiException(f'Received {resp.status!r} response code from {endpoint!r}')
middlewared.plugins.kubernetes_linux.k8s.exceptions.ApiException: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-ombi-403fee05'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/lifecycle.py", line 153, in post_start_internal
await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1306, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/service.py", line 944, in delete
rv = await self.middleware._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 84, in do_delete
raise CallError(f'Unable to delete {self.KUBERNETES_RESOURCE.OBJECT_HUMAN_NAME}: {e}')
middlewared.service_exception.CallError: [EFAULT] Unable to delete Daemonset: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-ombi-403fee05'
[2023/01/13 10:15:33] (ERROR) KubernetesService.post_start_internal():158 - Failed to remove 'svclb-sabnzb-sabnzbd-4dc710ce' daemonset
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 82, in do_delete
await self.KUBERNETES_RESOURCE.delete(name, **options)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 118, in delete
return await cls.call(cls.uri(
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 84, in call
return await cls.api_call(uri, mode, body, headers, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 45, in api_call
async with cls.request(endpoint, mode, body, headers, timeout) as resp:
File "/usr/lib/python3.9/contextlib.py", line 175, in __aenter__
return await self.gen.__anext__()
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s/client.py", line 33, in request
raise ApiException(f'Received {resp.status!r} response code from {endpoint!r}')
middlewared.plugins.kubernetes_linux.k8s.exceptions.ApiException: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-sabnzb-sabnzbd-4dc710ce'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/lifecycle.py", line 153, in post_start_internal
await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1306, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/service.py", line 944, in delete
rv = await self.middleware._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1255, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1152, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1284, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/kubernetes_linux/k8s_base_resources.py", line 84, in do_delete
raise CallError(f'Unable to delete {self.KUBERNETES_RESOURCE.OBJECT_HUMAN_NAME}: {e}')
middlewared.service_exception.CallError: [EFAULT] Unable to delete Daemonset: Received 404 response code from '/apis/apps/v1/namespaces/kube-system/daemonsets/svclb-sabnzb-sabnzbd-4dc710ce'
[2023/01/13 10:15:41] (DEBUG) EtcService.generate():476 - No new changes for /etc/wsdd.conf
/usr/lib/python3.9/multiprocessing/resource_tracker.py:96: UserWarning: resource_tracker: process died unexpectedly, relaunching. Some resources might leak.
warnings.warn('resource_tracker: process died unexpectedly, '
[2023/01/13 10:21:00] (ERROR) DockerImagesService.check_update():36 - [EFAULT] Unable to retrieve latest image digest for registry=k8s.gcr.io image=sig-storage/csi-node-driver-registrar tag=v2.3.0: Timed out waiting for a response
 

Attachments

  • daemon.log.3.gz
    1.8 MB · Views: 135
  • error.3.gz
    538.2 KB · Views: 133
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Truenas Scale GUI Unreachable"

Similar threads

rikinn55
Realtek 8111E nic (integrated) turned off after boot Truenas Scale 22.12.0
Replies
5
Views
2K
rikinn55
rikinn55
tsofron@
Truenas Scale + Intel I5 1235
Replies
1
Views
1K
Arwen
Arwen
A
TrueNAS Scale Bluefin - Strange SMB share issues
Replies
1
Views
1K
azrelix
A
Cregganna
Feature Request: TrueNAS Scale Bluefin - GUI action to add managed certificate to /etc/ssl/certs
Replies
2
Views
2K
Cregganna
Cregganna
S
Networking and Kubernetes Problem
Replies
0
Views
1K
SnarkySnacks
S
Top