No GUI or SSH after pod stop

[ferpando]

Hello,
I'm pretty new to truenas scale.
I've been running it for a few months now without a problem.

My system:

version TrUeNAS-SCALE-22.12.3.3
buildtime 2023-07-25701:59:36+00:00
hostname truenas
physmem 20974878720
model AMD Ryzen 7 5700X 8-Core Processor
cores 6
physical_cores 6
loadavg 0.08
upt ime 0:43:02.239319
uptime_seconds 2582.239319119
system_serial
system_product Standard PC (Q35 + ICH9, 2009)
system_product_version pc-935-7.2
boottime 2023-08-22T10:15:15+00:00
system_manufacturer QEMU
ecc_memory true

Today I tried to update the app tailscale (truecharts version) and it failed.
Then I tried to stop the container and then all the problems started.

Truenas hanged and nothing was working anymore.

Hopefully I'm running truenas as a VM so at least I have access to the console.

SSH is not working, GUI is not working, ping is not working.
Shares are working and I can access them from other computers and the system boots apparently ok. No major errors in the console.

From the console I can access cli and linux, reboot, etc.
Volume is mounted and the rest of the system seems ok.

I manually deleted tailscale pod from the system and rebooted but still not working.

I really would appreciate any pointers in beginning to solve this problem.
What to look for, errors, logs...

Thanks in advance.

 

[sretalla]

Maybe the middleward is frozen...

systemctl restart middlewared

 

[ferpando]

Just tried but did not work.
It gave no errors.

I guess when the system reboot everything gets restarted.
I tried a few times with no success.

[2023/08/22 13:12:30] (INFO) middlewared. _init. () :876 - Starting TrueNAS-SCALE-22.12.3.3 middleware
[2023/08/22 13:12:35] (DEBUG) middlewared.setup():57 - Timezone set to Europe/Madrid
[2023/08/22 13:12:38] (DEBUG) middlewared.setup ():25 - Certificate setup for System complete
[2023/08/22 13:12:38] (DEBUG) middlewared. _plugins_setup () :997 - All plugins loaded
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks() :1010 - Setting up periodic task alert.flush_alerts to run every 3600 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks():1010 - Setting up periodic task alert.process_alerts to run every 60 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared. _setup_periodic_tasks():1010 - Setting up periodic task certificate.renew_certs to run every 86400 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks ():1010 Setting up periodic task certificateauthority.cr1_generation to run every 86400 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks():1010 - Setting up periodic task chart.release.periodic_chart_releases_update_checks to run evE
•y 86400 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks ():1010 - Setting up periodic task cluster job.process_queue to run every 3600 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks (): 1010 - Setting up periodic task kerberos.keytab.check_updated_keytab to run every 3600 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks():1010 - Setting up periodic task kmip.sync_keys to run every 86400 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared. _setup_periodic_tasks () :1010 - Setting up periodic task mail.send mail queue to run every 600 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks(): 1010 Setting up periodic task pool.dataset.sync_dh_keys to run every 86400 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks ():1010 - Setting up periodic task service.check_deprecated_services to run every 3600 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks ):1010 - Setting up periodic task smb.sharesec.check_share_info_tdb to run every 3600 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks ():1010 Setting up periodic task truecommand. health_check to run every 1800 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared._setup_periodic_tasks):1010 - Setting up periodic task vmware.delete_pending_snapshots to run every 10800.0 seconds
[2023/08/22 13:12:38] (DEBUG) middlewared. setup periodic tasks): 1010 - Setting up periodic task zettarepl.flush state to run every 3600 seconds
(2023/08/22 13:12:381 (DEBUG) middlewared initialize (1:1851 Accenting connections
cont@trupnas[/var/irel#

 

[ferpando]

Also tried reapplying network settings from the console, in case tailscale messed things up.. but nothing

 

[ferpando]

I see in the logs a bunch of these

64.198134] IPVS: r: UDP 172.17.0.10:53 - no destination available
64.198135] IPVS: r: UDP 172.17.0.10:53 - no destination available
64.198252] IPVS: r: UDP 172.17.0.10:53 - no destination available
64.200500] IPVS: r: UDP 172.17.0.10:53 - no destination available
64.200577] IPVS: r: UDP 172.17.0.10:53 - no destination available

 

[sretalla]

I see in the logs a bunch of these

Those are DNS requests for kubernetes... some apps trying to find something (and failing), should not impact the host.

 

[ferpando]

Should not.
Also have things like this:

{"log":"2023/08/22 12:25:49 magicsock: [warning] failed to force-set UDP read buffer size to 7340032: operation not permitted; using kernel default values (impacts throughput only)

"log":"2023/08/22 12:25:59 control: bootstrapDNS(\"derp8d.tailscale.com\", \"178.62.44.132\") for \"controlplane.tailscale.com\" error: Get \"https://derp8d.tailscale.com/bootstrap-dns?

{"log":"2023/08/22 12:25:53 logtail: dial \"log.tailscale.io:443\" failed: dial tcp: lookup log.tailscale.io on 172.17.0.10:53: server misbehaving (in 4.003s), trying bootstrap...

{"log":"2023/08/22 12:26:02 recursive: querying IPv4 addresses for: \"log.tailscale.io\"\n","stream":"stderr","time":"2023-08-22T12:26:02.463424182Z"}

Also kubernetes related but it is clear there is a network problem