Your output for k3s service status?

[Daisuke]

Can anyone with running Kubernetes apps post the k3s status? Here it is mines:

Code:

# systemctl status k3s > k3s.log
# cat k3s.log
● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/lib/systemd/system/k3s.service; disabled; vendor preset: disabled)
     Active: active (running) since Sun 2022-12-25 00:12:15 EST; 2min 28s ago
       Docs: https://k3s.io
    Process: 104745 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 104746 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 104747 (k3s-server)
      Tasks: 43
     Memory: 599.8M
        CPU: 47.563s
     CGroup: /system.slice/k3s.service
             └─104747 /usr/local/bin/k3s server

Dec 25 00:14:01 uranus k3s[104747]: time="2022-12-25T00:14:01-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:06 uranus k3s[104747]: time="2022-12-25T00:14:06-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:11 uranus k3s[104747]: time="2022-12-25T00:14:11-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:16 uranus k3s[104747]: time="2022-12-25T00:14:16-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:21 uranus k3s[104747]: time="2022-12-25T00:14:21-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:26 uranus k3s[104747]: time="2022-12-25T00:14:26-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:31 uranus k3s[104747]: time="2022-12-25T00:14:31-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:36 uranus k3s[104747]: time="2022-12-25T00:14:36-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 25 00:14:40 uranus k3s[104747]: {"level":"warn","ts":"2022-12-25T00:14:40.433-0500","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc001e4c1c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 25 00:14:41 uranus k3s[104747]: time="2022-12-25T00:14:41-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"

I opened NAS-119573 for that no such table: dbstat warning, should be related to the sqlite dbstat table being non existent when sqlite is not compiled with SQLITE_ENABLE_DBSTAT_VTAB=1 option.

 

[s_mario]

I am on a freshly installed TrueNAS-SCALE-22.12.3, and this problem still hasn't been fixed.

The "k3s_daemon.log" file is being updated every 5 seconds, which is causing constant noise from the HDDs.

Code:

Jun 14 09:51:40 truenas k3s[89140]: {"level":"warn","ts":"2023-06-14T09:51:40.662-0700","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc0008296c0/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Jun 14 09:51:41 truenas k3s[89140]: time="2023-06-14T09:51:41-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 14 09:51:46 truenas k3s[89140]: time="2023-06-14T09:51:46-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 14 09:51:51 truenas k3s[89140]: time="2023-06-14T09:51:51-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 14 09:51:56 truenas k3s[89140]: time="2023-06-14T09:51:56-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 14 09:52:01 truenas k3s[89140]: time="2023-06-14T09:52:01-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 14 09:52:07 truenas k3s[89140]: time="2023-06-14T09:52:07-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 14 09:52:12 truenas k3s[89140]: time="2023-06-14T09:52:12-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 14 09:52:17 truenas k3s[89140]: time="2023-06-14T09:52:17-07:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"

 

[sammael]

Both scales 22.12.3 run ~15 apps (mostly truecharts, some official, few docker and few TC docker-compose), 1st is built from random ebay parts, 2nd is installed inside qnap TS873a enclosure
1st:

Code:

● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/lib/systemd/system/k3s.service; disabled; preset: disabled)
     Active: active (running) since Tue 2023-06-13 17:13:54 BST; 4 days ago
       Docs: https://k3s.io
    Process: 13729 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 13731 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 13737 (k3s-server)
      Tasks: 49
     Memory: 1.2G
        CPU: 10h 40min 2.242s
     CGroup: /system.slice/k3s.service
             └─13737 "/usr/local/bin/k3s server"

Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.606531   13737 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="2dd397e8d0efe3e72a68b45971b139a1d768263>
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.606619   13737 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/ix-jellyseerr_jellyseerr-66cc879789-gtlb2_472c307d-441d-4476-b24c-442dabcbaeea/jellyseerr/0.log\": failed to >
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.614560   13737 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="869b4bd106294264f553c6fdc2f78b73be1a89b>
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.614686   13737 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_adf38446-e1cb-4a53-808a-e69fd24646e1/csi-provisioner/5.log\": failed to >
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.616282   13737 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="eda702c98d388873edc3f84ab963d41a8008da6>
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.616368   13737 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-ccw94_2f7a42bb-037f-449e-9254-ba7edda45f48/openebs-zfs-plugin/1.log\": failed to>
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.617668   13737 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="136e58be0466bb59db5b5dbaf4c2a4fabe9da36>
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.617763   13737 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_adf38446-e1cb-4a53-808a-e69fd24646e1/snapshot-controller/4.log\": failed>
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.634205   13737 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="ea117aae12443e75ac6f25f9c9d5a2f733cd536>
Jun 18 08:21:19 turbofreenas k3s[13737]: E0618 08:21:19.634297   13737 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_adf38446-e1cb-4a53-808a-e69fd24646e1/csi-resizer/5.log\": failed to reop>
lines 1-23/23 (END)

2nd:

Code:

● k3s.service - Lightweight Kubernetes
     Loaded: loaded (/lib/systemd/system/k3s.service; disabled; preset: disabled)
     Active: active (running) since Fri 2023-06-16 08:32:15 BST; 1 day 23h ago
       Docs: https://k3s.io
    Process: 12309 ExecStartPre=/sbin/modprobe br_netfilter (code=exited, status=0/SUCCESS)
    Process: 12311 ExecStartPre=/sbin/modprobe overlay (code=exited, status=0/SUCCESS)
   Main PID: 12312 (k3s-server)
      Tasks: 57
     Memory: 1.0G
        CPU: 9h 28min 52.034s
     CGroup: /system.slice/k3s.service
             └─12312 "/usr/local/bin/k3s server"

Jun 18 08:23:56 truenas k3s[12312]: E0618 08:23:56.494581   12312 cri_stats_provider.go:666] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/kube-system_openebs-zfs-controller-0_8df7bb7a-121a-44b0-8fcc-c4f3d1c8bbee/csi-provisioner/5.log\": no such file>
Jun 18 08:23:56 truenas k3s[12312]: E0618 08:23:56.495597   12312 cri_stats_provider.go:666] "Unable to fetch container log stats" err="failed to get fsstats for \"/var/log/pods/ix-nextcloud_nextcloud-docker-compose-587678cf4-477b5_37a76189-a8a6-445a-8245-182e14429190/nextcloud-docker-com>
Jun 18 08:23:56 truenas k3s[12312]: time="2023-06-18T08:23:56+01:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jun 18 08:23:58 truenas k3s[12312]: time="2023-06-18T08:23:58+01:00" level=info msg="Slow SQL (started: 2023-06-18 08:23:57.440006911 +0100 BST m=+172307.021207976) (total time: 1.093036516s): INSERT INTO kine(name, created, deleted, create_revision, prev_revision, lease, value, old_value>
Jun 18 08:23:58 truenas k3s[12312]: I0618 08:23:58.534169   12312 trace.go:205] Trace[1596506823]: "GuaranteedUpdate etcd3" audit-id:aa7e7f93-7063-4d51-b4ba-c37407e5bdae,key:/leases/kube-system/external-resizer-zfs-csi-openebs-io,type:*coordination.Lease (18-Jun-2023 08:23:57.436) (total >
Jun 18 08:23:58 truenas k3s[12312]: Trace[1596506823]: ---"Txn call finished" err:<nil> 1096ms (08:23:58.534)
Jun 18 08:23:58 truenas k3s[12312]: Trace[1596506823]: [1.097594532s] [1.097594532s] END
Jun 18 08:23:58 truenas k3s[12312]: I0618 08:23:58.534430   12312 trace.go:205] Trace[1020029107]: "Update" url:/apis/coordination.k8s.io/v1/namespaces/kube-system/leases/external-resizer-zfs-csi-openebs-io,user-agent:csi-resizer/v0.0.0 (linux/amd64) kubernetes/$Format,audit-id:aa7e7f93-7>
Jun 18 08:23:58 truenas k3s[12312]: Trace[1020029107]: ---"Write to database call finished" len:732,err:<nil> 1097ms (08:23:58.534)
Jun 18 08:23:58 truenas k3s[12312]: Trace[1020029107]: [1.098219007s] [1.098219007s] END
lines 1-23/23 (END)

 

[Daisuke]

Jira tickets closed because iX do not want to maintain a custom build.

 

[ULTIoNT]

Is there any update on this? I am also seeing the log about no dbstat table all the time.

Edit: Now it seems that the issue is tracked here, while iX insists that spam logging doesn't cause a functionality degradation so it doesn't justify building sqlite with custom configs.

 

[Louis2]

I understand that iX does not want to maintain a custom build. However I wonder what is causing the problem:
- sql-lite or
- k3s or
- truenas or
- ?
If it is not truenas, then someone should generate a ticket for the responsible party

 

[CacheMeIfYouCan]

I have the same issue on Scale 23.10.0.1, both of my servers.

 

[specter9mm]

I have the same issue here. It is a bit frustrating that they consider this a non issue.