kubeletes log rotation errors

[browntiger]

Is anyone else getting those log rotation errors in /var/log/k3s_daemon.log, related to zfs provisioning with openebs?

Code:

Dec 30 20:11:28 truenas k3s[6704]: E1230 20:11:28.780095    6704 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_90e6c164-8cd4-487e-8eac-a97d2aef4626/csi-provisioner/0.log\": failed to reopen container log \"87d0d98b540a0e0b9e402fb7ddd9ddaebba17ef3ec429b576f318cadae060d2d\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_90e6c164-8cd4-487e-8eac-a97d2aef4626/csi-provisioner/0.log" containerID="87d0d98b540a0e0b9e402fb7ddd9ddaebba17ef3ec429b576f318cadae060d2d"
Dec 30 20:11:28 truenas k3s[6704]: E1230 20:11:28.781475    6704 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="59acf4428794e2ed90b8c9198b96d116351f96d55062165473112ccf693f616e"
Dec 30 20:11:28 truenas k3s[6704]: E1230 20:11:28.781560    6704 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_90e6c164-8cd4-487e-8eac-a97d2aef4626/snapshot-controller/0.log\": failed to reopen container log \"59acf4428794e2ed90b8c9198b96d116351f96d55062165473112ccf693f616e\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_90e6c164-8cd4-487e-8eac-a97d2aef4626/snapshot-controller/0.log" containerID="59acf4428794e2ed90b8c9198b96d116351f96d55062165473112ccf693f616e"
Dec 30 20:11:28 truenas k3s[6704]: E1230 20:11:28.784084    6704 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="5e4e80544ba22c6f2d5573744afc283b180caabf3ebfa68010ded568f3b4e039"
Dec 30 20:11:28 truenas k3s[6704]: E1230 20:11:28.784192    6704 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-controller-0_90e6c164-8cd4-487e-8eac-a97d2aef4626/csi-resizer/0.log\": failed to reopen container log \"5e4e80544ba22c6f2d5573744afc283b180caabf3ebfa68010ded568f3b4e039\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-controller-0_90e6c164-8cd4-487e-8eac-a97d2aef4626/csi-resizer/0.log" containerID="5e4e80544ba22c6f2d5573744afc283b180caabf3ebfa68010ded568f3b4e039"
Dec 30 20:11:28 truenas k3s[6704]: E1230 20:11:28.795296    6704 remote_runtime.go:1168] "ReopenContainerLog from runtime service failed" err="rpc error: code = Unknown desc = docker does not support reopening container log files" containerID="9bfc8f1a4448a2d3edaa9ec315d12ad75b59f7cf712ea08c55fb01e6e3625c17"
Dec 30 20:11:28 truenas k3s[6704]: E1230 20:11:28.795374    6704 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-kxhcv_f4899758-5c51-44c1-8bfd-12a85f55296f/openebs-zfs-plugin/0.log\": failed to reopen container log \"9bfc8f1a4448a2d3edaa9ec315d12ad75b59f7cf712ea08c55fb01e6e3625c17\": rpc error: code = Unknown desc = docker does not support reopening container log files" path="/var/log/pods/kube-system_openebs-zfs-node-kxhcv_f4899758-5c51-44c1-8bfd-12a85f55296f/openebs-zfs-plugin/0.log" containerID="9bfc8f1a4448a2d3edaa9ec315d12ad75b59f7cf712ea08c55fb01e6e3625c17"
Dec 30 20:11:30 truenas k3s[6704]: time="2022-12-30T20:11:30-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Dec 30 20:11:35 truenas k3s[6704]: {"level":"warn","ts":"2022-12-30T20:11:35.274-0500","logger":"etcd-client","caller":"v3@v3.5.3-k3s1/retry_interceptor.go:62","msg":"retrying of unary invoker failed","target":"etcd-endpoints://0xc000437340/kine.sock","attempt":0,"error":"rpc error: code = Unknown desc = no such table: dbstat"}
Dec 30 20:11:35 truenas k3s[6704]: time="2022-12-30T20:11:35-05:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"

 

[notanumba]

Yes, i can confirm these erros:

Jan 20 20:35:00 nas k3s[12714]: time="2023-01-20T20:35:00+01:00" level=info msg="Using CNI configuration file /etc/cni/net.d/00-multus.conf"
Jan 20 20:35:00 nas k3s[12714]: E0120 20:35:00.335512 12714 container_log_manager.go:266] "Failed to rotate log for container" err="failed to rotate log \"/var/log/pods/kube-system_openebs-zfs-node-l8dmr_951822c8-edc2-42e3-9781-4fac17d75c21/openebs-zfs-plugin/0.log\": failed to rotate log \"/var/log/pods>

 

[notanumba]

Guess i found it:

Check the pool where your system is located an make sure it has free space available.
My pool was at 100% because i forgot to remove the snapshots from VM volumes i located there too, after i moved them.

System Settings \ Advanced:

After freeing space an restarting the service via shell it seems normal:

restart:
systemctl restart syslog-ng

check:
systemctl status syslog-ng.service

 

[cooki3monster]

Did not work for me, system dataset pool has a lot of space left. Those errors coming up every 10s in the journal.

 

[blacksteel75]

is there a solution for this? i see the same in my logs.

 

[fpc]

I'm getting the exact same error log, fresh installation and only turned on SMB Shares after creating a 16TB Pool...