k3s isn't deploying after pool migration

gavinlee12

Cadet
Joined
Mar 8, 2024
Messages
4
Hi guys,

Currently, my k3s instance is in shambles at the moment after I did a pool migration. The story is that I initially migrated the k3s pool from the Data pool to the Root pool by setting the k3s pool to Root. This resulted in the k3s instance being unable to deploy any of my containers as some of the volumes were referencing the Data pool.

I then reimport and renamed the Root pool to the Data pool. Some containers are up unfortunately some of the containers aren't starting as some of the volumes are in read-only mode:

Code:
MountVolume.SetUp failed for volume "pvc-0b537fb0-d4c4-48e7-b566-33f806651c8e" : rpc error: code = Internal desc = rpc error: code = Internal desc = zfs: mount failed err : not able to mount, cannot mount '/mnt/var/lib/kubelet/pods/a13c8fe3-c213-4dfe-b20d-2536d6c8c7b8/volumes/kubernetes.io~csi/pvc-0b537fb0-d4c4-48e7-b566-33f806651c8e/mount': failed to create mountpoint: Read-only file system 


I checked the zfs datasets, zfs pools and also the mount command and from the looks of it none of them are in read-only mode:

Code:
root@TreeNAS[/home/admin]# zfs list -o name,readonly | grep -we on
root@TreeNAS[/home/admin]#


Code:
root@TreeNAS[/home/admin]# zpool list -o name,readonly
NAME       RDONLY
Data       off
Media      off
boot-pool  off


Code:
root@TreeNAS[/home/admin]# mount | grep -w ro
ramfs on /run/credentials/systemd-sysusers.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup-dev.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-sysctl.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)
ramfs on /run/credentials/systemd-tmpfiles-setup.service type ramfs (ro,nosuid,nodev,noexec,relatime,mode=700)


Is someone able to point me in the right direction or list other troubleshooting steps for this issue?
 
Top