All apps with a PVC are stuck in "Deploying" and never start. The Application Events Log shows the following, which suggests there's a mountpoint problem:
2021-11-18 23:42:53
MountVolume.SetUp failed for volume "pvc-2e971231-3c61-4d34-969b-70f60029c7fb" : rpc error: code = Internal desc = rpc error: code = Internal desc = zfs: mount failed err : not able to mount, cannot mount '/mnt/var/lib/kubelet/pods/5927b158-0ba2-46d8-9e8a-4c70c8471aad/volumes/kubernetes.io~csi/pvc-2e971231-3c61-4d34-969b-70f60029c7fb/mount': failed to create mountpoint: Read-only file system
Changing the mountpoint manually to "legacy" was suggested by @ornias and it resolves the issue temporarily e.g. if you apply a command similar to this:
...the app will start again.
Unfortunately, a reboot reverts the change back to the broken state.
This affects new app installations, as well as existing app installations, so simply reinstalling the apps does not resolve the issue and the system seems to be in a permanently broken state.
The system got into this state while reinstalling TrueNAS and restoring a saved configuration file.
In the debug attached to the jira report, the TrueCommand & Plex apps have had the above workaround applied. Mealy was installed after this issue began. All other apps are in their pre-reinstallation state.
If there's a way to resolve the issue without a complete new system reinstallation, I'd appreciate the advice.
I have reported as: https://jira.ixsystems.com/browse/NAS-113416?filter=-2
This issue seems similar to this posting @waqarahmed : https://www.truenas.com/community/threads/apps-stuck-deploying.94217/
2021-11-18 23:42:53
MountVolume.SetUp failed for volume "pvc-2e971231-3c61-4d34-969b-70f60029c7fb" : rpc error: code = Internal desc = rpc error: code = Internal desc = zfs: mount failed err : not able to mount, cannot mount '/mnt/var/lib/kubelet/pods/5927b158-0ba2-46d8-9e8a-4c70c8471aad/volumes/kubernetes.io~csi/pvc-2e971231-3c61-4d34-969b-70f60029c7fb/mount': failed to create mountpoint: Read-only file system
Changing the mountpoint manually to "legacy" was suggested by @ornias and it resolves the issue temporarily e.g. if you apply a command similar to this:
Code:
root@scale[~]# zfs set mountpoint=legacy ixapps/ix-applications/releases/truecommand/volumes/pvc-7d9e5ab6-e00d-4240-a795-08edf93baac0
...the app will start again.
Unfortunately, a reboot reverts the change back to the broken state.
This affects new app installations, as well as existing app installations, so simply reinstalling the apps does not resolve the issue and the system seems to be in a permanently broken state.
The system got into this state while reinstalling TrueNAS and restoring a saved configuration file.
In the debug attached to the jira report, the TrueCommand & Plex apps have had the above workaround applied. Mealy was installed after this issue began. All other apps are in their pre-reinstallation state.
If there's a way to resolve the issue without a complete new system reinstallation, I'd appreciate the advice.
I have reported as: https://jira.ixsystems.com/browse/NAS-113416?filter=-2
This issue seems similar to this posting @waqarahmed : https://www.truenas.com/community/threads/apps-stuck-deploying.94217/
