Apps Stuck Deploying

[Chankster]

After an unexpected power loss I've got several apps that are stuck in the deploying state. Logs indicate:

Code:

Jul 13 14:27:19 nas k3s[292203]: E0713 14:27:19.166040  292203 nestedpendingoperations.go:301] Operation for "{volumeName:kubernetes.io/csi/zfs.csi.openebs.io^pvc-b24cb2f9-d9df-40c1-ab3a-4dfd2262b1b0 podName: nodeName:}" failed. No retries permitted until 2021-07-13 14:27:20.16600941 -0500 CDT m=+64415.824312683 (durationBeforeRetry 1s). Error: "MountVolume.SetUp failed for volume \"pvc-b24cb2f9-d9df-40c1-ab3a-4dfd2262b1b0\" (UniqueName: \"kubernetes.io/csi/zfs.csi.openebs.io^pvc-b24cb2f9-d9df-40c1-ab3a-4dfd2262b1b0\") pod \"gaps-6b48448d8d-c5trs\" (UID: \"13867a54-01c3-4c7c-8a5f-700451b4b891\") : rpc error: code = Internal desc = rpc error: code = Internal desc = zfs: mount failed err : not able to mount, cannot mount '/mnt/var/lib/kubelet/pods/13867a54-01c3-4c7c-8a5f-700451b4b891/volumes/kubernetes.io~csi/pvc-b24cb2f9-d9df-40c1-ab3a-4dfd2262b1b0/mount': failed to create mountpoint: Read-only file system\n"

 

[morganL]

Are some apps OK and some Apps having problems?

If so, better to list the Apps with issues.

 

[Chankster]

Plex is ok but everything else I've tried is not.

 

[Chankster]

Seems that anything using a PVC has the issue I listed above.

 

[ornias]

It seems you are using TrueCharts, please be aware we have our own support channels as listed on our website.
Your issue is known and the cause and fixes are discussed on our discord.

 

[waqarahmed]

@Chankster can you please email me a debug of your system at waqar at the rate of ixsystems.com ? We have a known issue which has been fixed in the nightlies but i would still like to confirm and make sure that's indeed the case. Thank you