Scale 22.12-RC.1- after upgrade no "available applications"

[bacardi8]

Hi Guys,

After my upgrade I run into some issues regarding apps.
I only have 3 apps installed (plex, emby, jellyfin), these 3 apps had the same issue as, mostly all you guys also had, that they stucked on deploying.

so after making some hotspath and kubernetes setting changes, ( see other threads in this forum) these 3 apps is back running.
They update fine, the deploy fine and they run fine.

The biggest issue, so far, I now have is that there is no available applications, no matter how many restart I make, and also restarting of middlewared dont
make any difference. All the catalog sync's runs through with succes, but available applications is empty and the blue wheel keeps on running.

Maybe some of you guys has same issue, or maybe had similar issues.
For now I think I will have to wait on the release coming out around 13-15 December, to see if this will make a difference and bring the apps back.

Cheers

 

[eroji]

I had to completely delete the ix-applications dataset and let it recreate it, along with all the applications. Hopefully a better migration method is available.

 

[bacardi8]

Hi eroji,

Ohh Shit,
Yes, let's hope there will be a better way to fix that.

Like I mentioned, I will wait and see how the upcoming release will handle it.

Thanks for your answer.

 

[Daisuke]

You have 2 pools? If yes, move ix-applications to a different pool (tick migrate apps), then move it back. See this post.

 

[samyapsul]

Updated to 22.12-RC.1 today and ran into the same issue with Available Applications infinitely spinning.
I got it working after unticking "Validate host path" under Kubernetes settings and then restarting middlewared via ssh.

 

[bacardi8]

Hi Daisuke,

Thanks for your answer and for the link to a good post.

I only have 1 pool, so I can't migrate the dataset to a different pool. I'm really new to Truenas and linux so I didn't know that about to
run on a dedicated software pool of dual SSDs, even I have SSD's available on my system.

So if I understand you right, then you would suggest me to delete the ix-application data-set on my existing pool, then create a new pool
for "software" by using 2 of same SSD's and then choose this pool for apps in the future..???

BTW....why you want to use 2 SSD's for this..??.......to run in "mirror"...??

Cheers

 

[Daisuke]

to run in "mirror"

Correct, you need dual SSDs for an applications pool. It will make a big difference. Plus, you can easily fix issues like this one. :) Technically, you will have a default pool with all your slow spinners where you store your archived data and a software pool with dual SSDs for your Kubernetes apps or VMs. You can run the OS on a single SSD, connected to an internal USB port.

 

[bacardi8]

Sounds good and make sense I think.

My biggest worry right now is just IF available applications still don't come up after I delete my existing ix-application data-set,
then I would not be able to install my apps again.

 

[Daisuke]

If you delete the ix-applications dataset, all apps will be gone. Since you have 2 available SSDs, create a software pool and move the dataset there, by choosing the newly created software pool like into screenshot. Your apps will live there from now on and you will not lose any existing settings, presuming there are no other issues.

 

[bacardi8]

Yes thats true, but for now I can at least run Plex, Emby, Jellyfin like it is.
Only I cant install more apps due to they are not there.

But anyway, I also don't get any further like it is now.

Thanks again,
I will let you know how I did and how it went

 

[bacardi8]

hmmm........I get a warning during the migration of the ix-application data-set to the new pool.
it's seems to go very very slow, and this warning :

• Device /dev/disk/by-partuuid/06dbf8ca-7189-4e18-9d56-4f32c144e275 is causing slow I/O on pool software

You know what that means...???

 

[bacardi8]

Something about smr drives are too slow.

I run WD Red 14TB plus drives, so that's not the case here.

 

[bacardi8]

Update :

Really really strange.........
3 brandnew SSD's 1TB each all comes up with the same error (slow I/O) during migration of ix-applications from main pool.
I analyzed one by one, no issues, I wiped them and all good. It looks like that the issue start showing up after migration of
around 20-30GB, I then used a fourth SSD (500GB) as single only, this migrate without issues.

But anyway.....after migration I still don't have any "available applications" so my issue seems to follow the ix-data from the main pool.

Only way left I think, is to delete ix-datasets from both pools and start over again.

 

[samyapsul]

Update :

Really really strange.........
3 brandnew SSD's 1TB each all comes up with the same error (slow I/O) during migration of ix-applications from main pool.
I analyzed one by one, no issues, I wiped them and all good. It looks like that the issue start showing up after migration of
around 20-30GB, I then used a fourth SSD (500GB) as single only, this migrate without issues.

But anyway.....after migration I still don't have any "available applications" so my issue seems to follow the ix-data from the main pool.

Only way left I think, is to delete ix-datasets from both pools and start over again.

Do you use truechart's catalog? I forgot to mention in my post that I also removed it and one of their apps. Apparently, it was seeking updates for an app that was no longer available in the catalog and that may have been part of the solution. Afterwards, I re-added the truecharts catalog and it works fine.

 

[kyle_macaulay]

Updated to 22.12-RC.1 today and ran into the same issue with Available Applications infinitely spinning.
I got it working after unticking "Validate host path" under Kubernetes settings and then restarting middlewared via ssh.

i love you i love you i love you i love you. LOL this is the only thing that has fixed my apps to successfully delpy after hours of troubleshooting.

 

[bacardi8]

Hi samyapsul,

yes that could also have been the case in my situation.
I deleted all ix-datasets and now all is up running as normal again.

 

[picklefish]

I've tried all of these hot tips and I get
`Error: [EFAULT] Failed to backup kubernetes cluster: [EFAULT] Unable to connect to kubernetes cluster`

This is super frustrating.

 

[picklefish]

Apps > Settings > Unset Pool
Restart System
Select pool

Fixed mine

 

[jhmw01]

I've tried almost everything to get the apps back up and running. I've unclicked host path, restart middlewared, unset and rebooted. My apps get stuck at deploy and give this message:
1 node(s) had untolerated taint {ix-svc-stop: }. preemption: 0/1 nodes are available: 1 Preemption is not helpful for scheduling.
Help!

 

[Daisuke]

describe node nodename will tell you exactly where is the issue:

Code:

# k3s kubectl get nodes
NAME         STATUS   ROLES                  AGE    VERSION
ix-truenas   Ready    control-plane,master   165d   v1.25.3+k3s-9afcd6b9-dirty
# k3s kubectl describe node ix-truenas