Cannot install any apps or docker images from any sources, kubernetes issue?

[HeroRareheart]

I've made 2 previous posts of poor quality while EXTREAMLY angry over this and I'm going to do my best to make a third one that is properly written and HOPE I get help this time. 2 weeks ago I deployed TrueNAS Scale 22.02.0 to my dell r710 server. the setup went fine and everything was good until I went to install apps. I added the TrueCharts catalog and nothing from that catalog or the official one would install and after troubleshooting with TrueCharts support discord I found despite nobody in the TrueCharts discord needing to do this I had to go to the advanced settings in the settings dropdown menu on the apps screen and select an interface and set gateway IP then restart the system in order to download apps. I left everything for a few days before trying to deploy the romses/mineos dockerhub image and it failed and I assumed I didn't know enough about docker. After a full week of watching tutorials and reading documentation I discovered I'm probably having an issue with kubernetes. I've clean installed TrueNAS Scale 22.02.0 from a new piece of install media created with a fresh install image to rule out the installer messing up, and also because statisfactory stopped functioning out of nowhere, and after a through check of everything I'm having the exact same issue and can't fix it this time. I've attached the Application Events log and all logs generated, unless empty, that's created when I try to deploy Plex from the official catalog. All TrueCharts apps generate the same logs. If more information is needed please let me know, I'll be adding my system specs to my signature soon.

Just tried to get the log it generated but it's empty when I download it and so I took the text in it's terminal and pasted it into a file which I've called a log add added a screenshot of the failed task manager task for it.

 

[HeroRareheart]

Also screw it here's my entire debug, might as well.

 

[HeroRareheart]

Slight update, I was able to work around the issue buy pulling the image I want from the terminal then deploying it but after a restart all the installed apps are gone. This is not reliable and if I can't get things working I'm going to need to switch my OS.

 

[morganL]

Slight update, I was able to work around the issue buy pulling the image I want from the terminal then deploying it but after a restart all the installed apps are gone. This is not reliable and if I can't get things working I'm going to need to switch my OS.

Sorry to hear you are having so many issues. It's a complex system and difficult for anyone to diagnose remotely.. especially if the symptoms are abnormal. You might have 2 configuration issues and a bug... the symptoms would be very weird.

My suggestion is to do the following:

1) describe your high level goals for the system...services, Apps, networking - people can review the plan and advise on how to get there.

2) Start from a base system that works (verify the hardware) and proceed one step at a time

3) Test each step that is testable.. eg networking is working, specific container is working

4) Stop at the 1st issue and diagnose that one issue --- a system with multiple issues is extremely hard to diagnose

5) where you have an issue, document how you configured the system... what were the steps taken for that issue to occur

If you do this, its possible to work out whether its a configuration issue or a bug.. If it is a bug, is it a known one with a workaround or a new one? Why were you the 1st to find that bug and how could it be avoided?

The ramses/mineos app you are running .. are you sure it's suitable for running on Kubernetes? It seems to require root access and apt-get. Should it be run as a VM? I haven't had to work though this issue before, perhaps someone else has reviewed and can chime in.

 

[HeroRareheart]

1) describe your high level goals for the system...services, Apps, networking - people can review the plan and advise on how to get there.

I want to use the system as an as and store files on it such as music, Word Documents, and pictures so that I can easily access them from all of my systems at home. I also want to run the statisfactory app from TrueCharts so my friends can play Satisfactory with me without me hosting from my main rig. I want to deploy MineOS so I can get the Minecraft server we all play on back up and running. I'd like to deploy pihole as well. Preferably I would like each one of these things to use a separate interface but at the moment I just want things to work so I will not worry about that right now. It would also be nice to be able to VPN into my home network and access my files from anywhere but again I will not worry about this right now as I just want the system to function first.

2) Start from a base system that works (verify the hardware) and proceed one step at a time

I know that everything worked fine when I had Ubuntu server installed so the hardware is good and upon a fresh install of TrueNAS without configuring anything other than a storage pool and a user I am able to connect to that storage pool from my Windows rig. Without touching any of the configuration though deploying apps fails with the same errors and I'm unable to ping outside websites from the shell indicating that network configuration is necessary. I am uncertain if the network configuration I have set up is correct but after setting it up as shown in the attached screenshots I can add new catalogs successfully to the apps tab and I can ping outside websites from the shell such as google.com however apps still fail to deploy with the same errors. I'm going to attach the error logs and stop here and wait for a suggestion because from what other people have said to me it should just work at this point. I'm also going to include my system debug again because although that's probably not helpful it won't hurt.

The ramses/mineos app you are running .. are you sure it's suitable for running on Kubernetes? It seems to require root access and apt-get. Should it be run as a VM? I haven't had to work though this issue before, perhaps someone else has reviewed and can chime in.

The instructions included with the image are copied and pasted from the original Docker image and have absolutely nothing to do with actually deploying the docker. I did figure out what options I was supposed to include to get it to deploy and while my work around was still working I was able to get it to deploy just fine. Regardless focusing on getting statisfactory to deploy should probably be my next step.

 

[HeroRareheart]

Update, out of nowhere after leaving everything the same statisfactory launched itself successfully somehow. That said the log is still empty and now gives me no errors saying it can't get the log for one reason or another. The new application events log is attached:

 

[morganL]

Update, out of nowhere after leaving everything the same statisfactory launched itself successfully somehow. That said the log is still empty and now gives me no errors saying it can't get the log for one reason or another. The new application events log is attached:

Ok, so the question is whether Statisfactory app is healthy - both software and Config. Has anyone else deployed?

Do any other more common Apps deploy as expected?

 

[ian351c]

I'm noticing two things:
A - You have IPv6 addresses and gateways configured
2 - Your errors in the Apps seem to be with fetching docker images

So... this might be a long shot but: maybe try deleting the IPv6 addresses and routes on all of your network interfaces (and/or uncheck the "autoconfigure IPv6" box) (Also, why does it look like they're all plugged in but only one is being used?). I know many operating systems will prioritize IPv6 over IPv4, and if TrueNAS/Linux thinks it can fetch the docker images via IPv6, it might try. And if that fails (which is likely), it may not retry on IPv4. I haven't run across that on TrueNAS (I've never had any IPv6 info assigned to any of my NAS interfaces) but I have run across this on OSes that try to auto configure IPv6 (Windows/MacOS) when an IPv6 capable router is on the network (whether it can successfully route IPv6 traffic or not).

Good luck, fellow MineOS/Satisfactory fan!

 

[HeroRareheart]

Ok, so the question is whether Statisfactory app is healthy - both software and Config. Has anyone else deployed?

Do any other more common Apps deploy as expected?

No other apps work and like a few days ago others in the TrueCharts discord got it to deploy.

 

[HeroRareheart]

try deleting the IPv6 addresses and routes on all of your network interfaces

I'll give that a shot.

Also, why does it look like they're all plugged in but only one is being used?

Because that's how I configured it. I did that to cut back on possible networking issues while troubleshooting.

I have run across this on OSes that try to auto configure IPv6 (Windows/MacOS) when an IPv6 capable router is on the network

the router the server is plugged into has ipv6 support turned off but my main router has it on... I wonder if that's the issue... I will try turning on ipv6 support on that router after I try the other two steps.

 

[morganL]

I'll give that a shot.

Because that's how I configured it. I did that to cut back on possible networking issues while troubleshooting.

the router the server is plugged into has ipv6 support turned off but my main router has it on... I wonder if that's the issue... I will try turning on ipv6 support on that router after I try the other two steps.

If most standard Apps are not deploying reliably, then networking would seem a likely cause.

 

[morganL]

I'll give that a shot.

Because that's how I configured it. I did that to cut back on possible networking issues while troubleshooting.

the router the server is plugged into has ipv6 support turned off but my main router has it on... I wonder if that's the issue... I will try turning on ipv6 support on that router after I try the other two steps.

If most standard Apps are not deploying reliably, then networking would seem a likely cause.

 

[HeroRareheart]

A - You have IPv6 addresses and gateways configured...try deleting the IPv6 addresses and routes on all of your network interfaces

Uhh according to the network config screen I don't but the dashboard tile says I do... wtf... anyhow if I delete the config and make it again it still has an IPV6 address no matter what I do and the config is the same as with DHCP. I will move on to configuring the router to use ipv6 and see if that fixes things.

 

[HeroRareheart]

Uhh I assume this is a normal config for IPV6, I've never messed with it though and it's very different. Any tips?

 

[HeroRareheart]

Ahh damn, same errors... I'm going to remove the 2nd router from the equation in the morning and have 1 interface hooked straight into the main router. If this fails I think it's time to admit that some rocks we tricked into thinking have kicked my ass. I think my pulling via the shell work around is functioning though as statisfactory will deploy given time but still spits out RPC errors and nothing else will deploy no mater how much time I give it.

 

[HeroRareheart]

WHAT THE HELL. Nothing was working and I tried adding an interface and gateway IP to Kubernetes which didn't work the last half dozen times and now it's just, magically fine... I think this must be a TrueNAS Scale bug at this point.

 

[HeroRareheart]

Ok I'm 100% done now. It had all magicked itself better and now magicked itself broken again, this is 100% a TrueNAS Scale bug. One of my deployed docker images is magickly working so I'm purging the other ones, submitting a bug report, and not touching anything until the next update. If it's still broken by the next major update I'm throwing out using TrueNAS Scale until it's out of Beta.