SOLVED TrueNas Scale Server cannot sync the catalog and I tried everything

[Johnephen]

I have a TrueNas Scale Server and I have the problem that it cannot update itself with the catalog. I had this problem a while ago and since there wasn't much on the server, I simply reinstalled the server. But now that's a bad solution. That's why I wanted to ask what can be done. I checked the bios time, it should match the system and the real time (although some notifications are sent with a difference of 2 hours) and both my router and the server have 1.1.1.1 as DNS server and I have 1.0.0.1 and 8.8.8.8 also tested.

the Error:

Critical​

Failed to sync TRUENAS catalog: [EFAULT] Failed to clone 'https://github.com/truenas/charts.git' repository at '/mnt/main/ix-applications/catalogs/github_com_truenas_charts_git_master' destination: Cmd('git') failed due to: exit code(128) cmdline: git clone -v https://github.com/truenas/charts.git /mnt/main/ix-applications/catalogs/github_com_truenas_charts_git_master stderr: 'Cloning into '/mnt/main/ix-applications/catalogs/github_com_truenas_charts_git_master'... POST git-upload-pack (175 bytes) POST git-upload-pack (gzip 1617 to 846 bytes) error: RPC failed; curl 56 Recv failure: Connection timed out error: 7456 bytes of body are still expected fetch-pack: unexpected disconnect while reading sideband packet fatal: early EOF fatal: fetch-pack: invalid index-pack output '​

2023-10-19 12:35:16 (Europe/Berlin)

Critical​

Failed to sync TRUECHART catalog: [EFAULT] Failed to clone 'https://github.com/truecharts/catalog' repository at '/mnt/main/ix-applications/catalogs/github_com_truecharts_catalog_main' destination: Cmd('git') failed due to: exit code(128) cmdline: git clone -v https://github.com/truecharts/catalog /mnt/main/ix-applications/catalogs/github_com_truecharts_catalog_main stderr: 'Cloning into '/mnt/main/ix-applications/catalogs/github_com_truecharts_catalog_main'... fatal: unable to access 'https://github.com/truecharts/catalog/': Could not resolve host: github.com '​

2023-10-19 12:35:17 (Europe/Berlin)

 

[LarsR]

Is your bios time set to UTC? The local timezone should only be set in the gui, not the bios.
In addition is your default route set (your routers ip)?

 

[probain]

Are you behind a privacy-VPN or similar?
I had to go through some hoops due to that. See my Jira-ticket below, and see if that could help maybe?
Jira-link

 

[Johnephen]

Is your bios time set to UTC? The local timezone should only be set in the gui, not the bios.
In addition is your default route set (your routers ip)?

The gui time, under Truenas in the operating system, is in the correct time zone and also set correctly. And yes, my default router is set up.

 

[probain]

The gui time, under Truenas in the operating system, is in the correct time zone and also set correctly. And yes, my default router is set up.

Easy ways to test it. Since system time and truenas time can actually differ.

Open a command Shell. And type

Code:

date

Do you get the correct time there too?
I had this the other day. Inside Truenas I had correct time.. But the underlying system time was six months old

 

[Johnephen]

Easy ways to test it. Since system time and truenas time can actually differ.

Open a command Shell. And type

Code:

date

Do you get the correct time there too?
I had this the other day. Inside Truenas I had correct time.. But the underlying system time was six months old

Yes, after the "date" command the time is actually shifted two hours. it should be 21:55 but it says 23:55

 

[Johnephen]

Are you behind a privacy-VPN or similar?
I had to go through some hoops due to that. See my Jira-ticket below, and see if that could help maybe?
Jira-link

I use an Ubuntu VM under TrueNas with easy-wg, because it caused problems under TrueNas itself after reinstalling easy-wg and yes, after I switched off the VM and restarted the system, the Available Applications overview works again. Do I really have to turn off my VM every time I want to update my applications or install new ones?

 

[Johnephen]

and I try to install wg-easy as an application again, but it has only been deploying for 5 minutes and is stuck there. Could that also fix the problem? It would definitely be better than running a VM just because of wg-easy.

and I gave the VM a dedicated Lan port via TrueNas under VM->Devices -> PCI.

 

[probain]

Well, you should probably try and set the correct time in the shell too. At least it won't make things worse.
I had good results in that regard using timedatectl in the cli/shell
source

sudo timedatectl set-time YYYY-MM-DD HH:MM:SS

...where YYYY is the year, MM is the month, and DD is the day, HH is the hour, MM is the minutes, SS is the seconds.
e.g..

sudo timedatectl set-time "2000-01-01 23:00:01"

Did you read my JIRA-ticket I mentioned previously/above?
In there I listed how I fixed my issues with getting catalogues behind a privacy-VPN. As I had to adjust a little how git was configured.
There are references there as well to other forumposts on stackoverflow, where similar issues are discussed.

Hope that it could be somewhat helpful in trying to fix the issues at least.

 

[Johnephen]

Did you read my JIRA-ticket I mentioned previously/above?
In there I listed how I fixed my issues with getting catalogues behind a privacy-VPN. As I had to adjust a little how git was configured.
There are references there as well to other forumposts on stackoverflow, where similar issues are discussed.

Hope that it could be somewhat helpful in trying to fix the issues at least.

I used the command that was recommended there ("git config --global core.compression 0"), but now I have errors in my networks and the applications don't work with the VM running, nor does connecting to Github.

 

[probain]

I used the command that was recommended there ("git config --global core.compression 0"), but now I have errors in my networks and the applications don't work with the VM running, nor does connecting to Github.

How or why git would affect your network connections, is beyond me.

 

[Johnephen]

I don't know either.
But I tested this command ("git config --global --unset core.compression") to reset that but it's still like that.

 

[probain]

Looking at your picture. It seems you have multiple IP's on the same LAN-subnet? I wouldn't have it that way personally. Makes it difficult to rule out routing issues.
But you could otherwise try and troubleshoot from your backup-config. If you can't get around the weird network issues that popped up. But again, that is for git and how git communicates. And really shouldn't affect your NIC's.

But you could try and adjust to different values for core.compression. -1 is the ZLIB-default, 1-9 is how aggressive the compression should work. 0 is obviously 'off'. To see if that alleviates it.
source

 

[probain]

Or just comment out/delete the line in .gitconfig in the user homefolder you where in, when you invoked the command. E.g. /root/.gitconfig

Or you can edit it directly with

git config --edit --global

 

[Johnephen]

Looking at your picture. It seems you have multiple IP's on the same LAN-subnet? I wouldn't have it that way personally. Makes it difficult to rule out routing issues.
But you could otherwise try and troubleshoot from your backup-config. If you can't get around the weird network issues that popped up. But again, that is for git and how git communicates. And really shouldn't affect your NIC's.

But you could try and adjust to different values for core.compression. -1 is the ZLIB-default, 1-9 is how aggressive the compression should work. 0 is obviously 'off'. To see if that alleviates it.
source

One NIC is connected to my local network and I use the second NIC directly with a 2.5G card in my computer. The third Nic was actually for the VM, but I have now simply changed it. Now the VM with wg-easy also runs over the first NIC and only one is connected to the router. but that didn't help to fix my problem either.

 

[probain]

And you have removed/disabled the configs for the NICs too? Not just taken out the cables?
If you then want the VMs to be able to communicate with the Truenas Scale hypervisor host over a shared NIC. You have to use a bridge.

But basically. Try and simplify your config. And add features i.e. complexity one step at a time.

 

[Johnephen]

It looks like completely deleting the NIC 3 settings fixed it. I won't be able to test it further until later, but it looks good so far. I'll get back to you in 1-2 hours. Thanks in advance for all the help

 

[Johnephen]

OK, here's my update: It works and I'm using wg-easy outside the VM again. The solution was that I broke my Docker installation of wg-easy from Truecharts a while ago and, to be on the safe side, I named the application "wg-easy-new" and created the host directory by hand so that it doesn't automatically use incorrect files who doesn't want to start the application. No matter whether via VM or when the application is switched on, there are no longer the errors that I stated at the beginning of the thread and I don't have to run a VM just for a VPN. Thank you @probain again for your help, helpful thoughts and your patience, as I can imagine that with the information I have given here, analyzing the problem is extremely challenging. Have a wonderful day and a lovely weekend.

 

[probain]

Happy to help and that you got it sorted out