Update check suddenly stops working (13.0-U6.1)

[jaywest]

Not too long ago, maybe a few weeks, I upgraded to 13.0-U6.1 from the update just previous to that. Update seemed to go fine.

So just a couple days ago, I went to check for updates by clicking "check for updates" from the dashboard and the update screen appears. Used to, it would show my current train in the dropdown by default, and the blue circle appears while it scans for updates. It would then either show updates available or say 'no updates available'. Thats definitely not what has been happening the past few days.

Now, when it goes to the update screen, it does not show my current train - that is just blank. There's no spinning blue circle. It just sits there. A few minutes later, maybe 3 or 5 minutes, it has discovered my current train and shows it correctly (TrueNAS-13.0-Stable - Release train for TrueNAS 13.0 [release]), but further down it prints a message in red:

Unable to connect to url https://update-master.ixsystems.com/TrueNAS/TrueNAS-13.0-STABLE/LATEST: Automatic update check failed. Please check system network settings.

I have googled for that message, and the examples I see of people posting questions about it seem to always point to (as the message says) a fundamental network issue, such as "hey, I cant ping 8.8.8.8". But in my case, I can ping it. I can ping outside internet hosts by ip or nickname. As best as I can tell, networking IS working just fine on truenas (both for internal dns and external dns).

I have put that URL string in my laptop browser and from my home network (nas is local to that) I can get a response ("LATEST" file, appears to be json/xml.) and it looks to me like what I'd think is normal. So I don't think there is anything wrong at ix-systems update. This makes me suspect its something on my nas, but since I can get to that update url from other machines locally I am not sure where to look for what might be going awry. Can someone point me in a direction to look?

Thanks in advance for any suggestions!

 

[jaywest]

I should add, I can drop to a shell on the truenas core box and do
curl https://update-master.ixsystems.com/TrueNAS/TrueNAS-13.0-STABLE/LATEST
and it works, I get the update info displayed. So I am relatively confident that this is NOT a network issue, my system can definitely get out.

I'm stumped, any help appreciated :\

 

[joeschmuck]

Strange problem. My 13.0-U6.1 does show the train. And I'm sure you have rebooted your NAS already.

What version did you upgrade from?

The other thing you might try, which it's isn't really intrusive:
1) Backup your TrueNAS Configuration (you should not need it however it is good practice).
2) ASSUMING YOU DID NOT UPGRADE ZFS FEATURE FLAGS !!! go to step 3, If you did upgrade the feature flags, jump to step 6.
3) In the GUI, System -> Boot -> Select the three dot on the right for the previous version that you upgraded from and make it active.
4) Reboot.
5) Does the GUI work normally again for this one problem? Yes proves the upgrade caused the issue, No means you may have got it elsewhere and may never know. Regardless, keep going to step 6.
6) Download the current ISO of TrueNAS CORE 13.0-U6.1 and use that ISO to upgrade to the current version.
7) Does that fix the GUI issue? If Yes, Fantastic. If No, you may have a configuration issue, but I know you said the CLI worked.

If you desire you can reset the TrueNAS configuration to factory and then configure the bare minimum to get it on the internet to see if the problem can be corrected. I really hope just reinstalling form the ISO fixes the problem for you.

I have had problems and others have as well sometimes upgrading from the GUI internet update. I don't know the cause and honestly, it you tried it again it might even work. But I have never had a problem using the ISO upgrade method.

I'm not sure if 'ifconfig' will give you any valuable data, again since the CLI worked.

 

[jaywest]

Sorry it's taken me a bit to get back, I headed out of town the date of your reply and and will be back monday.

What did I upgrade from:

So, I had upgraded from 13.0-U6 to 13.0-U6.1 via the 'check for updates' gui, but cannot check for future updates once I got there.

No problem on config backups: In addition to having them sent to the system pool via a cron job (system is documented to do that automatically but doesn't), the directory where these config backups are kept is the source of a daily cloud-sync task to my google drive.

I hadn't upgraded zfs feature flags for at least the last 2 or 3 updates, but was considering doing just that last week. Glad I never got around to it!

I'm pretty hesitant to try the rollback you mention above while I am out of town. Once I get back, I'll give those steps a try.

I was hoping to track down the exact script or logs that were causing it and see if that shed light on a solution, but will try the steps above and advise. Thanks to all for input!

 

[joeschmuck]

You should be able to roll back to any of the 13.0 versions without issue. Make sure you backup that configuration file first to make life easier, just in case you need to use it. If you added/changed user accounts for example.

 

[jaywest]

So.....

I rolled back from 6.1 to 6. The gui behaves 'better' on the update page, but still has issues.

After reboot I click on 'check for updates' from the dashboard it goes to that page but no spinning blue circle, and it doesn't display my current train. I waited a good 5 or 10 minutes, and it just sat there.

I then clicked the 'refresh/tryagain' button (circle with arrow icon) and while it still doesn't show my current train, it does now show the update files. The files from->to looked correct, so figured I'd let the gui try to do the update and I then clicked 'download updates' and this is what I got:

Immediately after doing this, I popped to the shell on the machine and did curl https://update-master.ixsystems.com/TrueNAS/trueNAS-13.0-STABLE/LATEST and received the following output:

root@truenas[~]# curl https://update-master.ixsystems.com/TrueNAS/TrueNAS-13.0-STABLE/LATEST
{
"BuildTime": "1701731278",
"Packages": [
{
"Checksum": "d74fc8142b150f361aa7475fe0e7141219c7f42cb7c590ddc51f515ea9192986",
"FileSize": 888142752,
"Name": "base-os",
"RequiresReboot": true,
"Version": "13.0-U6.1-00016cd29a68bb56a924d307f65ff55c"
},
{
"Checksum": "bf720ab2ce199cb6db0e4e4a7a83694ba226535c8a2159c06f2cd5f4c95df686",
"FileSize": 6975760,
"Name": "freebsd-pkgdb",
"RequiresReboot": false,
"Version": "13.0-U6.1-00016cd29a68bb56a924d307f65ff55c"
},
{
"Checksum": "af1122c77e90d3498cf374c1630cae92d68654f75dfa1c332a15032912075ba8",
"FileSize": 58464,
"Name": "freenas-pkg-tools",
"RequiresReboot": false,
"Version": "13.0-U6.1-00016cd29a68bb56a924d307f65ff55c"
}
],
"Sequence": "00016cd29a68bb56a924d307f65ff55c",
"Signature": "NVDhxZn4GNNqNHlg0eZQjZi4Nw+TQV59OoukOsTeLkiZre9he3h5FBnpVj3Z1icgnyxCSIuTmFx6s8gDD64zOFpZ2wEeu9/BMx2teT5huow9GB2DmgiZSCWvq5LVyBh3mdQgxgVsrzOT7Yjl0Bv4rWKgDGh8JqBg8Q1rWQC1gl7KTpa1ePUdbv8qVfEzwe0BCcTc03p0uCQmrSo/8x8XkZc/bHgx++xVErtTEYlVTU/6mB2WxJenvqQ9SP5aaI8beYO4Ry5HS08OjFB2Yfba3MJFdCuDegqTVtapbBBNjgeF/Tj9yBIAR9Bodsid9l/HBjoqA1yIK79hyVAdp39qTg==",
"Train": "TrueNAS-13.0-STABLE",
"UpdateCheckProgram": {
"Checksum": "306c6ca7407560340797866e077e053627ad409277d1b9da58106fce4cf717cb",
"Kind": "ValidateUpdate",
"Name": "ValidateUpdate-ou4mdhxx.txt"
},
"Version": "TrueNAS-13.0-U6.1"
}#

Which, while I am not smart enough to know for sure - seems to indicate to me that my system can in fact communicate with the update server as expected, and its something other than a network issue?

I was thinking next I would try updating to 6.1 via the ISO method you mentioned, unless the above indicates a different approach? For the moment, I'm leaving the machine at 6 although I suspect I could just change the boot environment back to 6.1 (not that that helps).

Hopefully unrelated - I did see some other posts about this where the user didn't have a default gateway set in the general config. I dont either, but thats because my system gets all IP settings via DHCP upon boot. After DHCP, it definitely has a default gateway (just not stored in the general config) and definitely can get out to public systems from the command line.

 

[joeschmuck]

I would do the ISO method. Hopefully you are cloning the boot environment so you can return to a known good one, then try the ISO update. If that fails then do a full ISO install, then restore your config file.