Unable to update CLI or Web GUI, DNS is resolving.

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
To all that may advise,

I have had this issue for a bit now. I have essentially just ignored until today. Its finally bugged me enough to deal with it. I cannot seam to get FreeNAS to update on the CLI or the WebGUI. I get these errors. From the WebGUI I get


Code:
Update server could not be reached

b''


Traceback
Traceback (most recent call last):
File "/usr/local/lib/freenasOS/Manifest.py", line 682, in RunValidationProgram
subprocess.check_output(valid_script, preexec_fn=PreExecHook, stderr=subprocess.STDOUT)
File "/usr/local/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command './tmp2bbz755h' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./freenasUI/system/views.py", line 1686, in update_check
train=updateobj.get_train(),
File "/usr/local/lib/freenasOS/Update.py", line 863, in CheckForUpdates
new_manifest.RunValidationProgram(cache_dir)
File "/usr/local/lib/freenasOS/Manifest.py", line 684, in RunValidationProgram
raise Exceptions.UpdateInvalidUpdateException(err.output.rstrip())


From the CLI using "freenas-update -v check" I get

Code:
% freenas-update -v check
[freenasOS.Configuration:601] TryGetNetworkFile(['https://update-master.ixsystem                                     s.com/FreeNAS/FreeNAS-11.2-STABLE/LATEST'])
[freenas-update:195] [Errno 13] Permission denied: '/var/db/system/tmpql4i1u77'
Traceback (most recent call last):
  File "/usr/local/bin/freenas-update", line 169, in DoDownload
    rv = Update.DownloadUpdate(train, cache_dir, pkg_type=pkg_type, ignore_space                                     =ignore_space)
  File "/usr/local/lib/freenasOS/Update.py", line 898, in DownloadUpdate
    latest_mani = conf.FindLatestManifest(train, require_signature=True)
  File "/usr/local/lib/freenasOS/Configuration.py", line 1090, in FindLatestMani                                     fest
    reason="GetLatestManifest",
  File "/usr/local/lib/freenasOS/Configuration.py", line 639, in TryGetNetworkFi                                     le
    retval = tempfile.TemporaryFile(dir=self._temp)
  File "/usr/local/lib/python3.6/tempfile.py", line 622, in TemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/local/lib/python3.6/tempfile.py", line 262, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: '/var/db/system/tmpql4i1u77'
Received exception during download phase, cannot update


I can ping dns.google.com which resolvs to 8.8.8.8 of course and I can ping update-master.ixsystems.com.

I'm chasing my tail here. Any ideas on what I should do short of reinstalling FreeNAS and restoring a config. That's such a short way out that doesn't hardly result in learning ways around FreeBSD or FreeNAS for that matter. Please advise.

Thanks in Advance,

DFTWD-ADMIN
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
Also I'm using 11.2 U6.
Just noticed a few minutes ago that the Plugins section page is no longer showing plugins.

Thanks
To all that may advise,

I have had this issue for a bit now. I have essentially just ignored until today. Its finally bugged me enough to deal with it. I cannot seam to get FreeNAS to update on the CLI or the WebGUI. I get these errors. From the WebGUI I get


Code:
Update server could not be reached

b''


Traceback
Traceback (most recent call last):
File "/usr/local/lib/freenasOS/Manifest.py", line 682, in RunValidationProgram
subprocess.check_output(valid_script, preexec_fn=PreExecHook, stderr=subprocess.STDOUT)
File "/usr/local/lib/python3.6/subprocess.py", line 356, in check_output
**kwargs).stdout
File "/usr/local/lib/python3.6/subprocess.py", line 438, in run
output=stdout, stderr=stderr)
subprocess.CalledProcessError: Command './tmp2bbz755h' died with <Signals.SIGABRT: 6>.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "./freenasUI/system/views.py", line 1686, in update_check
train=updateobj.get_train(),
File "/usr/local/lib/freenasOS/Update.py", line 863, in CheckForUpdates
new_manifest.RunValidationProgram(cache_dir)
File "/usr/local/lib/freenasOS/Manifest.py", line 684, in RunValidationProgram
raise Exceptions.UpdateInvalidUpdateException(err.output.rstrip())


From the CLI using "freenas-update -v check" I get

Code:
% freenas-update -v check
[freenasOS.Configuration:601] TryGetNetworkFile(['https://update-master.ixsystem                                     s.com/FreeNAS/FreeNAS-11.2-STABLE/LATEST'])
[freenas-update:195] [Errno 13] Permission denied: '/var/db/system/tmpql4i1u77'
Traceback (most recent call last):
  File "/usr/local/bin/freenas-update", line 169, in DoDownload
    rv = Update.DownloadUpdate(train, cache_dir, pkg_type=pkg_type, ignore_space                                     =ignore_space)
  File "/usr/local/lib/freenasOS/Update.py", line 898, in DownloadUpdate
    latest_mani = conf.FindLatestManifest(train, require_signature=True)
  File "/usr/local/lib/freenasOS/Configuration.py", line 1090, in FindLatestMani                                     fest
    reason="GetLatestManifest",
  File "/usr/local/lib/freenasOS/Configuration.py", line 639, in TryGetNetworkFi                                     le
    retval = tempfile.TemporaryFile(dir=self._temp)
  File "/usr/local/lib/python3.6/tempfile.py", line 622, in TemporaryFile
    (fd, name) = _mkstemp_inner(dir, prefix, suffix, flags, output_type)
  File "/usr/local/lib/python3.6/tempfile.py", line 262, in _mkstemp_inner
    fd = _os.open(file, flags, 0o600)
PermissionError: [Errno 13] Permission denied: '/var/db/system/tmpql4i1u77'
Received exception during download phase, cannot update


I can ping dns.google.com which resolvs to 8.8.8.8 of course and I can ping update-master.ixsystems.com.

I'm chasing my tail here. Any ideas on what I should do short of reinstalling FreeNAS and restoring a config. That's such a short way out that doesn't hardly result in learning ways around FreeBSD or FreeNAS for that matter. Please advise.

Thanks in Advance,

DFTWD-ADMIN
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
Please provide your system specs (as per the forum posting instructions.) It is not possible to comment without some idea about your system.

Are you booting from flash drive or SSD?
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
FreeNAS-11.2u6, Asrock 2750 32gig ECC
120GB SSD System boot. 4x 8TB Toshiba in RaidZ2, 4x TB in RaidZ2,
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
Are you able to log into the web GUI? It seems to act as if networking is not set up correctly. Are you running DHCP or did you assign a static IP address. What is the rest of your network?
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
Thank you for the response. I highly doubt its the network ths system has been running in this same configuration for better part of 2 years, however; that was my first knee jerk reaction. The system has access to the internet as my OP states. DNS and gateway config are correct. I can access the box from the network in both my personal net and server net.
This box exists in multiple subnets. There are 3 nics in this box.1 nic to a personal network static ip and is currently and actively serving SMB and AFP clients. This is the same subnet the main gateway is on which I access the webGUI on. The second and third nics are assigned to a LAGG0 which is actively hosting iSCSI lun for a Xenserver stack and a few other vlans which are secure and dedicated for some of the VM appliances. This primarily serves as the main Storage Repository for Xenserver. The box is getting out. Even when I place ixsystems.com and sub domains respectively in the hosts file for local resolving it still returns the same outcome. I can ping it but I cannot update.
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
ifconfig -a returns
Code:

Last login: Sun Feb  9 15:12:29 2020 from 172.30.1.27
FreeBSD 11.2-STABLE (FreeNAS.amd64) #0 r325575+5920981193f(HEAD): Mon Sep 16 23:00:13 UTC 2019

        FreeNAS (c) 2009-2019, The FreeNAS Development Team
        All rights reserved.
        FreeNAS is released under the modified BSD license.

        For more information, documentation, help or support, go here:
        http://freenas.org
Welcome to AWPNAS
Could not chdir to home directory /nonexistent: No such file or directory
% ifconfig -a
re0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=82099<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC,LINKSTATE>
        ether c4:e9:84:03:05:d2
        hwaddr c4:e9:84:03:05:d2
        inet 172.30.1.201 netmask 0xffffff00 broadcast 172.30.1.255
        inet 172.30.1.202 netmask 0xffffff00 broadcast 172.30.1.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether d0:50:99:d0:1e:29
        hwaddr d0:50:99:d0:1e:29
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether d0:50:99:d0:1e:29
        hwaddr d0:50:99:d0:1e:2a
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
        inet 127.0.0.1 netmask 0xff000000
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
        groups: lo
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether d0:50:99:d0:1e:29
        inet 10.10.20.6 netmask 0xffffff00 broadcast 10.10.20.255
        inet 10.10.20.7 netmask 0xffffff00 broadcast 10.10.20.255
        inet 10.10.20.9 netmask 0xffffff00 broadcast 10.10.20.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        groups: lagg
        laggproto lacp lagghash l2,l3,l4
        laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
        laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
vlan3: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=600303<RXCSUM,TXCSUM,TSO4,TSO6,RXCSUM_IPV6,TXCSUM_IPV6>
        ether d0:50:99:d0:1e:29
        inet 10.10.6.6 netmask 0xffffff00 broadcast 10.10.6.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet autoselect
        status: active
        vlan: 3 vlanpcp: 0 parent interface: lagg0
        groups: vlan
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 02:7e:6d:57:1b:00
        nd6 options=1<PERFORMNUD>
        groups: bridge
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0:2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 9 priority 128 path cost 2000
        member: vnet0:1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 8 priority 128 path cost 2000
        member: re0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 1 priority 128 path cost 20000
vnet0:1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: zoneminder as nic: epair0b
        options=8<VLAN_MTU>
        ether c4:e9:84:40:ed:c2
        hwaddr 02:21:10:00:08:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
vnet0:2: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: bru-server as nic: epair0b
        options=8<VLAN_MTU>
        ether c4:e9:84:19:da:52
        hwaddr 02:21:10:00:09:0a
        nd6 options=1<PERFORMNUD>
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        groups: epair
%
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
So I'm not understanding. Is this a technical forum for assistance or is this a technical forum for waiting on ppl to get a round to it. I'm sure I'm not the only one that has had a this issue and if the issue is indeed network related then I would assume someone would pride themselves on stating I'm an idiot or help me and get on with it. So again I ask with upmost respect to this community, can someone please assist me in resolving this issue? I'd hate to make a multi million dollar decision over a product that has a lackadasical support community associated with it.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
So I'm not understanding. Is this a technical forum for assistance or is this a technical forum for waiting on ppl to get a round to it. I'm sure I'm not the only one that has had a this issue and if the issue is indeed network related then I would assume someone would pride themselves on stating I'm an idiot or help me and get on with it. So again I ask with upmost respect to this community, can someone please assist me in resolving this issue? I'd hate to make a multi million dollar decision over a product that has a lackadasical support community associated with it.
No clue what your problem is. Maybe you should be paying someone to fix your issues if there is millions of dollars on the line.

You can try cleaning up var/db/system/tmpql4i1u77 and starting the upgrade over. Permission denied error seems like an easy thing to fix. Just delete the tmp file.

I do know you could download the manual update file and upgrade using that. It's on offline upgrade so no internet connection needed.
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
It won't take the manual upgrades either same complaint from the system. in addition, var/db/system/tmpql4i1u77 does not exist as you can see from the snippet below. ls from /var/db/system/ directory. The directory is owned by root:wheel and 755. Seems correct. Also the Manifest file shows version as 11.3 if i'm understanding the way the manifest works. I have also attached the code below for whats written in the Manifest file.

Snipit var-db-system-LS.JPG


MANIFEST file in /var/db/system/update/
Code:
{
    "BuildTime": "1580156629",
    "Packages": [
        {
            "Checksum": "042de96d2447c81787b1f7d558f944d11c774c1702c8aad20af499343a0cd2c3",
            "FileSize": 619121587,
            "Name": "base-os",
            "RequiresReboot": true,
            "Version": "11.3-RELEASE-a67e6fa145e21a7cf074b3ec4e95676c"
        },
        {
            "Checksum": "35d2c806af8c0832a87aee8d8eab12e27df381a430f8a17f44fdbc97c24ea6c7",
            "FileSize": 7574821,
            "Name": "freebsd-pkgdb",
            "RequiresReboot": false,
            "Version": "11.3-RELEASE-a67e6fa145e21a7cf074b3ec4e95676c"
        },
        {
            "Checksum": "4c35b27ae18c454801edccc0ff063892df806432f78f6ab9881ca7d34b78bd5e",
            "FileSize": 58342,
            "Name": "freenas-pkg-tools",
            "RequiresReboot": false,
            "Version": "11.3-RELEASE-a67e6fa145e21a7cf074b3ec4e95676c"


Not sure if this states information about the forward update or what is currently installed. if what is currently installed is the answer then this file is not correct. Tried again to manually install 11.3 and got the same results. snippet below for version proof.

version.JPG
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
If I were in your situation, I would remove the FreeNAS machine from your complicated network, pick up another boot device and then build a new system from scratch. At least, that would rule out a hardware problem. It is not possible to offer more specific advice from the information you have given. Disconnect your data drives to make certain you don't accidentally harm your data.

I suggested earlier that it may be a network problem, and I still think it could be. However, it could also be a permissions problem created by messing around with files at a low level. A new install should get around all of that.
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
Before I start the rebuild process, does any one else have any other suggestions? I would really like to figure this out correctly so that one can learn from this issue by proper troubleshooting channels and possibly help any one else that may find them self's in this same predicament.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Click the verify install bottom in the gui. See if it fixed anything.

Create a new install USB and then upload your config. If nothing else works.
 

Fredda

Guru
Joined
Jul 9, 2019
Messages
608
Welcome to AWPNAS
Could not chdir to home directory /nonexistent: No such file or directory
% ifconfig -a
....
% freenas-update -v check
[freenasOS.Configuration:601] TryGetNetworkFile(['https://update-master.ixsystems.com/FreeNAS/FreeNAS-11.2-STABLE/LATEST'])
[freenas-update:195] [Errno 13] Permission denied: '/var/db/system/tmpql4i1u77'
This looks very strange. Your prompt looks not as the root prompt, no home directory.

Are you sure, you are trying to update as root? Only root can update.
When I try as non-root I get exactly the same error as you.
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
Yes the attempts are from root as much as it goes against all knowledge of unix based systems and security but its what it is.
Update:
I did obtain a new SSD and as suspected the restoration of the config with a fresh install resolves the entire scenario which was always understood as an option. I did install and restore to the same version as the original issue so it's still booting 11.2 u7. I'm gonna let it run a few days as marked stable before attempting to update to see if the issue arises again. I would like to keep this open as I venture into booting this original issue into a VM environment and attempting to figure out what caused this. Maybe #Fredda is on to something. What if the root no longer had root privileges per say. Is that a heard of thing? If so it would have to be rare I'd think and again if so how does that even happen?
 

Fredda

Guru
Joined
Jul 9, 2019
Messages
608
I think there must have been something corrupt with you install.
  • Cannot change into HOME of root. /nonexistant is for some functional accounts who never need to log in.
  • Prompt is not the usual root prompt. But the cause of that might be missing ~/.cshrc
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
I think there must have been something corrupt with you install.
  • Cannot change into HOME of root. /nonexistant is for some functional accounts who never need to log in.
  • Prompt is not the usual root prompt. But the cause of that might be missing ~/.cshrc
Wonder if he changed the root default shell, that's a big no no.
 

dftwd-admin

Dabbler
Joined
Feb 9, 2020
Messages
13
I wouldnt rule that out based on the complexity of the issue; however, I would like to think I have enough base unix knowledge not to mess with the root shell. Any and all system adjustments are done from the WebGUI anyways. also I cannot recall doing anything with the system from the cli nor can I think of any reason why one would even consider such changes to the root shell. But who know at this point. Maybe I've been hackzored. Still yet to be determined.
 
Top