Upgrade from TrueNAS-22.02.1 -> TrueNAS-22.02.2 failing with 'Devlinks'

[Xero7650]

I tried kicking off an upgrade to scale 22.02.2 today, and keep getting error 'Devlinks' when I try. I rebooted the NAS and I am still getting this error, anyone run into this and get past it?

 

[Xero7650]

Update​

Error: 'DEVLINKS'

 

[morganL]

Please follow "forum rules" with details of your hardware?
Have you managed to rollback?

 

[Xero7650]

The update never went through, so there was nothing to roll back. I just cant upgrade.

Hardware:
Dell 730XD
[2x] Intel(R) Xeon(R) CPU E5-2697 v3 @ 2.60GHz
593939716 RAM
[2x] Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection
[2x] Ethernet controller: Intel Corporation I350 Gigabit Network Connection (rev 01)
[1x] LSI-SAS9200-8E
[1x] NetAPP DS4246 Disk Array filled with 2TB SAS Disks.
[1x] NetAPP DS2246 Disk Array with with 1.8TB SAS flash disks

 

[morganL]

So did the updated software get downloaded?
Can you verify the network connectivity...

 

[Xero7650]

I can say that yes the update appears downloaded, it shows I can apply a pending update, it just fails to install.

east / west traffic to the NAS is working fine - I have a plex container running and its working fine. I can also say that internet connectivity northbound seems to also be working fine, as I tested downloading via torrent from a container worked fine as well.

 

[Xero7650]

I downloaded updates again, and it succeeded.

This is what im getting, when I try and update.

 

[morganL]

It may be related to another issue I have heard about with SAS HBAs...
I'd suggest that you report-a-bug and we track the issue down if its not known.

The known bug is here:

https://jira.ixsystems.com/browse/NAS-116821

 

[csmcsm]

im having the same issue, how can i fix this to actually get the update processed?

 

[csmcsm]

nobody with the same problem? i cant update truenas from
TrueNAS-22.02.1 -> TrueNAS-22.02.3

 

[indivision]

Perhaps if everyone posted their hardware used and any other particular circumstances we may be able to recognize a pattern?

 

[csmcsm]

yes correct, sorry
i use supermicro sc417 server with Serial Attached SCSI controller: Broadcom / LSI SAS3008 PCI-Express Fusion-MPT SAS-3 (rev 02)

i tried
cli -c 'system advanced update kernel_extra_options="ahci.ahciem_enable=0"'

but no success, i still get the same devlinks error

 

[Samuel Tai]

Kernel extra options only take effect on the next reboot. Is your LSI flashed to IT mode?

 

[csmcsm]

yes i know (sorry i did not mentioned i actually rebooted) and yes flashed in IT mode

 

[csmcsm]

not sure if it helps, but i also get devlinsk errors if i try to create a new pool:
KeyError
'DEVLINKS'

Error: concurrent.futures.process._RemoteTraceback: """ Traceback (most recent call last): File "/usr/lib/python3.9/concurrent/futures/process.py", line 243, in _process_worker r = call_item.fn(*call_item.args, **call_item.kwargs) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 114, in main_worker res = MIDDLEWARE._run(*call_args) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 45, in _run return self._call(name, serviceobj, methodobj, args, job=job) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/worker.py", line 39, in _call return methodobj(*params) File "/usr/lib/python3/dist-packages/middlewared/plugins/zfs_/disks.py", line 30, in get_disks for link in (dev.properties['DEVLINKS'] or '').split(): File "/usr/lib/python3/dist-packages/pyudev/device/_device.py", line 1067, in __getitem__ raise KeyError(prop) KeyError: 'DEVLINKS' """ The above exception was the direct cause of the following exception: Traceback (most recent call last): File "/usr/lib/python3/dist-packages/middlewared/main.py", line 175, in call_method result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/disk_/availability.py", line 21, in get_unused reserved = await self.middleware.call('disk.get_reserved') File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/plugins/disk_/availability.py", line 39, in get_reserved return await self.middleware.call('boot.get_disks') + await self.middleware.call('pool.get_disks') File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1257, in _call return await methodobj(*prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf return await func(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1129, in nf res = await f(*args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/plugins/boot.py", line 122, in get_disks return await self.middleware.call('zfs.pool.get_disks', BOOT_POOL_NAME) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1308, in call return await self._call( File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1265, in _call return await self._call_worker(name, *prepared_call.args) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1271, in _call_worker return await self.run_in_proc(main_worker, name, args, job) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1186, in run_in_proc return await self.run_in_executor(self.__procpool, method, *args, **kwargs) File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1169, in run_in_executor return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs)) KeyError: 'DEVLINKS'

 

[Samuel Tai]

Thanks, that was helpful. Basically, DEVLINKS means the systemd-created links under /dev/disks for your pool are missing some paths.

This is from the middleware plugin /usr/lib/python3/dist-packages/middlewared/plugins/zfs_/disks.py:

Code:

            # these are the the various by-{partuuid/label/id/path} etc labels
            if dev.properties['DEVTYPE'] == 'partition':
                for link in (dev.properties.get('DEVLINKS') or '').split():
                    sys_devices[link.removeprefix('/dev/')] = dev.find_parent('block').sys_name

I'm not sure how to remedy this, other than a clean reinstall of 22.02.3.

 

[csmcsm]

i downloaded the iso of 22.02.3 and did a offline update (boot by cd,..) - now everything seems to work!