Is U3 on track & experience

[Froos]

Updated from 12.0 to U3 today (could not update to U2 because of CXBG reboot issue)

System still goes into reboot loop after upgrade to U3. Either the Chelsio driver is not working as designed of I have a different problem :(

 

[Samuel Tai]

My upgrade was uneventful, and my Dashboard behaves properly. @Patrick M. Hausen, have you tried moving the system dataset to a different device and back?

 

[Froos]

This is what I see in the middleware.log:

[2021/04/14 15:34:06] (ERROR) PoolService.import_on_boot():1744 - Failed to import Vol0
concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
File "/usr/local/lib/python3.8/concurrent/futures/process.py", line 239, in _process_worker
r = call_item.fn(*call_item.args, **call_item.kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 94, in main_worker
res = MIDDLEWARE._run(*call_args)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 45, in _run
return self._call(name, serviceobj, methodobj, args, job=job)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/worker.py", line 39, in _call
return methodobj(*params)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 371, in import_pool
self.logger.error(
File "libzfs.pyx", line 391, in libzfs.ZFS.__exit__
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/zfs.py", line 362, in import_pool
raise CallError(f'Pool {name_or_guid} not found.', errno.ENOENT)
middlewared.service_exception.CallError: [ENOENT] Pool 13297304568331885618 not found.
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/pool.py", line 1736, in import_on_boot
self.middleware.call_sync('zfs.pool.import_pool', pool['guid'], {
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1258, in call_sync
return self.run_coroutine(self._call_worker(name, *prepared_call.args))
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1294, in run_coroutine
return fut.result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 432, in result
return self.__get_result()
File "/usr/local/lib/python3.8/concurrent/futures/_base.py", line 388, in __get_result
return await self.run_in_proc(main_worker, name, args, job)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1136, in run_in_proc
return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
middlewared.service_exception.CallError: [ENOENT] Pool 13297304568331885618 not found.
[2021/04/14 15:34:43] (DEBUG) middlewared.dumpdev_configure():24 - Configured "/dev/da13p1" device for crash dumps.

Any suggestions?

 

[Patrick M. Hausen]

@Patrick M. Hausen, have you tried moving the system dataset to a different device and back?

Nope, will try.

 

[c77dk]

Updated from 12.0 to U3 today (could not update to U2 because of CXBG reboot issue)

System still goes into reboot loop after upgrade to U3. Either the Chelsio driver is not working as designed of I have a different problem :(

No problem with my Chelsio - so might be something else (or I might just be lucky)

 

[Constantin]

Upgrade to from TrueNAS to TNU3 went fine.

Middleware complained post-upgrade when I unlocked the pool that the iocage had the wrong permissions, along with a bevy of errors. Given that I have zero jails, plug-ins, and like add-ons working here, is there a way to force TrueNAS to rebuild the iocage directory?

The pool still unlocked as expected, so I'm not too worried.

 

[Patrick M. Hausen]

If you have zero jails, just delete the entire iocage dataset from the "Storage --> Pools " interface and start over. The structure will automatically be created when you configure your first jail or plugin.

 

[Constantin]

Just tried that and got the following error:

Looks like the UI is having some HTML rendering issues in addition to problems with having the directory destroyed.

Should I just do it from the command line?

 

[Patrick M. Hausen]

Should I just do it from the command line?

I would and in fact I have numerous times.

 

[Patrick M. Hausen]

My upgrade was uneventful, and my Dashboard behaves properly. @Patrick M. Hausen, have you tried moving the system dataset to a different device and back?

Does not fix the problem. Neither did another reboot afterwards.

Edit: this is the relevant part of middlewared.log, I guess - at least that is all that is in there after a browser refresh of the dashboard.

Code:

[2021/04/14 16:52:27] (WARNING) middlewared.process():47 - EventSource 'reporting.realtime' run() failed
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/middlewared/event.py", line 45, in process
    self.run()
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/reporting/events.py", line 166, in run
    'speeds': self.get_interface_speeds(),
  File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/reporting/events.py", line 129, in get_interface_speeds
    if m := RE_BASE.match(interface['state']['active_media_subtype']):
TypeError: expected string or bytes-like object
[2021/04/14 16:52:29] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update-master.ixsystems.com:443
[2021/04/14 16:52:29] (DEBUG) urllib3.connectionpool._make_request():428 - https://update-master.ixsystems.com:443 "GET /TrueNAS/trains_redir.json HTTP/1.1" 200 233
[2021/04/14 16:52:29] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/trains.txt', 'https://update-master.ixsystems.com/TrueNAS/trains.txt'])
[2021/04/14 16:52:29] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.ixsystems.com:443
[2021/04/14 16:52:30] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.ixsystems.com:443 "GET /TrueNAS/trains.txt HTTP/1.1" 301 169
[2021/04/14 16:52:30] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.freenas.org:443
[2021/04/14 16:52:30] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.freenas.org:443 "GET /TrueNAS/trains.txt HTTP/1.1" 200 442
[2021/04/14 16:52:30] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/trains.txt', 'https://update-master.ixsystems.com/TrueNAS/trains.txt']):  Read 442 bytes total
[2021/04/14 16:52:30] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update-master.ixsystems.com/TrueNAS/TrueNAS-12.0-STABLE/LATEST'])
[2021/04/14 16:52:30] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update-master.ixsystems.com:443
[2021/04/14 16:52:31] (DEBUG) urllib3.connectionpool._make_request():428 - https://update-master.ixsystems.com:443 "GET /TrueNAS/TrueNAS-12.0-STABLE/LATEST HTTP/1.1" 200 1602
[2021/04/14 16:52:31] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update-master.ixsystems.com/TrueNAS/TrueNAS-12.0-STABLE/LATEST']):  Read 1602 bytes total
[2021/04/14 16:52:31] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem'])
[2021/04/14 16:52:31] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update-master.ixsystems.com:443
[2021/04/14 16:52:31] (DEBUG) urllib3.connectionpool._make_request():428 - https://update-master.ixsystems.com:443 "GET /updates/ix_crl.pem HTTP/1.1" 200 1028
[2021/04/14 16:52:31] (DEBUG) freenasOS.Configuration.CheckFreeSpace():77 - CheckFreeSpace(path=/tmp/tmpu0cq60xm.pem, pool=None, required=1028)
[2021/04/14 16:52:31] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update-master.ixsystems.com/updates/ix_crl.pem']):  Read 1028 bytes total
[2021/04/14 16:52:31] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():606 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt', 'https://update-master.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt'])
[2021/04/14 16:52:31] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.ixsystems.com:443
[2021/04/14 16:52:32] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.ixsystems.com:443 "GET /TrueNAS/Validators/ValidateUpdate-mg3ef525.txt HTTP/1.1" 301 169
[2021/04/14 16:52:32] (DEBUG) urllib3.connectionpool._new_conn():955 - Starting new HTTPS connection (1): update.freenas.org:443
[2021/04/14 16:52:32] (DEBUG) urllib3.connectionpool._make_request():428 - https://update.freenas.org:443 "GET /TrueNAS/Validators/ValidateUpdate-mg3ef525.txt HTTP/1.1" 200 185
[2021/04/14 16:52:32] (DEBUG) freenasOS.Configuration.CheckFreeSpace():77 - CheckFreeSpace(path=/tmp/tmp_1a2_cgo, pool=None, required=185)
[2021/04/14 16:52:32] (DEBUG) freenasOS.Configuration.TryGetNetworkFile():745 - TryGetNetworkFile(['https://update.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt', 'https://update-master.ixsystems.com/TrueNAS/Validators/ValidateUpdate-mg3ef525.txt']):  Read 185 bytes total

 

[Constantin]

Well, just tried that and all the devices are allegedly busy.

# rm -R *
rm: download: Device busy
rm: images: Device busy
rm: jails: Device busy
rm: log: Device busy
rm: releases: Device busy
rm: templates: Device busy

when I traverse to jails in the GUI, I get the following message:

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 137, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self,
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1206, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/local/lib/python3.8/site-packages/middlewared/main.py", line 1110, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.8/concurrent/futures/thread.py", line 57, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/schema.py", line 977, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/jail_freebsd.py", line 711, in query
self.check_dataset_existence()
File "/usr/local/lib/python3.8/site-packages/middlewared/plugins/jail_freebsd.py", line 1019, in check_dataset_existence
IOCCheck(migrate=True, reset_cache=True)
File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_check.py", line 58, in __init__
self.__check_datasets__()
File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_check.py", line 131, in __check_datasets__
iocage_lib.ioc_common.logit({
File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_common.py", line 107, in logit
callback(content, exception)
File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_common.py", line 80, in callback
raise callback_exception(message)
RuntimeError: Dataset "iocage" has exec=off (should be on)

So I presume I have to chmod the directory first?

 

[Patrick M. Hausen]

These are all datasets. You need to zfs destroy -r <poolname>/iocage.

 

[Constantin]

Happiness! There is a giant smoking crater where iocage once was. Thank you!

 

[Froos]

Tried to reboot to 12U3. it still goes into reboot loop.

An error I notice is "Failed to sync interfaces". Last thing I see I did not expect and before it flips into reboot is "Updating CPU Microcode"

 

[MikeyG]

Cool - I will do the update soon.....Due to the recent silent data corruption thing, acow was disabled is it back with U3 ?

Did you ever get an answer about acow being re-enabled?

 

[kherr]

my upgrade was a nightmare .... took over 3 hours ... after a fresh, wipe boot drives, install everything looked more hopeful till a reoccurring issue rose it's head again, the system would not boot by itself after the config file was restored. throughout the process, when warm rebooting of the system, I would have to go down and turn it off .. let drives spin down .. and power up again. Never had any issue like this before since I built it.

Although everything (files/jails/settings) were there, Plex has been up to old tricks that were resolved in the last update.

In many years of updating f/w on my Qnap boxes and until now .... everything had been smooth sailing.

BACKUP that CONFIG file ......

Oh well ... live'n'learn ....

 

[Constantin]

Careful, you may also be dealing with a boot-related hardware problem. Is this a system based on a C2xxx series Intel atom processor? That’s how my C2750D4I ASRock board started to go down the drain.

 

[kherr]

no R5 3600 Ryzen ... never had any other issues. it started doing this with this upgrade process ... once I had back up ... no sense tempting fate.

 

[tiberiusQ]

Did you ever get an answer about acow being re-enabled?

Nope, interesting isn't it (...)

 

[Kris Moore]

Nope, interesting isn't it (...)

Acow was not reenabled.