Issues with cron (TrueNAS SCALE)

[cub1t]

Hi All,

I've been having issues with cron, but I suspect that the issue is affecting other things. I'm closer to finding out the issue after changing the syslog level to Debug.

Here is the stack trace from /var/log/middlewared.log when I edit or create a cron job in the webUI:

[2021/12/01 19:34:53] (DEBUG) middlewared.setup():286 - Root dataset for pool 'boot-pool' is not available, temporarily setting up system dataset on boot pool

[2021/12/01 19:35:02] (ERROR) EtcService.generate():394 - Failed to render mako:cron.d/middlewared
Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 383, in generate
rendered = await renderer.render(path)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 55, in render
return await self.service.middleware.run_in_thread(do)
File "/usr/lib/python3/dist-packages/middlewared/utils/run_in_thread.py", line 10, in run_in_thread
return await self.loop.run_in_executor(self.run_in_thread_executor, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 47, in do
return tmpl.render(
File "/usr/lib/python3/dist-packages/mako/template.py", line 476, in render
return runtime._render(self, self.callable_, args, data)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 878, in _render
_render_context(
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 920, in _render_context
_exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
File "/usr/lib/python3/dist-packages/mako/runtime.py", line 947, in _exec_template
callable_(context, *args, **kwargs)
File "/tmp/mako/usr/lib/python3/dist-packages/middlewared/etc_files/cron.d/middlewared.mako.py", line 70, in render_body
for job in middleware.call_sync("pool.scrub.query", [["enabled", "=", True]]):
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1326, in call_sync
return self.run_coroutine(methodobj(*prepared_call.args))
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1366, in run_coroutine
return fut.result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
return self.__get_result()
File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
raise self._exception
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1131, in nf
res = await f(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1263, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/service.py", line 821, in query
result = await self.middleware.call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1310, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1267, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1263, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 164, in query
result = await self._queryset_serialize(
File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 216, in _queryset_serialize
return [
File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 217, in <listcomp>
await self._extend(data, extend, extend_context, extend_context_value, select)
File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 232, in _extend
data = await self.middleware.call(extend, data)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1310, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1267, in _call
return await methodobj(*prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 4336, in pool_scrub_extend
data['pool'] = pool['id']
TypeError: 'NoneType' object is not subscriptable

The output of zfs list:

NAME USED AVAIL REFER MOUNTPOINT
boot-pool 10.3G 81.7G 96K none
boot-pool/.system 4.49G 81.7G 1.27G legacy
boot-pool/.system/configs-b439d6a91c22424e915d4f54b2799491 96K 81.7G 96K legacy
boot-pool/.system/configs-e0279ae4a993433892ab5740df1028a2 5.66M 81.7G 5.66M legacy
boot-pool/.system/cores 12.8M 1011M 12.8M legacy
boot-pool/.system/ctdb_shared_vol 96K 81.7G 96K legacy
boot-pool/.system/glusterd 104K 81.7G 104K legacy
boot-pool/.system/rrd-b439d6a91c22424e915d4f54b2799491 5.68M 81.7G 5.68M legacy
boot-pool/.system/rrd-e0279ae4a993433892ab5740df1028a2 118M 81.7G 118M legacy
boot-pool/.system/samba4 2.50M 81.7G 456K legacy
boot-pool/.system/services 96K 81.7G 96K legacy
boot-pool/.system/syslog-b439d6a91c22424e915d4f54b2799491 312K 81.7G 312K legacy
boot-pool/.system/syslog-e0279ae4a993433892ab5740df1028a2 3.08G 81.7G 3.08G legacy
boot-pool/.system/webui 96K 81.7G 96K legacy
boot-pool/ROOT 5.76G 81.7G 96K none
boot-pool/ROOT/22.02-RC.1-2 5.76G 81.7G 5.76G legacy
boot-pool/grub 5.73M 81.7G 5.73M legacy
tank 18.0T 2.57T 19.0M /mnt/tank

It would seem that there might be an issue with the naming of the boot pools?

If anyone can shed a bit of light on what I might do to diagnose this further I'd appreciate it!

 

[cub1t]

A year later and cron still doesn't work from the webui!

 

[NugentS]

It does here. I have a half dozen cron jobs running just fine. In both Scale & Core

 

[joeschmuck]

A year later and cron still doesn't work from the webui!

You should do as the forum rules suggest, provide a complete listing of your hardware, what version of TrueNAS you are running, how your pool has been created, and exactly what isn't working in the GUI, etc. CRON works for me as well and I'm sure 99.99% of the people using it.

Also, have you tried to recreate the boot pool? When I say that, I mean reformat the drive(s) and then reinstall (not upgrade) the TrueNAS application.

Lastly, did you submit a bug report?

 

[cub1t]

I thought that maybe I had inherited some incorrect configuration through updating from the first alpha releases to the current stable.

So I did a fresh install of 22.02.4 today to my existing boot-pool and the installer created a new dataset named 22.02.4-1 (so as not to overwrite boot-pool/ROOT/22.02.4).

After restoring my database backup and checking some test cron jobs, it still doesn't work! I'm not prepared to fully re-install TrueNAS to my boot disk because I don't have any confidence that it will make a difference.

I'm going to have to create a bug report like you suggested. Thanks for your replies none the less.

 

[cub1t]

EDIT: I decided to do a re-install and reformat of my boot disk.

After a completely new install, cron is still broken.

Code:

[2022/11/13 19:22:34] (ERROR) EtcService.generate():459 - Failed to render mako:cron.d/middlewared
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 449, in generate
    rendered = await renderer.render(path, ctx)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 43, in render
    return await self.service.middleware.run_in_thread(do)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1209, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1206, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/etc.py", line 34, in do
    return tmpl.render(
  File "/usr/lib/python3/dist-packages/mako/template.py", line 476, in render
    return runtime._render(self, self.callable_, args, data)
  File "/usr/lib/python3/dist-packages/mako/runtime.py", line 878, in _render
    _render_context(
  File "/usr/lib/python3/dist-packages/mako/runtime.py", line 920, in _render_context
    _exec_template(inherit, lclcontext, args=args, kwargs=kwargs)
  File "/usr/lib/python3/dist-packages/mako/runtime.py", line 947, in _exec_template
    callable_(context, *args, **kwargs)
  File "/run/mako/etc_files/cron.d/middlewared.mako.py", line 70, in render_body
    for job in middleware.call_sync("pool.scrub.query", [["enabled", "=", True]]):
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1364, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1404, in run_coroutine
    return fut.result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 433, in result
    return self.__get_result()
  File "/usr/lib/python3.9/concurrent/futures/_base.py", line 389, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1140, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/service.py", line 907, in query
    result = await self.middleware.call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1345, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1294, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1272, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 164, in query
    result = await self._queryset_serialize(
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 216, in _queryset_serialize
    return [
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 217, in <listcomp>
    await self._extend(data, extend, extend_context, extend_context_value, select)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/datastore/read.py", line 232, in _extend
    data = await self.middleware.call(extend, data)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1345, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1294, in _call
    return await methodobj(*prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/pool.py", line 4300, in pool_scrub_extend
    data['pool'] = pool['id']
TypeError: 'NoneType' object is not subscriptable

Nothing to do but to go to Jira

 

[joeschmuck]

I'm not prepared to fully re-install TrueNAS to my boot disk because I don't have any confidence that it will make a difference.

Then I'd recommend you disconnect the boot drive and use a freshly formatted USB Flash Drive, install to this device and boot from it.

Also, why haven't you provided your system build information, what CRON jobs you have, any VM's/Jails, where they might reside? Don't be shy, we can't help you if we have no idea what you are using.

 

[NugentS]

@cub1t . Cron does work
Please post a detailed hardware spec as per forum rules