Jail issue and NFS service message after upgrade

C

cadamwil

Explorer
Joined
Sep 6, 2013
Messages
60
I seem to be having two issues after upgrading to RC1.

1: I get a warning message about NFS could not bind to 0.0.0.0
* NFS services could not bind to specific IP addresses, using 0.0.0.0.

2: I cannot start my plex Jail, and I believe my upgrade failed. Listed below is the terminal output when I attempted to upgrade my plex jail/plugin.

Code:
/usr/src/usr.sbin/unbound/anchor/Makefile
/usr/src/usr.sbin/unbound/checkconf/Makefile
/usr/src/usr.sbin/unbound/control/Makefile
/usr/src/usr.sbin/unbound/daemon/Makefile
Installing updates... done.
Running upgrade...
plex is not running, starting jail

Command output:
plex is not running, starting jail

Command output:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py", line292, in upgrade_basejail
    iocage_lib.ioc_exec.SilentExec(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_exec.py", line 268, in __init__
    self.output = list(silent)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_exec.py", line 259, in exec_jail
    raise iocage_lib.ioc_exceptions.CommandFailed(
iocage_lib.ioc_exceptions.CommandFailed: [b'jexec: jail "ioc-plex" not found\n', b'']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/zfs.py", line 20, in run
    cp.check_returncode()
  File "/usr/local/lib/python3.8/subprocess.py", line 444, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['zfs', 'rollback', '-r', 'ZFS2/iocage/jails/plexmediaserver-plexpass@ioc_plugin_upgrade_2020-09-16']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/iocage_cli/upgrade.py", line 40,in cli
    ioc.IOCage(jail=jail, skip_jails=skip_jails).upgrade(release)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/iocage.py", line 2102, in upgrade
    new_release = ioc_plugin.IOCPlugin(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_plugin.py", line 1338, in upgrade
    new_release = iocage_lib.ioc_upgrade.IOCUpgrade(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py", line302, in upgrade_basejail
    self.__rollback_jail__(name=snap_name)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py", line410, in __rollback_jail__
    iocage.rollback(name)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/iocage.py", line 1531, in rollback
    Snapshot(f'{ds.name}@{name}').rollback(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 138, in rollback
    return rollback_snapshot(self.resource_name, options)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/zfs.py", line 221, inrollback_snapshot
    return run(['zfs', 'rollback', *flags, snap]).returncode == 0
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/zfs.py", line 22, in run
    raise ZFSException(cp.returncode, cp.stderr)
iocage_lib.zfs.ZFSException: cannot open 'ZFS2/iocage/jails/plexmediaserver-plexpass@ioc_plugin_upgrade_2020-09-16': dataset does not exist

root@freenas:~ # iocage list
+------+--------------------------+-------+--------------+---------------+
| JID  |           NAME           | STATE |   RELEASE    |      IP4      |
+======+==========================+=======+==============+===============+
| None | plex                     | down  | 11.3-RELEASE | 10.0.1.210/24 |
+------+--------------------------+-------+--------------+---------------+
| None | plexmediaserver-plexpass | down  | 11.3-RELEASE | 10.0.1.210/24 |
+------+--------------------------+-------+--------------+---------------+


Any help would be greatly appreciated.
 
C

cadamwil

Explorer
Joined
Sep 6, 2013
Messages
60
Not sure if it's related, but I am also seeing the following message in my console scroll or whatever it's called at the bottom of the screen.

Code:
Sep 16 09:58:56 freenas 1 2020-09-16T09:58:56.919091-05:00 freenas.local mountd 3783 - - can't open /etc/zfs/exports
Sep 16 10:01:42 freenas 1 2020-09-16T10:01:42.278935-05:00 freenas.local mountd 3783 - - can't open /etc/zfs/exports
Sep 16 10:01:42 freenas 1 2020-09-16T10:01:42.856295-05:00 freenas.local mountd 3783 - - can't open /etc/zfs/exports
Sep 16 10:03:16 freenas 1 2020-09-16T10:03:16.145048-05:00 freenas.local mountd 3783 - - can't open /etc/zfs/exports
Sep 16 10:22:36 freenas 1 2020-09-16T10:22:36.462696-05:00 freenas.local mountd 3783 - - can't open /etc/zfs/exports
Sep 16 10:22:37 freenas 1 2020-09-16T10:22:37.671479-05:00 freenas.local mountd 26292 - - can't open /etc/zfs/exports
 
Kris Moore

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
cadamwil said:
I seem to be having two issues after upgrading to RC1.

1: I get a warning message about NFS could not bind to 0.0.0.0
* NFS services could not bind to specific IP addresses, using 0.0.0.0.
Click to expand...

That usually means you have your NFS server setup to bind to an invalid IP address. 0.0.0.0 is the catch all to listen on all interfaces.

Go to Services -> NFS -> Configure and look for the Bind IP Address section. Reset it to correct IP / address.
1600277429878.png


cadamwil said:
2: I cannot start my plex Jail, and I believe my upgrade failed. Listed below is the terminal output when I attempted to upgrade my plex jail/plugin.

Code:
/usr/src/usr.sbin/unbound/anchor/Makefile
/usr/src/usr.sbin/unbound/checkconf/Makefile
/usr/src/usr.sbin/unbound/control/Makefile
/usr/src/usr.sbin/unbound/daemon/Makefile
Installing updates... done.
Running upgrade...
plex is not running, starting jail

Command output:
plex is not running, starting jail

Command output:
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py", line292, in upgrade_basejail
    iocage_lib.ioc_exec.SilentExec(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_exec.py", line 268, in __init__
    self.output = list(silent)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_exec.py", line 259, in exec_jail
    raise iocage_lib.ioc_exceptions.CommandFailed(
iocage_lib.ioc_exceptions.CommandFailed: [b'jexec: jail "ioc-plex" not found\n', b'']

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/zfs.py", line 20, in run
    cp.check_returncode()
  File "/usr/local/lib/python3.8/subprocess.py", line 444, in check_returncode
    raise CalledProcessError(self.returncode, self.args, self.stdout,
subprocess.CalledProcessError: Command '['zfs', 'rollback', '-r', 'ZFS2/iocage/jails/plexmediaserver-plexpass@ioc_plugin_upgrade_2020-09-16']' returned non-zero exit status 1.

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
    sys.exit(cli())
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 764, in __call__
    return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 717, in main
    rv = self.invoke(ctx)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 1137, in invoke
    return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 956, in invoke
    return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.8/site-packages/click/core.py", line 555, in invoke
    return callback(*args, **kwargs)
  File "/usr/local/lib/python3.8/site-packages/iocage_cli/upgrade.py", line 40,in cli
    ioc.IOCage(jail=jail, skip_jails=skip_jails).upgrade(release)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/iocage.py", line 2102, in upgrade
    new_release = ioc_plugin.IOCPlugin(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_plugin.py", line 1338, in upgrade
    new_release = iocage_lib.ioc_upgrade.IOCUpgrade(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py", line302, in upgrade_basejail
    self.__rollback_jail__(name=snap_name)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/ioc_upgrade.py", line410, in __rollback_jail__
    iocage.rollback(name)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/iocage.py", line 1531, in rollback
    Snapshot(f'{ds.name}@{name}').rollback(
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/dataset.py", line 138, in rollback
    return rollback_snapshot(self.resource_name, options)
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/zfs.py", line 221, inrollback_snapshot
    return run(['zfs', 'rollback', *flags, snap]).returncode == 0
  File "/usr/local/lib/python3.8/site-packages/iocage_lib/zfs.py", line 22, in run
    raise ZFSException(cp.returncode, cp.stderr)
iocage_lib.zfs.ZFSException: cannot open 'ZFS2/iocage/jails/plexmediaserver-plexpass@ioc_plugin_upgrade_2020-09-16': dataset does not exist

root@freenas:~ # iocage list
+------+--------------------------+-------+--------------+---------------+
| JID  |           NAME           | STATE |   RELEASE    |      IP4      |
+======+==========================+=======+==============+===============+
| None | plex                     | down  | 11.3-RELEASE | 10.0.1.210/24 |
+------+--------------------------+-------+--------------+---------------+
| None | plexmediaserver-plexpass | down  | 11.3-RELEASE | 10.0.1.210/24 |
+------+--------------------------+-------+--------------+---------------+


Any help would be greatly appreciated.
Click to expand...

That issue does sound like a real bug. Please file a bug report at jira.ixsystems.com and include your debug file, so we can investigate for release.
 
C

cadamwil

Explorer
Joined
Sep 6, 2013
Messages
60
Kris Moore said:
That usually means you have your NFS server setup to bind to an invalid IP address. 0.0.0.0 is the catch all to listen on all interfaces.

Go to Services -> NFS -> Configure and look for the Bind IP Address section. Reset it to correct IP / address.
View attachment 41549



That issue does sound like a real bug. Please file a bug report at jira.ixsystems.com and include your debug file, so we can investigate for release.
Click to expand...
Kris, Thanks for the response, I have filed a bug with the jail issue.

On the NFS, I checked the config and it was set to the 10.0.1.100 (correct LAN) IP. I then changed the System > General Web Interface IPv4 Address to 10.0.1.100 from 0.0.0.0, and this did no affect the issue. Screenshots below

NFS Config
Services_NFS_Options.jpg

Original System > General config
System_General.jpg

Network Interfaces
Network_Interfaces.jpg

System General after change
System_General2.jpg

Let me know if you think this might be a bug as well or just something a little off in my config.
 
C

cadamwil

Explorer
Joined
Sep 6, 2013
Messages
60
I also see the info below when I restarted NFS in the console messages area
Code:
Sep 16 13:56:04 freenas Security policy loaded: MAC/ntpd (mac_ntpd)
Sep 16 13:56:04 freenas 1 2020-09-16T13:56:04.406047-05:00 freenas.local daemon 3813 - - 2020-09-16 13:56:04,405:wsdd ERROR(pid 3814): error while sending packet on ix0: [Errno 50] Network is down
Sep 16 13:56:04 freenas 1 2020-09-16T13:56:04.483494-05:00 freenas.local daemon 3813 - - 2020-09-16 13:56:04,483:wsdd ERROR(pid 3814): error while sending packet on ix0: [Errno 50] Network is down
Sep 16 13:56:04 freenas 1 2020-09-16T13:56:04.631631-05:00 freenas.local daemon 3813 - - 2020-09-16 13:56:04,631:wsdd ERROR(pid 3814): error while sending packet on ix0: [Errno 50] Network is down
Sep 16 13:56:04 freenas 1 2020-09-16T13:56:04.928366-05:00 freenas.local daemon 3813 - - 2020-09-16 13:56:04,928:wsdd ERROR(pid 3814): error while sending packet on ix0: [Errno 50] Network is down
Sep 16 13:56:05 freenas kernel: ix0: link state changed to UP
Sep 16 14:05:18 freenas 1 2020-09-16T14:05:18.585304-05:00 freenas.local /middlewared 2311 - - Fingerprint of the certificate used in UI : 20:A9:5F:31:FB:67:52:FF:A7:2B:81:7D:58:22:FA:BB:18:67:F8:47
Sep 16 14:06:08 freenas 1 2020-09-16T14:06:08.036930-05:00 freenas.local mountd 4656 - - can't open /etc/zfs/exports
Sep 16 14:06:08 freenas nfsd: can't register svc name
 
Kris Moore

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,471
That does sound like a possible bug now. I'd say go ahead and file a ticket for the NFS issue as well and lets dig into it. Could be a setting in your config still, but we should explore that to be sure and see if we can fix it during upgrade with a migration.
 
C

cadamwil

Explorer
Joined
Sep 6, 2013
Messages
60
Kris Moore said:
That does sound like a possible bug now. I'd say go ahead and file a ticket for the NFS issue as well and lets dig into it. Could be a setting in your config still, but we should explore that to be sure and see if we can fix it during upgrade with a migration.
Click to expand...
Will do, sounds like a plan. Thanks again Kris.
 
G

gwaitsi

Patron
Joined
May 18, 2020
Messages
243
i upgraded to truenas 12 from freenas 11.3U5 and have the same issue. log is full of these errors.
my IP is correctly bound too.

in my case, i established that the jails were causing the mountd 3783 - - can't open /etc/zfs/exports error messages - even though they do not use mountd. Connecting to an NFS share from an allowed client give an accepted message with the ip address of the client. stopping a jail and restarting it, causes about a dozen of these errors
 
Last edited:
holz

holz

Cadet
Joined
Nov 6, 2020
Messages
1
Same here since the update to 12.0-RELEASE.
"(...)mountd 39469 - - can't open /etc/zfs/exports"
 
A

AndrewParsons

Dabbler
Joined
Jun 14, 2016
Messages
40
Plus one, it too have the same error after upgrading to TrueNAS core.

Code:
Nov 16 15:13:26 freenas 1 2020-11-16T15:13:26.262921-07:00 freenas.local mountd 1693 - - can't open /etc/zfs/exports


Please let me know if I can do or assist with.

Andrew
 
You must log in or register to reply here.

Similar threads

hescominsoon
unable to remove disk
Replies
2
Views
1K
hescominsoon
hescominsoon
Scampicfx
Bug TrueNAS-12.0-U1.1: Adding "Authorized Access iSCSI User" fails
Replies
5
Views
1K
Scampicfx
Scampicfx
Patrick M. Hausen
No update from 12.0 nightlies to BETA
Replies
3
Views
3K
Kris Moore
Kris Moore
J
Beta2, unable to extend pool; camcontrol error
Replies
0
Views
920
jenksdrummer
J
J
Failed Error when clicking Expand Pool
Replies
8
Views
2K
JeMoeder
J
Top