Locked out of jails "jexec: execvp: /bin/sh: Permission denied\n"

[ultrask14]

TrueNAS-12.0-U2

- After adding permissions to syncthing during adding mountpoints, none of my previously installed plugins start and none of their jails start.
- Using the GUI to start each plugin comes back with no a spinny wheel then no response.
- Attempting to try to force start a jail using:

Code:

iocage console syncthing -f

gives me the following error:

Code:

  + Starting services FAILED
ERROR:
[b'jexec: execvp: /bin/sh: Permission denied\n', b'']

Refusing to start syncthing: exec_start failed

- Using

Code:

less /var/log/iocage.log

gives me nothing useful.

I'm at a loss, any plugin that has been installed manually through jails I've made myself are completely unaffected. Please help

 

[Samuel Tai]

- After adding permissions to syncthing during adding mountpoints, none of my previously installed plugins start and none of their jails start.

Try setfacl -b -R /mnt/<name of your iocage pool>/iocage. This will strip out all ACLs from the iocage dataset and its child datasets.

 

[ultrask14]

Try setfacl -b -R /mnt/<name of your iocage pool>/iocage. This will strip out all ACLs from the iocage dataset and its child datasets.

I use the code you gave me on my jail named MInecraft_Server (yes the first "I" should be caps) i get the following error:

Code:

root@freenas[~]# setfacl -b -R /mnt/MInecraft_Server/iocage

setfacl: /mnt/MInecraft_Server/iocage: acl_get_link_np() failed: No such file or directory

just to make sure, I've tried this on all the jails for the plugins that wont start and i get the same error.

 

[ultrask14]

Try setfacl -b -R /mnt/<name of your iocage pool>/iocage. This will strip out all ACLs from the iocage dataset and its child datasets.

Also, attempting to install a new version of syncthing gives me the following error:

Code:

Error: syncthing had a failure Exception: RuntimeError Message: + Starting services FAILED ERROR: [b'jexec: execvp: /bin/sh: Permission denied\n', b'', b'', b''] Refusing to start syncthing: exec_start failed Partial plugin destroyed

 

[Samuel Tai]

Not the name of your jail; the name of your pool. What's the output of zpool status, and zfs get org.freebsd.ioc:active <name of pool>?

 

[ultrask14]

Not the name of your jail; the name of your pool.>[/ICODE]?

Apologies, that was clear, i should have read your message properly. The output seems limited, i have 5 or so jails that wont start however i only see a jail that i can already open called "bitwarden" as i had made it manually, all manually made jails with manually installed plugins are unaffected:

Code:

root@freenas[~]# setfacl -b -R /mnt/General/iocage
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/random: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/urandom: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/ptmx: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/fd: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/fd/0: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/fd/1: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/fd/2: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/fd/3: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/fd/4: pathconf(_PC_ACL_NFS4) failed: Bad file descriptor
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/fd/4: acl_get_link_np() failed: Bad file descriptor
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/stdin: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/stdout: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/stderr: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/null: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/zero: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/zfs: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/crypto: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/pts: acl_get_link_np() failed: Operation not supported
setfacl: /mnt/General/iocage/jails/bitwarden/root/dev/pts/0: acl_get_link_np() failed: Operation not supported

What's the output of zpool status, and zfs get org.freebsd.ioc:active <name of pool>?

The output says "Errors, no known data errors" for any of my pools.

 

[Samuel Tai]

Slow down, and please provide the output of the requested commands, within [ CODE ] [ /CODE ] blocks. (Remove spaces between the brackets for the real output.)

• What's zpool status?
• What's zfs get org.freebsd.ioc:active General?

When you say you changed permissions for Syncthing, what exactly did you do, or thought you did?

 

[ultrask14]

Slow down, and please provide the output of the requested commands, within [ CODE ] [ /CODE ] blocks. (Remove spaces between the brackets for the real output.)

• What's zpool status?
• What's zfs get org.freebsd.ioc:active General?

When you say you changed permissions for Syncthing, what exactly did you do, or thought you did?

Apologies.

Code:

 
  pool: General
 state: ONLINE
  scan: scrub repaired 0B in 01:28:12 with 0 errors on Sun Feb 14 09:28:13 2021
config:

        NAME                                          STATE     READ WRITE CKSUM
        General                                       ONLINE       0     0     0
          gptid/7d26a3bc-37de-11eb-848a-d8cb8ac2f7a9  ONLINE       0     0     0

Code:

root@freenas[~]# zfs get org.freebsd.ioc:active General
NAME     PROPERTY                VALUE                   SOURCE
General  org.freebsd.ioc:active  yes                     local

I thought i was adding a user called "syncthing" to have full control permissions through ACL for inside the /iocage/syncthing/ jail. I attempted to add a mountpoint for a new folder syncthing had made but needing to restart the jail but it would not shut down so I restarted the system. When the system loaded this is when i realised no jails would start, i could not SSH in or download new plugins.

 

[Samuel Tai]

Hmm, I think I have an idea what you did. What does grep -i syncthing /etc/passwd show? Also grep -i syncthing /etc/group.

 

[ultrask14]

I'm glad one of us does!

for your first command:

Code:

root@freenas[~]# grep -i syncthing /etc/passwd
sync:*:983:1006:syncthing:/nonexistent:/bin/sh

and the second seemingly doesnt do anything:

Code:

root@freenas[~]# grep -i syncthing /etc/group
root@freenas[~]# grep -i syncthing /etc/group
root@freenas[~]#

I did uninstall a couple of the redundant plugins that i thought i could reinstall quickly which includes syncthing. Of course as ive shown earlier im unable to install new plugins currently:

Code:

Error: syncthing had a failure Exception: RuntimeError Message: + Starting services FAILED ERROR: [b'jexec: execvp: /bin/sh: Permission denied\n', b'', b'', b''] Refusing to start syncthing: exec_start failed Partial plugin destroyed

Also, i checked the users listing on the server and the syncthing user is actually called "sync" would you like me to try the commands again?

 

[Samuel Tai]

Yes, please try the greps with just sync. However, that didn't confirm my hypothesis you'd accidentally created the sync user with root's ID 0. What does iocage list show?

 

[ultrask14]

Hmm, I think I have an idea what you did. What does grep -i syncthing /etc/passwd show? Also grep -i syncthing /etc/group.

Sure, lets try this again:

Code:

root@freenas[~]# grep -i sync /etc/passwd
sync:*:983:1006:syncthing:/nonexistent:/bin/sh

Second Command:

Code:

root@freenas[~]# grep -i sync /etc/group
builtin_users:*:545:qbittorrent,sync,mcserver
admin:*:1000:plexuser,radarr,sonarr,jackett,qbittorrent,sync,www
sync:*:1006:

I'm convinced I've seriously messed the Jail ACL Permissions.. might be an observation on my current position rather than a useful point. I was sure i had made the sync UID 1001.

Thank you for your help by the way

 

[ultrask14]

Yes, please try the greps with just sync. However, that didn't confirm my hypothesis you'd accidentally created the sync user with root's ID 0. What does iocage list show?

oh and you needed iocage list, my apologies:

Code:

root@freenas[~]# iocage list
+------+------------------+-------+--------------+------------+
| JID  |       NAME       | STATE |   RELEASE    |    IP4     |
+======+==================+=======+==============+============+
| None | MInecraft_Server | down  | 12.1-RELEASE | DHCP       |
+------+------------------+-------+--------------+------------+
| 2    | Media_Downloads  | up    | 12.2-RELEASE | DHCP       |
+------+------------------+-------+--------------+------------+
| 3    | Media_Management | up    | 12.2-RELEASE | DHCP       |
+------+------------------+-------+--------------+------------+
| 4    | bitwarden        | up    | 12.2-RELEASE | 172.16.0.2 |
+------+------------------+-------+--------------+------------+
| None | nextcloud        | down  | 12.1-RELEASE | DHCP       |
+------+------------------+-------+--------------+------------+
| None | radarr           | down  | 12.1-RELEASE | DHCP       |
+------+------------------+-------+--------------+------------+
| None | sonarr           | down  | 12.1-RELEASE | DHCP       |
+------+------------------+-------+--------------+------------+
| None | tomsmediaserver  | down  | 12.1-RELEASE | DHCP       |
+------+------------------+-------+--------------+------------+

 

[Samuel Tai]

OK, if the UID and GID are both supposed to be 1001, you can correct that in the GUI.

Now, please attach the output of ls -l /bin, ls -l /usr/bin, ls -l /sbin, and ls -l /usr/sbin. Your inability to SSH concerns me.

 

[flikamasha]

So I am having this issue as well when I am trying to install any new plugins as well.

For testing purposes I just named it tmp.
I have tried this with both MineOS (which is what I am actually trying to setup) and Plex (already have an instance of it installed)
I was able to successfully install the Asigra Backup plugin though.
I will also note that when looking at plugins not all have what their current version is. Asigra shows its version number, MineOS and Plex currently do not. Guessing this could be causing issues?

Code:

Install
Error: tmp had a failure Exception: RuntimeError Message: + Starting services FAILED ERROR: [b'jexec: execvp: /bin/sh: Permission denied\n', b'', b'', b''] Refusing to start tmp: exec_start failed Partial plugin destroyed