SOLVED USB (Z-Wave) device no longer shows up in iocage jail on FreeNAS 11.2

[tprelog]

Update: This has now been solved by a quick fix in post #2 and a slightly more advanced (probably better) approach in post #10

Post #2 suggests setting devfs_ruleset=3 which exposes most of the host /dev tree to the jail. This works but more of /dev is exposed than needed.

Post #10 seems to be a better approach by exposing only the cua* devices (how zwave and zigbee USB shows up). This can accomplished with a custom ruleset. Additionally this requires a FreeNAS start-up script to recreate the ruleset each time FreeNAS is (re)booted

Now back to the originally scheduled post:

More problems with iocage after FreeNAS 11.2-BETA3 and my Google-foo continues to fail me.
I've been using an Aeotec Z-Wave USB controller with Home Assistant running in iocage since FreeNAS 11.1. This was still working perfectly in FreeNAS 11.2-BETA3. For as long as I have been using this device it has shown up both on FreeNAS and inside my jail as /dev/cuaU0

I have checked the version of iocage in both 11.2-BETA3 and 11.2-RC2 and it appears to have remained the same so I'm assuming this was a change (or bug) made by FreeNAS

Code:

[troy@vNasB3 ~]$ iocage --version
Version 1.0 ALPHA 1

Here is a list of devices that appear from my jail console in FreeNAS 11.2-BETA3
sudo iocage console hass-dev

Code:

root@hass-dev:~ # ls -l /dev
total 4
crw-r--r--  1 root  wheel     0x2a Sep 29 12:19 acpi
crw-r-----  1 root  operator  0x58 Sep 29 12:19 ada0
crw-r-----  1 root  operator  0x5a Sep 29 12:19 ada0p1
crw-r-----  1 root  operator  0x5b Sep 29 12:19 ada0p2
crw-r-----  1 root  operator  0x59 Sep 29 12:22 ada1
crw-r-----  1 root  operator  0x64 Sep 29 12:22 ada1p1
crw-r-----  1 root  operator  0x69 Sep 29 12:22 ada1p1.eli
crw-r-----  1 root  operator  0x66 Sep 29 12:22 ada1p2
crw-rw-r--  1 root  operator  0x2c Sep 29 12:19 apm
crw-rw----  1 root  operator  0x2b Sep 29 12:19 apmctl
crw-------  1 root  wheel     0x32 Sep 29 12:19 atkbd0
crw-------  1 root  kmem       0xc Sep 29 12:19 audit
crw-------  1 root  wheel      0xb Sep 29 12:19 auditpipe
crw-------  1 root  wheel     0x13 Sep 29 12:48 bpf
lrwxr-xr-x  1 root  wheel        3 Sep 29 12:45 bpf0 -> bpf
crw-rw-rw-  1 root  wheel     0x35 Sep 29 12:19 bpsm0
dr-xr-xr-x  2 root  wheel      512 Sep 29 12:45 cam
crw-r-----  1 root  operator  0x43 Sep 29 12:19 cd0
crw-------  1 root  wheel      0x7 Sep 29 12:45 console
crw-------  1 root  wheel     0x1f Sep 29 12:19 consolectl
crw-r-----  1 root  kmem      0x1c Sep 29 12:19 cpuctl0
crw-r-----  1 root  kmem      0x1d Sep 29 12:19 cpuctl1
crw-rw-rw-  1 root  wheel     0x3a Sep 29 12:19 crypto
crw-rw-rw-  1 root  wheel      0xd Sep 29 12:19 ctty
crw-rw----  1 uucp  dialer    0x7c Sep 29 12:51 cuaU0
crw-rw----  1 uucp  dialer    0x7d Sep 29 12:51 cuaU0.init
crw-rw----  1 uucp  dialer    0x7e Sep 29 12:51 cuaU0.lock
crw-------  1 root  wheel      0x8 Sep 29 12:19 devctl
crw-------  1 root  wheel      0x9 Sep 29 12:19 devctl2
cr--r--r--  1 root  wheel     0x3f Sep 29 12:19 devstat
dr-xr-xr-x  2 root  wheel      512 Sep 29 12:45 dtrace
dr-xr-xr-x  2 root  wheel      512 Sep 29 12:19 fd
crw-------  1 root  wheel     0x21 Sep 29 12:19 fido
crw-rw-rw-  1 root  wheel      0xe Sep 29 12:19 full
crw-r-----  1 root  operator   0x4 Sep 29 12:19 geom.ctl
dr-xr-xr-x  2 root  wheel      512 Sep 29 12:45 gptid
crw-------  1 root  wheel     0x1e Sep 29 12:19 io
crw-------  1 root  wheel     0x22 Sep 29 12:19 iscsi
lrwxr-xr-x  1 root  wheel        6 Sep 29 12:45 kbd0 -> atkbd0
lrwxr-xr-x  1 root  wheel        7 Sep 29 12:45 kbd1 -> kbdmux0
crw-------  1 root  wheel     0x24 Sep 29 12:19 kbdmux0
crw-------  1 root  wheel     0x28 Sep 29 12:19 klog
crw-r-----  1 root  kmem      0x27 Sep 29 12:19 kmem
dr-xr-xr-x  2 root  wheel      512 Sep 29 12:45 led
lrwxr-xr-x  1 root  wheel       14 Sep 29 12:45 log -> ../var/run/log
crw-------  1 root  wheel      0xa Sep 29 12:19 mdctl
crw-r-----  1 root  kmem      0x26 Sep 29 12:19 mem
crw-r-----  1 root  operator  0x51 Sep 29 12:19 mlx5ctl
crw-------  1 root  kmem      0x23 Sep 29 12:19 nfslock
crw-rw-rw-  1 root  wheel      0xf Sep 29 13:00 null
crw-------  1 root  operator  0x40 Sep 29 12:19 pass0
crw-------  1 root  operator  0x41 Sep 29 12:19 pass1
crw-------  1 root  operator  0x42 Sep 29 12:19 pass2
crw-r--r--  1 root  wheel     0x11 Sep 29 12:19 pci
crw-rw-rw-  1 root  wheel     0x34 Sep 29 12:19 psm0
crw-rw-rw-  1 root  wheel     0x12 Sep 29 12:19 ptmx
dr-xr-xr-x  2 root  wheel      512 Sep 29 13:04 pts
crw-r--r--  1 root  wheel      0x5 Sep 29 12:19 random
crwx------  1 root  wheel     0x50 Sep 29 12:19 rdma_cm
dr-xr-xr-x  2 root  wheel      512 Sep 29 12:45 reroot
crw-------  1 root  wheel     0x1b Sep 29 12:19 snp
lrwxr-xr-x  1 root  wheel        4 Sep 29 12:45 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 Sep 29 12:45 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 Sep 29 12:45 stdout -> fd/1
crw-------  1 root  wheel     0x20 Sep 29 12:19 sysmouse
crw-------  1 root  wheel     0x79 Sep 29 12:51 ttyU0
crw-------  1 root  wheel     0x7a Sep 29 12:51 ttyU0.init
crw-------  1 root  wheel     0x7b Sep 29 12:51 ttyU0.lock
crw-------  1 root  wheel     0x44 Sep 29 12:20 ttyv0
crw-------  1 root  wheel     0x45 Sep 29 12:20 ttyv1
crw-------  1 root  wheel     0x46 Sep 29 12:20 ttyv2
crw-------  1 root  wheel     0x47 Sep 29 12:20 ttyv3
crw-------  1 root  wheel     0x48 Sep 29 12:20 ttyv4
crw-------  1 root  wheel     0x49 Sep 29 12:20 ttyv5
crw-------  1 root  wheel     0x4a Sep 29 12:20 ttyv6
crw-------  1 root  wheel     0x4b Sep 29 12:20 ttyv7
crw-------  1 root  wheel     0x4c Sep 29 12:19 ttyv8
crw-------  1 root  wheel     0x4d Sep 29 12:19 ttyv9
crw-------  1 root  wheel     0x4e Sep 29 12:19 ttyva
crw-------  1 root  wheel     0x4f Sep 29 12:19 ttyvb
crw-------  1 root  wheel     0x36 Sep 29 12:19 ufssuspend
lrwxr-xr-x  1 root  wheel        9 Sep 29 12:45 ugen0.1 -> usb/0.1.0
lrwxr-xr-x  1 root  wheel        9 Sep 29 12:51 ugen0.2 -> usb/0.2.0
lrwxr-xr-x  1 root  wheel        9 Sep 29 12:45 ugen1.1 -> usb/1.1.0
lrwxr-xr-x  1 root  wheel        6 Sep 29 12:45 urandom -> random
dr-xr-xr-x  2 root  wheel      512 Sep 29 12:45 usb
crw-r--r--  1 root  operator  0x3b Sep 29 12:19 usbctl
crw-------  1 root  operator  0x3c Sep 29 12:19 xpt0
crw-rw-rw-  1 root  wheel     0x10 Sep 29 12:19 zero
crw-rw-rw-  1 root  operator  0x37 Sep 29 12:19 zfs

Now after updating to FreeNAS 11.2-RC2 my jail console no longer shows the USB device (among a huge list of other things)
sudo iocage console hass-dev

Code:

root@hass-dev:~ # ls -l /dev
total 1
crw-------  1 root  wheel     0x13 Nov 16 17:21 bpf
lrwxr-xr-x  1 root  wheel        3 Sep 29 13:22 bpf0 -> bpf
crw-rw-rw-  1 root  wheel     0x3a Nov 16 17:20 crypto
dr-xr-xr-x  2 root  wheel      512 Nov 16 17:20 fd
crw-rw-rw-  1 root  wheel      0xf Nov 16 17:21 null
crw-rw-rw-  1 root  wheel     0x12 Nov 16 17:20 ptmx
dr-xr-xr-x  2 root  wheel      512 Nov 16 17:21 pts
crw-r--r--  1 root  wheel      0x8 Sep 29 13:22 random
lrwxr-xr-x  1 root  wheel        4 Sep 29 13:22 stderr -> fd/2
lrwxr-xr-x  1 root  wheel        4 Sep 29 13:22 stdin -> fd/0
lrwxr-xr-x  1 root  wheel        4 Sep 29 13:22 stdout -> fd/1
lrwxr-xr-x  1 root  wheel        6 Sep 29 13:22 urandom -> random
crw-rw-rw-  1 root  wheel     0x10 Sep 29 13:22 zero
crw-rw-rw-  1 root  operator  0x37 Nov 16 17:20 zfs

Any direction, links or even the proper search terms for Google to information on how to re-enable my USB device inside the jail would be greatly appreciated.

 

[tprelog]

Setting my Home Assistant jail properties devfs_ruleset=3has restored the use of my USB z-wave controller inside the jail

I actually still use the command line for iocage so
sudo iocage set devfs_ruleset=3 jail-name
Opposite the web-GUI, using the command line allows changing jail properties (and mounting storage) while the jail is running. However in this case a full-restart of the jail was necessary for the change to take effect.
( -s soft-restart did not work)
sudo iocage restart jail-name

I don't even know how I figured that out (really just guessing with trail and error) but it works. Now listing all devices from the jail console seems to be the full list again
sudo iocage console hass-dev

Code:

root@hass-dev:~ # ls -l /dev
total 4
crw-r--r--  1 root  wheel	 0x2a Nov 16 23:44 acpi
crw-r-----  1 root  operator  0x5c Nov 16 23:44 ada0
crw-r-----  1 root  operator  0x5e Nov 16 23:44 ada0p1
crw-r-----  1 root  operator  0x5f Nov 16 23:44 ada0p2
crw-r-----  1 root  operator  0x5d Nov 16 23:44 ada1
crw-r-----  1 root  operator  0x60 Nov 16 23:44 ada1p1
crw-r-----  1 root  operator  0x64 Nov 16 23:45 ada1p1.eli
crw-r-----  1 root  operator  0x61 Nov 16 23:44 ada1p2
crw-rw-r--  1 root  operator  0x2c Nov 16 23:44 apm
crw-rw----  1 root  operator  0x2b Nov 16 23:44 apmctl
crw-------  1 root  wheel	 0x32 Nov 16 23:44 atkbd0
crw-------  1 root  kmem	   0xc Nov 16 23:44 audit
crw-------  1 root  wheel	  0xb Nov 16 23:44 auditpipe
crw-------  1 root  wheel	 0x13 Nov 16 23:53 bpf
lrwxr-xr-x  1 root  wheel		3 Nov 16 23:53 bpf0 -> bpf
crw-rw-rw-  1 root  wheel	 0x35 Nov 16 23:44 bpsm0
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:53 cam
crw-r-----  1 root  operator  0x50 Nov 16 23:44 cd0
crw-------  1 root  wheel	  0x5 Nov 16 23:49 console
crw-------  1 root  wheel	 0x1f Nov 16 23:44 consolectl
crw-r-----  1 root  kmem	  0x1c Nov 16 23:44 cpuctl0
crw-r-----  1 root  kmem	  0x1d Nov 16 23:44 cpuctl1
crw-rw-rw-  1 root  wheel	 0x3a Nov 16 23:44 crypto
crw-rw-rw-  1 root  wheel	  0xd Nov 16 23:44 ctty
crw-rw----  1 uucp  dialer	0x70 Nov 16 23:45 cuaU0
crw-rw----  1 uucp  dialer	0x71 Nov 16 23:45 cuaU0.init
crw-rw----  1 uucp  dialer	0x72 Nov 16 23:45 cuaU0.lock
crw-------  1 root  wheel	  0x6 Nov 16 23:44 devctl
crw-------  1 root  wheel	  0x7 Nov 16 23:44 devctl2
cr--r--r--  1 root  wheel	 0x43 Nov 16 23:44 devstat
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:53 dtrace
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:44 fd
crw-------  1 root  wheel	 0x21 Nov 16 23:44 fido
crw-rw-rw-  1 root  wheel	  0xe Nov 16 23:44 full
crw-r-----  1 root  operator   0x4 Nov 16 23:44 geom.ctl
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:53 gptid
crw-------  1 root  wheel	 0x1e Nov 16 23:44 io
crw-------  1 root  wheel	 0x22 Nov 16 23:44 iscsi
lrwxr-xr-x  1 root  wheel		6 Nov 16 23:53 kbd0 -> atkbd0
lrwxr-xr-x  1 root  wheel		7 Nov 16 23:53 kbd1 -> kbdmux0
crw-------  1 root  wheel	 0x24 Nov 16 23:44 kbdmux0
crw-------  1 root  wheel	 0x28 Nov 16 23:44 klog
crw-r-----  1 root  kmem	  0x27 Nov 16 23:44 kmem
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:53 led
lrwxr-xr-x  1 root  wheel	   14 Nov 16 23:53 log -> ../var/run/log
crw-------  1 root  wheel	  0xa Nov 16 23:44 mdctl
crw-r-----  1 root  kmem	  0x26 Nov 16 23:44 mem
crw-r-----  1 root  operator  0x55 Nov 16 23:44 mlx5ctl
crw-------  1 root  kmem	  0x23 Nov 16 23:44 nfslock
crw-rw-rw-  1 root  wheel	  0xf Nov 17 00:07 null
crw-------  1 root  operator  0x44 Nov 16 23:44 pass0
crw-------  1 root  operator  0x45 Nov 16 23:44 pass1
crw-------  1 root  operator  0x46 Nov 16 23:44 pass2
crw-r--r--  1 root  wheel	 0x11 Nov 16 23:44 pci
crw-rw-rw-  1 root  wheel	 0x34 Nov 16 23:44 psm0
crw-rw-rw-  1 root  wheel	 0x12 Nov 16 23:44 ptmx
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:53 pts
crw-r--r--  1 root  wheel	  0x8 Nov 16 23:45 random
crwx------  1 root  wheel	 0x54 Nov 16 23:44 rdma_cm
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:53 reroot
crw-------  1 root  wheel	 0x1b Nov 16 23:44 snp
lrwxr-xr-x  1 root  wheel		4 Nov 16 23:53 stderr -> fd/2
lrwxr-xr-x  1 root  wheel		4 Nov 16 23:53 stdin -> fd/0
lrwxr-xr-x  1 root  wheel		4 Nov 16 23:53 stdout -> fd/1
crw-------  1 root  wheel	 0x20 Nov 16 23:44 sysmouse
crw-------  1 root  wheel	 0x6d Nov 16 23:45 ttyU0
crw-------  1 root  wheel	 0x6e Nov 16 23:45 ttyU0.init
crw-------  1 root  wheel	 0x6f Nov 16 23:45 ttyU0.lock
crw-------  1 root  wheel	 0x47 Nov 16 23:45 ttyv0
crw-------  1 root  wheel	 0x48 Nov 16 23:45 ttyv1
crw-------  1 root  wheel	 0x49 Nov 16 23:45 ttyv2
crw-------  1 root  wheel	 0x4a Nov 16 23:45 ttyv3
crw-------  1 root  wheel	 0x4b Nov 16 23:45 ttyv4
crw-------  1 root  wheel	 0x4c Nov 16 23:45 ttyv5
crw-------  1 root  wheel	 0x4d Nov 16 23:45 ttyv6
crw-------  1 root  wheel	 0x4e Nov 16 23:45 ttyv7
crw-------  1 root  wheel	 0x4f Nov 16 23:44 ttyv8
crw-------  1 root  wheel	 0x51 Nov 16 23:44 ttyv9
crw-------  1 root  wheel	 0x52 Nov 16 23:44 ttyva
crw-------  1 root  wheel	 0x53 Nov 16 23:44 ttyvb
crw-------  1 root  wheel	 0x36 Nov 16 23:44 ufssuspend
lrwxr-xr-x  1 root  wheel		9 Nov 16 23:53 ugen0.1 -> usb/0.1.0
lrwxr-xr-x  1 root  wheel		9 Nov 16 23:53 ugen0.2 -> usb/0.2.0
lrwxr-xr-x  1 root  wheel		9 Nov 16 23:53 ugen1.1 -> usb/1.1.0
lrwxr-xr-x  1 root  wheel		6 Nov 16 23:53 urandom -> random
dr-xr-xr-x  2 root  wheel	  512 Nov 16 23:53 usb
crw-r--r--  1 root  operator  0x3b Nov 16 23:44 usbctl
crw-------  1 root  operator  0x3c Nov 16 23:44 xpt0
crw-rw-rw-  1 root  wheel	 0x10 Nov 16 23:44 zero
crw-rw-rw-  1 root  operator  0x37 Nov 16 23:44 zfs

It's really frustrating to search for answers when you don't know, what you don't know! Now that I can add devfs_ruleset to my Google search I can actually find relevant information to better understand what's happening here.

 

[nuneya]

This is great, and I'm glad I finally hit this after an hour of searching and trying to change devfs.rules to no effect. I wish there was an explanation though as to why this works..? When I run devfs rule -s 3 show, cua* does not appear there, so why exactly does this help?

 

[tprelog]

This is great, and I'm glad I finally hit this after an hour of searching and trying to change devfs.rules to no effect. I wish there was an explanation though as to why this works..? When I run devfs rule -s 3 show, cua* does not appear there, so why exactly does this help?

I haven't figured out the devfs.rules yet. No matter what I try, even with basic examples, I could never get any changes to /dev when trying to use a custom devfs.rule.

devfs_ruleset=3 seems to make the full /dev tree available inside the jail. I'm really learning everything about devfs.rules for the first time after I stumbled on the solution.

 

[wintered]

So I have exactly the same problem, but with my HUSBZB device!

I was also using devfs_ruleset=4. Looking at the file /dev/default/devfs.rules, it defines the rulesets. The wierd things is, that the ruleset 4 is suppose to include ruleset 3, so I don't understand why that makes a different.

Either way cua* isn't listed in ruleset 3 either :S, so confusing.

I tried to add the cua devices to ruleset 4 by adding the following as Init/Shutdown Scripts through the gui:
devfs rule -s 4 add path 'cua*' unhide

However after restarting the OS, the jail still wasn't coming up with the available devices.

Examing the ruleset, everything looks good:
root@ferret:~ # devfs rule -s 4 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 path cua* unhide


Manually forcing the jail to the correct ruleset works though?
devfs -m /mnt/iocage/jails/home/root/dev rule -s 4 applyset

Setting the ruleset to 3, thanks @tprelog does work, although it really shouldnt.

So my take away here, is that the jails are using a different set of devfs rulesets, other than the ones defined in /etc/default/devfs.rules.

 

[tprelog]

So I have exactly the same problem, but with my HUSBZB device!

I was also using devfs_ruleset=4. Looking at the file /dev/default/devfs.rules, it defines the rulesets. The wierd things is, that the ruleset 4 is suppose to include ruleset 3, so I don't understand why that makes a different.

Either way cua* isn't listed in ruleset 3 either :S, so confusing.

I tried to add the cua devices to ruleset 4 by adding the following as Init/Shutdown Scripts through the gui:
devfs rule -s 4 add path 'cua*' unhide

However after restarting the OS, the jail still wasn't coming up with the available devices.

Examing the ruleset, everything looks good:
root@ferret:~ # devfs rule -s 4 show
100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 path cua* unhide


Manually forcing the jail to the correct ruleset works though?
devfs -m /mnt/iocage/jails/home/root/dev rule -s 4 applyset

Setting the ruleset to 3, thanks @tprelog does work, although it really shouldn't.

So my take away here, is that the jails are using a different set of devfs rulesets, other than the ones defined in /etc/default/devfs.rules.

@wintered Thank you.

I'm going try your approach. After I figured out changing the devfs.rules was the fix, I've been trying to figure out how to do exactly this.

 

[tim.bishop]

Setting the ruleset to 3, thanks @tprelog does work, although it really shouldn't.

The reason ruleset 3 works is because it doesn't include the earlier rules. It's ruleset 1 that hides everything, and subsequent rules unhide certain devices. So setting a jail to ruleset 3 ends up showing everything because nothing is hidden.

What iocage is now doing (since when? at least 11.2 - 11.1 was fine) is creating a new ruleset for each jail as it starts it. If the jail is set to use any ruleset other than 4 it just copies it verbatim to the new ruleset and uses it. However, if it's set to ruleset 4, it produces its own default that looks like 4 would before being modified. So the old hack of tweaking ruleset 4 in a pre-init task no longer works.

For the more common use case of tun devices (usually for OpenVPN) it's as simple as enabling allow_tun (at least, in 11.2-RELEASE). But for /dev/cua*, like this case, you need to bodge around things again. This is what's working for me - I have a pre-init task containing:

devfs rule -s 5 add include 1; devfs rule -s 5 add include 2; devfs rule -s 5 add include 3; devfs rule -s 5 add include 4; devfs rule -s 5 add path 'cua*' unhide

To be honest, it might not need to include them all, but I went with the default set and then added my own rule on the end. Note that includes aren't recursive, so including 4 doesn't include 1-3.

Then I set my jail to use devfs rule 5. This gave the same rules as I previously had by modifying rule 4 in the previous FreeNAS release.

Hope that helps somebody!

 

[dmshimself]

I also wanted to access the cua devices from a jail and used this ruleset with 11.2 released version. When I try to start the jail, I get the error shown below. I have a working jail that ideally has vnet enabled, so I went with the less secure rule set 3 and that worked. If anyone knows how to resolve this, I'd be much happier using the idea of a rule set such as Tim produced. But I'm not really sure what this error means....

Code:

Error: concurrent.futures.process._RemoteTraceback:
"""
Traceback (most recent call last):
  File "/usr/local/lib/python3.6/concurrent/futures/process.py", line 175, in _process_worker
    r = call_item.fn(*call_item.args, **call_item.kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 128, in main_worker
    res = loop.run_until_complete(coro)
  File "/usr/local/lib/python3.6/asyncio/base_events.py", line 468, in run_until_complete
    return future.result()
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 88, in _run
    return await self._call(f'{service_name}.{method}', serviceobj, methodobj, params=args, job=job)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/worker.py", line 81, in _call
    return methodobj(*params)
  File "/usr/local/lib/python3.6/site-packages/middlewared/schema.py", line 668, in nf
    return f(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/plugins/jail.py", line 542, in start
    iocage.start()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 1654, in start
    callback=self.callback
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 66, in __init__
    self.__start_jail__()
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_start.py", line 481, in __start_jail__
    _callback=self.callback)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 81, in logit
    _callback(content, exception)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_common.py", line 64, in callback
    raise callback_exception(message)
RuntimeError:
Stopped home assistant due to VNET failure
"""

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 161, in call_method
    result = await self.middleware.call_method(self, message)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1109, in call_method
    return await self._call(message['method'], serviceobj, methodobj, params, app=app, io_thread=False)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1046, in _call
    return await self._call_worker(serviceobj, name, *args)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1073, in _call_worker
    job,
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 1004, in run_in_proc
    return await self.run_in_executor(self.__procpool, method, *args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/middlewared/main.py", line 989, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
RuntimeError:
Stopped home assistant due to VNET failure

 

[arcelle]

I'm running FreeNAS 11.2-RELEASE-U1 and have been fighting the USB pass through challenge to my Home Assistant jail. Thanks tprelog for all your helpful guides. Just as I was about to give up and use a raspberrypi, I want back to the FreeNAS User Guide, chapter 14 - Jails and discovered the following section that gave me the info needed to get access to my Zwave USB device in the jail.

I set up a new jail for Home Assistant and during the jail setup in the FreeNAS GUI I followed the above guide to allow for devfs_ruleset 3 which opens up all USB devices within the jail. BTW, I set enforce_statfs to 0 (zero) and the devfs ruleset to 3 in addition to the other settings listed above. There may be iocage CLI commands to do the same, but I'm a rank amateur, so beyond my current skills.

 

[tprelog]

OK I think I finally have this figured out. At least in the sense of a desired end result anyways. Thanks again @wintered and @tim.bishop who have both hinted at a better approach than devfs_ruleset=3.

What may be unclear to a novice user such as myself is the fact that it's the devfs.rules on the FreeNAS host it's self that need to be modified (Not rulesets inside the jail). In other words we create a custom devfs_ruleset on the FreeNAS host which allows us to expose our z-wave (or zigbee) USB sticks. Then set simply set the jail to use our new custom ruleset. Exactly how to do this (while seems simple now) was not so obvious at first.

Just to recap, on FreeNAS 11.2-RELEASE iocage jails, by default, have only a limited number of devices exposed in /dev Our zwave and zigbee devices are not in this list.

Code:

sudo iocage exec homeassistant ls /dev

bpf     bpf0    crypto  fd      null    ptmx    pts     random  stderr  stdin   stdout  urandom zero    zfs

Setting devfs_rulset=3 is a quick fix but exposes more of /dev then may be desired. (Bottom Post #2 shows list)

For zwave and zigbee USB devices to work we only only need to expose/dev/cua* . To achieve this we can use a custom devfs_ruleset. Since my OP I have switch to the GoControl HUSBZB-1 which is both a zwave and zigbee radio. The zwave radio remains /dev/cuaU0 and the zigbee radio is /dev/cuaU1. This seems a more desired result. Below is /dev tree using custom devfs_ruleset

Code:

sudo iocage exec homeassistant ls /dev

bpf             cuaU0           cuaU1           fd              ptmx            stderr          urandom
bpf0            cuaU0.init      cuaU1.init      log             pts             stdin           zero
crypto          cuaU0.lock      cuaU1.lock      null            random          stdout          zfs

A few quick notes before create our custom devfs_ruleset.
- Actually we are creating a script that will create the custom devfs_ruleset for us
- The name of this script trivial. In this example I called mine zwave-ruleset.sh
- This will create devfs_ruleset 99 on FreeNAS.
- I'm using 99 because it seems unlikely this would other wise be used by the host.
- Only rulesets 1-4 are defined in /etc/defaults/devfs.rules but I have no idea what voodoo the middle-ware is performing

First create a simple script zwave-ruleset.sh on your FreeNAS. This as mentioned is what will actually create the custom ruleset. I just used the console and created this file in my home directory with the following contents - Note: change 99 in ruleNum=99 if you want a different ruleset number:

ee zwave-ruleset.sh

Code:

#!/bin/sh

# Create custom devfs_ruleset with this NUMBER
ruleNum=99

/sbin/devfs rule -s ${ruleNum} add include 1
/sbin/devfs rule -s ${ruleNum} add include 2
/sbin/devfs rule -s ${ruleNum} add include 3
/sbin/devfs rule -s ${ruleNum} add path zfs unhide
/sbin/devfs rule -s ${ruleNum} add path 'bpf*' unhide
/sbin/devfs rule -s ${ruleNum} add path 'cua*' unhide

Be sure to make the script executable
chmod +x zwave-ruleset.sh

Simply running this script from the console works but these changes will be lost after reboot. (We'll fix that at the end) That's ok for now. We can still use this to create our ruleset and test everything works before adding it as a start-up script in the FreeNAS GUI.
sudo sh zwave-ruleset.sh

You can view the ruleset has been created correctly. sudo devfs rule -s 99 show should return the following:

Code:

sudo devfs rule -s 99 show
Password:

100 include 1
200 include 2
300 include 3
400 path zfs unhide
500 path bpf* unhide
600 path cua* unhide

I will mention here, when creating a new jail, it is possible to set the devfs_ruleset during creation as well. Just to clear though, you must create the custom ruleset before this will work! Example to create a new jail for Home Assistant using devfs_ruleset=99 we have just created above.
sudo iocage create -r 11.2-RELEASE boot=on dhcp=on bpf=yes vnet=on devfs_ruleset=99 -n homeassistant

If you're jail is already installed you just need to set the jail to use new ruleset. You can do this from the GUI as shown in the 2nd post (enter 99 instead of 3) or from the console. You'll need to restart the jail before it will use the new ruleset.

Here I find the console quicker. In this case the jail name is `homeassistant`.
sudo iocage set devfs_ruleset=99 homeassistant
sudo iocage restart homeassistant

If everything is working as expected, finally add zwave-ruleset.sh as a startup script in the FreeNAS GUI.
This final step is how we get FreeNAS to automatically recreate our custom ruleset during every (re)boot.

 

[tprelog]

@dmshimself I'm sorry I can't help specifically with your error but if your jail is working correctly with devfs_ruleset=3 maybe the method I posted above will work. I literally just figured this out today. Hopefully it works for everybody

@arcelle

BTW, I set enforce_statfs to 0 (zero)...

I started with those changes as well. In the end I found, at least in this case, that was not needed have reverted those settings back to their default values.

 

[dmshimself]

@dmshimself I'm sorry I can't help specifically with your error but if your jail is working correctly with devfs_ruleset=3 maybe the method I posted above will work. I literally just figured this out today. Hopefully it works for everybody

@arcelle

I started with those changes as well. In the end I found, at least in this case, that was not needed have reverted those settings back to their default values.

I went through this excellently described process and it worked fine for me. Many thanks indeed.

Perhaps a rule set editor as part of FN would be good so that any upgrades can take account of such things