Creating (and testing) iocage plugins

[diskdiddler]

I see the create a plugin documentation has now been published.
https://doc.freenas.org/11.2/plugins.html#create-a-plugin

So I'm starting to play around, figure out the commands to install my test plugin, I've got a git account and I have cloned the existing repository for plugin. (Downloaded a zip?)

What I want to do, is make a test plugin, which I can install, into my NAS without hassling the developers. How do I tell my FreeNAS machine "hey, look here for plugins" and point it to my C: or even my git repository?

Yes, this is a pretty elementary question from someone inexperienced, but I'm not seeing the answer in the guide?


EDIT: Thanks to Jurgen and Yoda in this thread, I've made my first (local) plugin for iocage

 

[Jurgen Segaert]

There's a command-line option to install a plugin from a local file. I'll post something more elaborate when I have a chance.

 

[diskdiddler]

Thanks very much Jurgen!

I was wondering if I had to set up some kind of development version of freenas in a vm and compile it, all kinds of stuff. .

 

[diskdiddler]

I have started messing around and made a VM to play with and learn.

Here's my file
sabnzbdtest.json

{
"name": "sabnzbdtest",
"release": "11.2-RELEASE",
"artifact": "/mnt/POOLNAME/dataset/Plugins/sabnzbdtest/",
"pkgs": [
"news/sabnzbdplus"
],
"packagesite": "http://pkg.cdn.trueos.org/iocage",
"fingerprints": {
"iocage-plugins": [
{
"function": "sha256",
"fingerprint": "226efd3a126fb86e71d60a37353d17f57af816d1c7ecad0623c21f0bf73eb0c7"
}
]
},
"official": false
}

You can see I've just borrowed someone elses json and tweaked it, to learn.
I have attempted redirecting the artifacts to a local folder.

Inside this folder is a ui.json, post_install.sh as per the instructions linked above.
To be fair, I have no idea what I'm doing, so, sorry for being impatient, but I thought I'd have a crack.

Here's my (guess!) at the install command.
iocage fetch -P /mnt/POOLNAME/dataset/Plugins/sabnzbdtest.json --name sabnzbdtest
Here's my error message:

Code:

root@freenas:/mnt/POOLNAME/dataset/Plugins # iocage fetch -P /mnt/POOLNAME/dataset/Plugins/sabnzbdtest.json --name sabnzbdtest ip4_address=vnet0
Plugin: sabnzbdtest
  Official Plugin: True
  Using RELEASE: 11.2-RELEASE
  Using Branch: 11.2-RELEASE
  Post-install Artifact: /mnt/POOLNAME/dataset/Plugins/sabnzbdtest/
  These pkgs will be installed:
	- news/sabnzbdplus
Traceback (most recent call last):
  File "/usr/local/bin/iocage", line 10, in <module>
	sys.exit(cli())
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 722, in __call__
	return self.main(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 697, in main
	rv = self.invoke(ctx)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 1066, in invoke
	return _process_result(sub_ctx.command.invoke(sub_ctx))
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 895, in invoke
	return ctx.invoke(self.callback, **ctx.params)
  File "/usr/local/lib/python3.6/site-packages/click/core.py", line 535, in invoke
	return callback(*args, **kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_cli/fetch.py", line 127, in cli
	ioc.IOCage().fetch(**kwargs)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/iocage.py", line 930, in fetch
	).fetch_plugin(name, props, 0, accept)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_plugin.py", line 122, in fetch_plugin
	props, pkg = self.__fetch_plugin_props__(conf, props, num)
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_plugin.py", line 341, in __fetch_plugin_props__
	for p in props)] + create_props
  File "/usr/local/lib/python3.6/site-packages/iocage_lib/ioc_plugin.py", line 340, in <listcomp>
	create_props = [f"{k}={v}" for k, v in (p.split("=")
ValueError: not enough values to unpack (expected 2, got 1)
root@freenas:/mnt/POOLNAME/dataset/Plugins #  

Again, thanks to anyone helping at all, hope this helps others too.

 

[Jurgen Segaert]

I posted a resource: https://forums.freenas.org/index.php?resources/create-an-unofficial-iocage-plugin.99/

 

[diskdiddler]

I don't know if we can edit / contribute to these docs but I'm having issues already, sorry,

Code:

That’s it! Before pushing the changes back to GitHub, initialize your global settings. (This is a one-time activity)
Code:
git config --global user.name "<Your github user name>"
git config --global user.email "<Your GitHub email address>"

That's not working for me,

Error message is :

error: key does not contain a section: jaxjexjox
(My GitUsername)

Command is:
git config --global jaxjexjox "jaxjexjox"

 

[Borre]

You should only change "<Your github user name>" as the text says, not user.name.

 

[Jurgen Segaert]

Correct, just do e.g. git config --global user.name "jaxjexjox"

 

[diskdiddler]

I literally said out loud, OH. That took a while.
I wonder how we can re-word that. Very dumb of me, thanks Jurgen, I'll keep fiddling today!

 

[diskdiddler]

Correct, just do e.g. git config --global user.name "jaxjexjox"

So I recommend you change the documentation to be like this.

git config --global user.name "<Your github user name>"
git config --global user.email "<Your GitHub email address>"
(Bold)

Alternatively re-word this
"That’s it! Before pushing the changes back to GitHub, initialize your global settings. (This is a one-time activity)"
To:
"That’s it! Before pushing the changes back to GitHub, initialize your global settings, by configuring the user.name and user.email flags. (this is a one-time activity)

Finally, most important, I'm stuck! :(

Code:

root@freenas:/mnt/POOLNAME/dataset/plugins/freenasplugintest # chmod +x post_install.sh
chmod: post_install.sh: Operation not permitted

(Yeah, I'm doing it as root, initially, just on my test VM to learn, will this impact things?)

Thanks

 

[8-bit Yoda]

You have your permissions clobbered by doing this on a Windows share. Search the forum for "operation not permitted" or "setfacl -b".

 

[diskdiddler]

Yoda, thank you!
I tried fixing this, it got messy so to ensure it was no longer an issue, I'm doing all my work in a unix dataset now, via nano. I assume 'as intended' by BSD-type people!


Anyhow! My progress continues, I am stuck yet again, but I'm getting closer.


First things first. Jurgen, how do I produce a debug output of what is going on?
My script is "Running post_install.sh" but not showing me what's going on within that script.

root@freenas:/mnt/POOLNAME/dataset/plugins # iocage fetch -P -n freenasplugintest.json vnet=on dhcp=on bpf=yes devfs_ruleset=5 interfaces=vnet0:bridge0 host_hostname=freenasplugintest
Plugin: sabnzbdtest
Official Plugin: False
Using RELEASE: 11.1-RELEASE
Using Branch: 11.2-RELEASE
Post-install Artifact: https://github.com/jaxjexjox/freenasplugintest.git
These pkgs will be installed:
- news/sabnzbdplus

Testing SRV response to iocage-plugins
Testing DNSSEC response to iocage-plugins

Installing plugin packages:
- news/sabnzbdplus...

Fetching artifact...

Branch 11.2-RELEASE does not exist at https://github.com/jaxjexjox/freenasplugintest.git!
Using "master" branch for plugin, this may not work with your RELEASE

Running post_install.sh

Command output:
An error occured! Please read above
root@freenas:/mnt/POOLNAME/dataset/plugins #

The way I'm reading that, it sounds like it's annoyed about 11.2-RELEASE not existing however it's not clear that is the primary cause of the failure?
(The jail does not get created, it's auto-deleted, due to the failure)


If you guys would like to review my newbie work, it's here.
https://github.com/jaxjexjox/freenasplugintest

My json is here.

Code:

{
  "name": "sabnzbdtest",
  "release": "11.1-RELEASE",
  "artifact": "https://github.com/jaxjexjox/freenasplugintest.git",
  "pkgs": [
	"news/sabnzbdplus"
  ],
  "packagesite": "http://pkg.cdn.trueos.org/iocage",
  "fingerprints": {
		  "iocage-plugins": [
				  {
				  "function": "sha256",
				  "fingerprint": "226efd3a126fb86e71d60a37353d17f57af816d1c7ecad0623c21f0bf73eb0c7"
		  }
		  ]
  },
  "official": false
}

Final note: I have manually created a working sabnzbd jail, simply by installing the package from freshports and then replacing the ini file as per my script.
I know that this particular package is already planned officially by the developers, however - since I know the exact steps, to create the jail from scratch, install the software and 'fix' the software, I figure this is a great test case for me.

Sorry for the long reply, really wish we were all in the same timezone.

 

[Jurgen Segaert]

I'm doing all my work in a unix dataset now, via nano. I assume 'as intended' by BSD-type people!

I like to use ee as editor. There's a resource on it here: https://forums.freenas.org/index.php?resources/editing-files-in-freenas-simple-and-easy-with-ee.71/

 

[diskdiddler]

Thanks Jurgen, I'll try either.
Any idea about the rest of the post, I wasn't expecting anyone awake at this time! I'd love to be able to continue and learn more tonight (it's only 5pm)

 

[Jurgen Segaert]

To troubleshoot the issue at hand:

I created a file sabnzbdtest.json with the JSON data you listed and tried to create the jail.

Notes:

• If you're working as root, you can ommit the sudo
• I'm using the option --keep_jail_on_failure True. This is brand new functionality to prevent a jail from being destroyed when there's an error. Based on the release you're using, the jail may or may not be cleaned up automatically whenever an error occurs, and that option may or may not work. I'm using the nightlies train FreeNAS-11.2-MASTER-201809080900 (85d5c75)

Code:

jurgen@freenas:/mnt/tank/demo % sudo iocage fetch -P -n sabnzbdtest.json ip4_addr="em0|192.168.0.219/24" --keep_jail_on_failure True
Plugin: sabnzbdtest
  Official Plugin: False
  Using RELEASE: 11.1-RELEASE
  Using Branch: 11.2-RELEASE
  Post-install Artifact: https://github.com/jaxjexjox/freenasplugintest.git
  These pkgs will be installed:
	- news/sabnzbdplus

Testing SRV response to iocage-plugins
Testing DNSSEC response to iocage-plugins

Installing plugin packages:
  - news/sabnzbdplus...

Fetching artifact...

Branch 11.2-RELEASE does not exist at https://github.com/jaxjexjox/freenasplugintest.git!
Using "master" branch for plugin, this may not work with your RELEASE

Running post_install.sh

Command output:
An error occured! Please read above

Same error. I go into the jail's console...

Code:

jurgen@freenas:/mnt/tank/demo % sudo iocage console sabnzbdtest
FreeBSD 11.2-STABLE (FreeNAS.amd64) #0 dac728946(freenas/11-stable): Sat Sep  8 05:52:49 EDT 2018

Welcome to FreeBSD!

Release Notes, Errata: https://www.FreeBSD.org/releases/
Security Advisories:   https://www.FreeBSD.org/security/
FreeBSD Handbook:	  https://www.FreeBSD.org/handbook/
FreeBSD FAQ:		   https://www.FreeBSD.org/faq/
Questions List: https://lists.FreeBSD.org/mailman/listinfo/freebsd-questions/
FreeBSD Forums:		https://forums.FreeBSD.org/

Documents installed with the system are in the /usr/local/share/doc/freebsd/
directory, or can be installed later with:  pkg install en-freebsd-doc
For other languages, replace "en" with a language code like de or fr.

Show the version of FreeBSD installed:  freebsd-version ; uname -a
Please include that output and any error messages when posting questions.
Introduction to manual pages:  man man
FreeBSD directory layout:	  man hier

Edit /etc/motd to change this login announcement.
root@sabnzbdtest:~ #

...and execute the post_install.sh script manually

Code:

root@sabnzbdtest:~ # ./post_install.sh
Starting sabnzbd.
Shared object "libdl.so.1" not found, required by "python2.7"
/usr/local/etc/rc.d/sabnzbd: WARNING: failed to start sabnzbd
Cannot 'stop' sabnzbd. Set sabnzbd_enable to YES in /etc/rc.conf or use 'onestop' instead of 'stop'.
usage: sleep seconds
mv: /usr/local/sabnzbdprep.ini: No such file or directory
Starting sabnzbd.
Shared object "libdl.so.1" not found, required by "python2.7"
/usr/local/etc/rc.d/sabnzbd: WARNING: failed to start sabnzbd
root@sabnzbdtest:~ #

The issue is this: Shared object "libdl.so.1" not found, required by "python2.7"

Anyway, get rid of that iXSystems repo and instead use vanilla FreeBSD; your JSON manifest should like like this:

Code:

{
  "name": "sabnzbdtest",
  "release": "11.1-RELEASE",
  "artifact": "https://github.com/jaxjexjox/freenasplugintest.git",
  "pkgs": [
	"news/sabnzbdplus"
  ],
  "packagesite": "http://pkg.FreeBSD.org/FreeBSD:11:amd64/quarterly",
  "fingerprints": {
		  "iocage-plugins": [
				  {
				  "function": "sha256",
				  "fingerprint": "b0170035af3acc5f3f3ae1859dc717101b4e6c1d0a794ad554928ca0cbb2f438"
		  }
		  ]
  },
  "official": false
}

Also in your post_install file, you want to add the following:

Code:

sysrc sabnzbd_enable=YES
service sabnzbd start

With these changes, it works fine.

 

[diskdiddler]

EDIT:
Scratch that, I've fixed it and thank you so much!

Now I need to learn, sorry multiple questions


1, I continued to have issues for HOURS despite copying every single little thing in your post where you got it working above!
The fault was my post_install.sh had "sleep 5s" not "sleep 5" in it, so it was erroring out.

My question: How/why did it work for you, if we're both using my freenasplugintest.git, with the same post_install.sh (?!?)



2, You told me to change the repo from the ixsystems one to a FreeBSD one.

My question(s): Why do I need to do this, should I always be using the FreeBSD one, how can I identify this in future? So I can learn from it.




BTW, the --keep jail : true thing? Works in Beta 3 already, not just nightly, it was handy, thank you!

 

[Jurgen Segaert]

Congratulations on getting it to work. Persistence goes a long way.

1) The main stumbling block was that missing libdl.so.1 library that prevented the
sabnzbd from starting. I just ran the commands manually in the jail as a quick-n-dirty test. Since I can't really change your repo anyway, the end to end test was up to you.
2) The TrueOS repo only seems to have a subset of the FreeBSD packages. Also, I suspect it is compiled for 11.2 and have some quirks in 11.1; when I searched the internet for that error message, most results pointed to incompatibility issue. Switching to the std. FreeBSD repo fixed the missing library issue, allowed the service to start and the admin webpage worked as expected.

Anyway, based on the complexity of the plugin, the process may have quite a few iterations: test the various commands at the command line, take notes, when you get the desired result, update the script, push the changes to GitHub and run the whole process again.

 

[diskdiddler]

Ok so I've been asked by someone to show how to get this working for other users, so here goes.

_PLEASE NOTE, I'M AN AMATEUR AND THIS MIGHT BREAK LATER ON AND I CAN'T GUARANTEE UPDATES!_
Going forward, the FreeNAS team or Josh or someone, will actually do this properly, mine has several dumb flaws in it.
(Example, there's a section where text should be EDITED by the installer, mine just replaces the whole file with an edited version, long term this will break stuff!)


That being said.
Shell into your FreeNAS machine as root.

Change to a directory you feel comfortable writing a file into, I believe you can delete it later.
Like "cd /mnt/tank/media"
or "cd /mnt/POOL/data/temp"
etc

Type in

"nano freenasplugintest.json"
Copy this text below.

Code:

{
  "name": "sabnzbdtest",
  "release": "11.2-RELEASE",
  "artifact": "https://github.com/jaxjexjox/freenasplugintest.git",
  "pkgs": [
   "news/sabnzbdplus"
  ],
  "packagesite": "http://pkg.cdn.trueos.org/iocage",
  "fingerprints": {
		 "iocage-plugins": [
				 {
				 "function": "sha256",
				 "fingerprint": "226efd3a126fb86e71d60a37353d17f57af816d1c7ecad0623c21f0bf73eb0c7"
		 }
		 ]
  },
  "official": false
}

Use shift insert to paste it in to Nano editor. With the right terminal package, the formatting should stay correct.
Press CTRL-O to write out the changes
Press CTRL-X to exit the editor.


Now you're back to the prompt, now copy this command and shift insert it into the shell / terminal.

iocage fetch -P -n freenasplugintest.json vnet=on dhcp=on bpf=yes devfs_ruleset=5 interfaces=vnet0:bridge0 host_hostname=freenasplugintest
(Yes, it has a horrible name, sorry)

This will make a DHCP based jail for you, including SabNZBD 2.3.4.
You may then need to stop the jail, edit it and change the mount points for the jail.

I do not know why it won't download 2.3.5, there's a way around this but I can't recall what it is.

I just tested this on a fresh 11.2 RC1 machine and it worked.

 

[diskdiddler]

For other experienced people here, I've attempted following these instructions (yeah, I know it would make a MESS)
https://forums.freenas.org/index.ph...-lets-encrypt-iocage.60375/page-2#post-477773

None the less, despite editing FreeBSD.CONF to attempt to force an update to 2.3.5, it's not working.
My plugin is shabby but functional, luckily when 11.2 ships and they make a proper sabnzbd plugin, you WILL be able to run 2 plugins at the same time and simply copy and paste the settings across from one tab to the other, to use a more official plugin.

 

[jeffdb27]

You have your permissions clobbered by doing this on a Windows share. Search the forum for "operation not permitted" or "setfacl -b".

Wow! Thanks so much. I had no idea that would clobber permissions. Never had any trouble prior to 11.2 but I'm so glad you helped me sort this out!