Introducing the new TrueNAS CLI

[morganL]

So are there any plans on making this work from a remote management host using the Restful API? Something like the linode-cli tool lets your do alot of management remotely.

I'm mostly looking to provision NFS datasets and shares and then make changes more easily. I'm running TrueNAS Professional 13.x on an M40 HA pair, so anything to make automating my upcoming migration would be really helpful!

Obviuosly you can do it remotely via ssh into TrueNAS and then use CLI.... can you be more specific about what you can't do?

 

[HoneyBadger]

So are there any plans on making this work from a remote management host using the Restful API? Something like the linode-cli tool lets your do alot of management remotely.

I'm mostly looking to provision NFS datasets and shares and then make changes more easily. I'm running TrueNAS Professional 13.x on an M40 HA pair, so anything to make automating my upcoming migration would be really helpful!

There is an existing RESTful API - you can access the documentation by appending /api/docs to your TrueNAS URL. Is that perhaps what you're looking for?

 

[Jfs]

I guess I'm looking to be able to script it from the linux box I used to manage my systems, and run a loop over it. Maybe like:

vols="test1 test2 prod2 oracle3 oracle4 oraclelogs1"
for vol in vols; do
truenas-cli pool dataset new -pool tank -path "tank/foo/${vol}"
truenas-cli nfs shares add -path "/mnt/tank/foo/${vol}" -ro true -mapuser "root" -network "192.169.2.0/24" -network "192.168.3.0/24"
truenas-cli nfs shares add -path "/mnt/tank/foo/${vol} -mapuser "root" -hosts "192.168.2.40" -comment "management host is RW"
done

And of course I'd have already setup an environment variable like "TRUENAS_API_KEY=<long string>" so that I dno't need to enter login info each time.

Right now to create six volumes, with some basic NFS shares, it's a damn lot of pointing and clicking on the web interface and trying to make sure I remember to hit all the correct options. I can do all this with the REST API, but it's painful doing this with 'curl' so I've been writing my own python scripts using the requests library to automate this stuff. But my python-fu is weak (I tried to use the force, I did!!! so it's slow going.

I feel that TrueNAS is going to get more commercial uptake if there are tools which allows simpler automation, and decent examples in the docs on how to make changes without having to use the web gui all the time.

 

[morganL]

I guess I'm looking to be able to script it from the linux box I used to manage my systems, and run a loop over it. Maybe like:

vols="test1 test2 prod2 oracle3 oracle4 oraclelogs1"
for vol in vols; do
truenas-cli pool dataset new -pool tank -path "tank/foo/${vol}"
truenas-cli nfs shares add -path "/mnt/tank/foo/${vol}" -ro true -mapuser "root" -network "192.169.2.0/24" -network "192.168.3.0/24"
truenas-cli nfs shares add -path "/mnt/tank/foo/${vol} -mapuser "root" -hosts "192.168.2.40" -comment "management host is RW"
done

And of course I'd have already setup an environment variable like "TRUENAS_API_KEY=<long string>" so that I dno't need to enter login info each time.

Right now to create six volumes, with some basic NFS shares, it's a damn lot of pointing and clicking on the web interface and trying to make sure I remember to hit all the correct options. I can do all this with the REST API, but it's painful doing this with 'curl' so I've been writing my own python scripts using the requests library to automate this stuff. But my python-fu is weak (I tried to use the force, I did!!! so it's slow going.

I feel that TrueNAS is going to get more commercial uptake if there are tools which allows simpler automation, and decent examples in the docs on how to make changes without having to use the web gui all the time.

So the needs are:
Remote scripting (on a linux management host)
-multiple NAS systems?
Not Curl.. so prefer CLI?? (what do others think?)
Better docs (we all agree)

The primary challenge is that the CLI does change with software revs. If you manage multiple systems, the CLI options will be different. Are there good solutions for this problem?

 

[Jfs]

So how badly does the API change and is the API versioned? I guess you need to add in something you can query to find out the capabilities of the API, so that people can handle failures gracefully. I've been poking around at various CLI generation tools but not making much progress.

And my python experiments are sorta working, but not perfectly. All the exception handling is a pain.

And if I manage multiple systems, maybe the CLI just needs to support versioned libraries so it knows which one to use? But since this is also (in my mind) just V13 and newer, a little bit of judicious additions to the API and a willingness to bump the version number when things change should be ok.

So if you goto htpp://<host>/api/v2.0/api/latest

I'd expect it to return something like '{ "version": "v2.9" }' or something like that, so that if the CLI supports v2.9, it can just switch to using that version of the API, otherwise it just supports v2.0. But maybe that's too much of an ask?

 

[Arwen]

So the needs are:
Remote scripting (on a linux management host)
-multiple NAS systems?
Not Curl.. so prefer CLI?? (what do others think?)
Better docs (we all agree)

The primary challenge is that the CLI does change with software revs. If you manage multiple systems, the CLI options will be different. Are there good solutions for this problem?

When my team managed a NetApp back around 2000, we, the Unix team used scripting to perform many tasks. In fact, adding multiple MS-Windows users was sent to us, the Unix team, because we could take a list and in less than a few minutes be done.

Performing the work through NetApp's GUI would have taken hours. And as @Jfs rightly pointed out, prone to introduce problems / mistakes.

So yes, a CLI with many of the functions that the GUI has would be helpful.


That said, rarer items could be de-prioritized for implementation, in favor of the more common and higher usage items. Like adding / removing users.

 

[morganL]

So yes, a CLI with many of the functions that the GUI has would be helpful.

So, there is a CLI with SCALE.. have you used it and is it acceptable?

The question is about how to script it?
Should the scripting be local or remote?
Local includes using ssh remotely into a system.

To be remote.. requires a smart CLI client.. I think
or the remote users scripts with the REST API.

What did netapp do?

 

[Arwen]

I have not needed the scripting ability with TrueNAS yet. It is just a home server for me.

In the case of NetApp, we did all the work by remote logging into the NetApp. Then performing local to the NetApp commands for the various work. That seemed fine as long as we could install our SSH keys and use a SSH agent for passwordless logins. Thus, a remote script could login to the NetApp, run a command, take the results and do what it needed. No REST API or Curl needed. Though I guess Expect could help as needed.

To be fair, I don't know what NetApp does today. Even though the company I work for, (for the last 4 years), has NetApps, they are managed by a NAS group. I am pigeon holed into the Unix SysAdmin group.

 

[ChrisRJ]

Finally a topic where I can apply the knowledge from my day job ;-)

I am surprised that the topic of API changes was thrown in so leisurely. In an enterprise context API stability and proper versioning are of utmost criticality. Simply because the API provider can never force (esp. if you think about a B2B scenario with 10k+ trading partners) the API consumers to switch to a new version at a certain point in time.

So breaking changes must be handled with extra care. It should also be added that too many developers, probably because they never operated their own product in a mission-critical context, are not aware of these problems. Therefore, the majority of breaking changes I have seen were actually not necessary. While the developer saw that the new incompatible version was better/more elegant, this was pretty much an irrelevant aspect for the users. All they saw is a major effort to deal with the change. The actual code change is never the problem here. But things need to get deployed and this means downtime, and synchronization with other teams.

In other words: A really strong mechanism should be implemented to avoid unnecessary breaking changes. Typically developers are not allowed to make such decisions on their own or within only their team. Instead a formal sign-off should be implemented. And test automation is a good way to ensure this rule is actually followed.

As to having the CLI available locally or from a management console, I think separation of concerns comes in. Having the remote capability separated from the CLI part, reduces complexity and ensures a cleaner architecture. It also does not tie things to a particular way of remote access. The latter can be become a real problem if a new access method must be supported. This separation also means that customers can implement their own methods, if they have the need.

There is of course more, but let me stop here for now and see what others think.

 

[WI_Hedgehog]

That said, rarer items could be de-prioritized for implementation, in favor of the more common and higher usage items. Like adding / removing users.

Given the forum's new-user topics, right now the most needed command is:

run minecraft-server

The second is:

autofix system --whole-thing

 

[morganL]

As to having the CLI available locally or from a management console, I think separation of concerns comes in. Having the remote capability separated from the CLI part, reduces complexity and ensures a cleaner architecture. It also does not tie things to a particular way of remote access. The latter can be become a real problem if a new access method must be supported. This separation also means that customers can implement their own methods, if they have the need.

Agreed on the API stability and versioning.

Do you have a good example of "remote CLI" (with scripting) vs "remote API"?

 

[ChrisRJ]

Do you have a good example of "remote CLI" (with scripting) vs "remote API"?

I would basically implement this as a script or binary that is part of the local TrueNAS installation. So something that can be invoked locally. The basic mechanism would be the use of SSH. This should also make it possible without much effort to incorporate it into systems like Ansible, Chef, Puppet, etc.

Another aspect to think about is how to integrate it with the REST API in such a way that both are compatible and doubling the work should be avoided of course. In the end I would probably envision the CLI as a local wrapper around the REST API. But this needs a bit more thinking than just the 5-10 minutes I spent so far.

My overall suggestion would probably be to gather requirements from users as well as architectural aspects. Decisions should be documented in publicly availabe ADRs (see here).

 

[morganL]

I would basically implement this as a script or binary that is part of the local TrueNAS installation. So something that can be invoked locally. The basic mechanism would be the use of SSH. This should also make it possible without much effort to incorporate it into systems like Ansible, Chef, Puppet, etc.

Another aspect to think about is how to integrate it with the REST API in such a way that both are compatible and doubling the work should be avoided of course. In the end I would probably envision the CLI as a local wrapper around the REST API. But this needs a bit more thinking than just the 5-10 minutes I spent so far.

My overall suggestion would probably be to gather requirements from users as well as architectural aspects. Decisions should be documented in publicly availabe ADRs (see here).

SSH is done.. as is REST API.

The challenge withe the local wrapper around the REST API is that there are potentially multiple NAS systems running different versions. I'd like to know about good solutions to that challenge.

 

[Arwen]

Potentially you could have a "version" command in the REST API that the CLI can invoke. Thus, scripts can pre-check the CLI and determine if they are able to run. Any change to the parameters, return data or actual actions would cause a "version" bump.

The script could for example check "version" for between 2 & 4, and if not, exit gracefully with printed error. The SysAdmin can then check their NAS, and find out it now has "version" 5. A review of the documentation could then be made of all the CLI calls to see if any difference exist, and if they need to handled different from versions 2, 3 & 4. And if so, that specific CLI call can be excepted out.


Another way to handle this that I have seen in real world, is backward and forward compatibility functions. For example, a "version" is passed to the library of functions, in addition to any other parameters needed. This does require an ever increasing library size as it would have to support all versions going back to when it was created. But, this also means zero changes to user scripts. They wrote it to version 1, and if the library of functions is now on version 9, the script still gets to run as if it is on version 1.

One additional way, is to have a deprecated function, that still works, but is not recommended for future scripting. So, you have "adduser X Y Z" command. But later you want to add a 4th parameter, so you then have "adduser2 X Y Z W". A pre-parser could be made that checks for "adduser" without the "2", and recommends changes.


What would be best, I don't know off hand, (I am available for bribery . But, having the ability to script some things, even through TrueCommand, might reduce mistakes and reduce SysAdmin time. (Unix SysAdmins love to script out their own jobs .


As I said before, less common items could / should be de-prioritized. While it might be nice to make a text file with how you want your pool created, all the pool's attributes setup, and then all the datasets, zVols & their attributes, this function is rarer. Not generally day to day, weekly or even monthly. But, a reverse of that "pool create" could be useful, displaying all pool & dataset / zVol configurations, devices, sizes and such for disaster recovery.

 

[morganL]

@Arwen Happy to bribe you or anyone else for a great scripting solution....

 

[Jfs]

So I've falled off the face of the earth, been upto my ears in other work, but now I'm back to my scripting work and needing to get a better solution. Or just even better docs and example of the TrueNAS Enterprise side on how to script out various tasks.

So for me, I really like just having a python script using the responses library to write simple scripts to either generate reports (say one showing all my datasets and how much space and compression their using) or simple scripts to let me generate datasets and apply NFS shares to them.

Or my current problem, creating a CIFS dataset, then applying a group membership to it properly. The API v2 for shares/smb doesn't seem to be what I'm using. More accurately, if I go and edit the permissions on a dataset, I cant' figure out how to view , much less edit, those settings using a simple curl command.

But back to the CLI discussion in general. I think what's needed here is also finer grained access controls. I would love to be able to generate an API key which only allows 'GET' commands, so that people can pull info from the system, but not make changes. And having another API key which can only create/modify datasets, but not delete them would be good too. Basically, some way to limit the access of various API Keys to just a subset of the Restful API would be awesome.

And using python and maybe the 'click' cli library would be a way to make this happen. And with 'tabulate' library as well to make nice reports. I've been happy with the linode-cli tool to create and manage objects in the Linode clould, so something like that would be useful. But also a way to give out subsets of access to just parts of the API space would be also key.

 

[Arwen]

Hmm, some attributes for different users:
- R/O or R/W, or Full Admin
- User controls only, (for User Admins)
- Snapshot & Replication, for Backup Admins
- Share control for SMB and or NFS

Plus, the ability to generate a text output that will do specific tasks, like:
- Network(s) configuration
- Pool(s) configuration
- Dataset(s) configuration

This would allow an automated extract of existing configuration, so that it can be either:
- Duplicated for another TrueNAS
- Saved for recovery
- Edited for another configuration implementation. Meaning you have pool "tank1", but you now want pool "tank2" with the same attributes, but of course different devices.


All that on top of a version to allow for improvements over time.

 

[blanchet]

I have not needed the scripting ability with TrueNAS yet. It is just a home server for me.

In the case of NetApp, we did all the work by remote logging into the NetApp. Then performing local to the NetApp commands for the various work. That seemed fine as long as we could install our SSH keys and use a SSH agent for passwordless logins. Thus, a remote script could login to the NetApp, run a command, take the results and do what it needed. No REST API or Curl needed. Though I guess Expect could help as needed.

To be fair, I don't know what NetApp does today. Even though the company I work for, (for the last 4 years), has NetApps, they are managed by a NAS group. I am pigeon holed into the Unix SysAdmin group.

ONTAP 9.11 (Netapp's Operaging System) has

• a CLI (access from SSH or the serial console)
• a web interface called SystemManager
• a REST API (never used yet)
• Ansible modules (never used yet)

Since ONTAP 9.8, the web interface has been dramatically simplified to be used by less skilled staff.
But the consequence is that you can do only the basic things with the web interface, and now many settings are available only from the CLI.

Ontap CLI is very verbose but

• the other vendors have also very verbose CLI
• the long option names are easy to understand
• it supports abbreviation
• the auto completion works well
• it is easy to find what you want with Google

Finally, in my group, we use

• Ontap webui only for visualization
• Ontap CLI for everything else because it is faster, especially to document the modification you have done.

Therefore it would be nice if we could manage TrueNAS Core from the CLI.

For the moment, we can partially manage TrueNAS Core from the CLI

• storage with the zfs command, typically to set quota or to manage snapshots
• jails with the iocage command

 

[Jfs]

Arwen,
You've got the idea. But even as a first step, just putting out some better examples of real world uses of the RestFul API would be a huge help. As usual, 10% of the code does the work and the other 90% does the error and exception handling. And that's never the fun part of the job. I've got some initial code for creating volumes, but stepping back and coming up with a useful base set of arguements or overall design of the CLI takes precedence.

For example, I've got a volume which has a group ownership, but for the life of me I haven't been able to figure out how that GID number translates to the AD group. So it's obviously got to be multiple calls to pull back the info, but from where? This is in regards to editing permissions of a dataset, if that helps.

I'm doing a curl POST call to filesystem/getacl, and I get back a result, but now how do I turn a GID into a name? Ah... now I found it. And not easy to figure out:

curl -s -X POST -H "Authorization: Bearer $KEY" "http://sanjfs/api/v2.0/group/get_group_obj" -d '{"gid":"100000513"}' | less
{
"gr_name": "MY_DOMAIN\\domain admins",
"gr_gid": 100000513,
"gr_mem": []
}

So now I try to actually map BACK from "MY_DOMAIN\\domain admins" in the future. Note the doubled back slashes! This is something to keep in mind down the line. So now I can write the python code to list all my datasets, get their ACLs and show the name of the group which has ownership. And then I can now create multiple volumes with the right permissions without having to know the GUI tricks.

 

[morganL]

Hmm, some attributes for different users:
- R/O or R/W, or Full Admin
- User controls only, (for User Admins)
- Snapshot & Replication, for Backup Admins
- Share control for SMB and or NFS

Plus, the ability to generate a text output that will do specific tasks, like:
- Network(s) configuration
- Pool(s) configuration
- Dataset(s) configuration

This would allow an automated extract of existing configuration, so that it can be either:
- Duplicated for another TrueNAS
- Saved for recovery
- Edited for another configuration implementation. Meaning you have pool "tank1", but you now want pool "tank2" with the same attributes, but of course different devices.


All that on top of a version to allow for improvements over time.

This is close to the goal for SCALE

There are API ACLs in the software now.... we just don't have easy ways to use them.

I'd also like automated extract of config to "cli commands".. because I'm an old Cisco engineer and that's what we did.
However, need some volunteers to help with this.