For those of you coming from other systems e.g. Unraid, proxmox, how are you finding TrueNAS Scale?

[marshalleq]

For me, it's a marked improvement on almost everything with the exception of apps.

Being used to having a direct sync up to docker hub, having Kubernetes in the middle does create some confusion and maybe even some concern. Particularly when you have a relatively complex networking setup - reverse proxy's etc. TrueNAS has done a marvellous job so far though.

How have you been finding it and how have you approached things differently to get them workable in TrueNAS Scale?

Thanks,

Marshalleq.

 

[Ixian]

How apps work is, guaranteed, going to confuse the hell out of a lot of people come release. The standard line is "Kubernetes is what most(?) businesses use, Ix is focused on the enterprise, e.g. it was the right choice". Yet that isn't universally true, and even if it was they've implimented Kubernetes so that you are meant to use it their way - through the UI - with no guarantees that any command-line tools will work, or keep working. Even kubectl, the basic cli tool for Kubernetes, needs to be aliased i.e. it doesn't appear available at first to users. I found that out digging around the forum.

Anyone schooled in K8s is going to be confused by this approach, since it's not how you work with it in many other environments. And if you aren't schooled in it, even the simplified UX - and I appreciate the work that's gone in to abstracting this, I know it isn't easy - is confusing. I'm puzzled by who this is aimed at.

Business that use K8s for container orchestration, or plan to, will want to do so in a standardized, cross platform way. End users who want a nice ZFS based NAS that runs easy to deploy docker apps will find this to be massively over-complicated. And the official stance of "well, you can use the cli tools, but no guarentees it won't break anything" or "you can leverage docker-compose, but no guarentees it won't break anything (in fact, in order to use docker - outside of the docker container support they made part of the K8s deployment - you have to modify the daemon.json file to get iptables support and host networking, and that file gets over-written on updates) or "if you really want docker just run it in a VM" - something that people already do today with TrueNAS Core and find lacking - none of this is particularly comforting.

I also appreciate what the Truecharts folks are doing with their curated apps, but that also introduces problems. Official docker-compose support - or at least semi-official - would have been a lot simpler for those that don't use K8s and there are a million support resources for it.

 

[Patrick M. Hausen]

@Ixian Seconded. I am still searching for a use case, honestly. As far as apps are concerned I am running all I need on CORE in a (to me!) much more transparent and maintainable fashion.

The glusterfs integration looks promising and FreeBSD is behind Linux as a server, here. But then you cannot use the sharing services on SCALE to serve from the cluster volume. And of course you would need some sort of CARP or other means of failover for e.g. smbd, too. For any of this to make sense.

Pretty underwhelmed at the moment.

 

[Ixian]

I'm still hoping they stick to the stance of "we're not here to help you troubleshoot why your docker-compose file won't work/etc. but we'll at least allow you to enable it/disable K8s without disruption" which is basically what I've read in other threads but haven't seen implemented yet.

For example. there's no option to enable/disable K8s in the service panel (the easiest way is to un-set the pool in the Apps gui, which appears to stop everything). That's minor. The bigger problem I've seen is the docker daemon.json file needs to be modified for networking to work and any mods to it appears to be over-written on update, at least in the nightlies. It's nightly, so may well change, my point being there isn't a lot of clarity around this.

 

[inman.turbo]

As far as Apps are concerned, I am in agreement with the rest of you. The UX is confusing, and I am not certain what the goals are for it. I would much prefer a more technical and standards based approach something like rancher, where you could if if all else fails download your ~/.kube/config file and accomplish what you need to with kubectl (although kubectl namespaced through k3s is pretty standard, especially on prem, in my experience).

TBH though we have used and even dismissed rancher on many projects in favor of a more standard, text based IaC approach. Check your deployments into a repo, even repos for one-off and maintenance and migration commands etc, run all thorough github workflows, or pipeline of your choice. Maybe something like that is possible with SCALE and I am just missing it, there is a way to add catalogs it looks like, what about automated deployments through webhooks, etc? Seems to me if you are going to be introducing the complexity that comes with Kubernetes, you'd better damn well be taking advantage of the entire ecosystem and all that it has to offer.

Business that use K8s for container orchestration, or plan to, will want to do so in a standardized, cross platform way.

Yes exactly. When you get to the point of needing K8s, you are at a level that leans heavily on standards. And if you can't "Deploy anywhere" you're missing the point.

 

[Ixian]

This has been in the devnotes for some time:

SCALE allows Kubernetes to be disabled. The user will then have access to the native container services within Debian. This will include Docker, LXC (Q1 2021) or any other Kubernetes distribution. There will be a Container Storage Interface (CSI) that can couple the container services with the SCALE storage capabilities. Users can script these capabilities and then use 3rd-party tools like Portainer to manage them. This approach can be used in SCALE 20.10 and later.

However I haven't yet seen that to be the case, in 21.06 or the nightlies (21.08), without doing some CLI workarounds. I don't know if the strategy has changed or just priorities.

With the latest 21.08 nightlies you can technically disable the K3s service (by un-setting the pool) and use Docker/Docker-Compose from the CLI, though as mentioned you need to edit /etc/docker/daemon.json, remove the restriction on iptables and host networking, and restart the docker service before you can really use it. I've done this on my test box and gotten Portainer running, then a couple test compose files I have. It works as expected, but at least in the case of the nightlies when you update the daemon.json is reset and you have to re-edit it.

Docker and Docker-Compose living as second-class, might-break workarounds isn't great so hoping this improves.

 

[Patrick M. Hausen]

However I haven't yet seen that to be the case, in 21.06 or the nightlies (21.08), without doing some CLI workarounds. I don't know if the strategy has changed or just priorities.

Your quote explicitly says "20.10 and later". 20.10 > 20.8.

 

[stavros-k]

20.10 > 20.8.

Yes, but also 20.10 < 21.06 ;)

 

[inman.turbo]

Your quote explicitly says "20.10 and later". 20.10 > 20.8.

But 20.10 < 21.08

 

[Patrick M. Hausen]

Yes, but also 20.10 < 21.06 ;)

Oops

 

[Basil Hendroff]

SCALE 21.06 Setting up single container apps using the GUI isn't too uncomfortable. It doesn't appear to be possible to use the GUI to set up apps that require container sets. I thought I could use Portainer to circumvent the issue, but various forum threads suggest this doesn't appear to be recommended or supported. Personally, I prefer a supported method to run Docker and Docker-Compose over the enterprise strategy being adopted. As it stands, it feels like using a sledgehammer to kill an ant. Overall opinion: A big thumbs down in the area of container support.

Commodus Gives a Thumbs Down

From the movie Gladiator http://www.imdb.com/title/tt0172495/Produced By:DreamWorks SKG Universal Pictures Scott Free ProductionsMill FilmC & LDawlizRed Wago...

www.youtube.com

 

[Ixian]

SCALE 21.06 Setting up single container apps using the GUI isn't too uncomfortable. It doesn't appear to be possible to use the GUI to set up apps that require container sets. I thought I could use Portainer to circumvent the issue, but various forum threads suggest this doesn't appear to be recommended or supported. Personally, I prefer a supported method to run Docker and Docker-Compose over the enterprise strategy being adopted. As it stands, it feels like using a sledgehammer to kill an ant. Overall opinion: A big thumbs down in the area of container support.

I dislike the "Launch Docker Interface" wizard, personally - too "clicky" (lots of mouse movements required), some options aren't clear regarding what they do, and port mappings are a problem - you can't map any external ports below 9000. It's going to confuse a lot of folks.

 

[stavros-k]

I dislike the "Launch Docker Interface" wizard, personally - too "clicky" (lots of mouse movements required), some options aren't clear regarding what they do, and port mappings are a problem - you can't map any external ports below 9000. It's going to confuse a lot of folks.

I can agree on the "too clicky - scrolling" thing. For the apps we should use the full screen and have the ability to spread out things. Also with that, we could write comments to explain what everything does exactly and fit more info.
But the current GUI is very tight and limiting.

 

[anodos]

@Ixian Seconded. I am still searching for a use case, honestly. As far as apps are concerned I am running all I need on CORE in a (to me!) much more transparent and maintainable fashion.

The glusterfs integration looks promising and FreeBSD is behind Linux as a server, here. But then you cannot use the sharing services on SCALE to serve from the cluster volume. Anf of course you would need some sort of CARP or other means of failover for e.g. smbd, too. For any of this to make sense.

Clustering for smbd was merged into master fairly recently. It's fully active-active. ctdb handles IP failover. Most aspects of SMB configuration have clustered backend now once gluster is configured. For example for API users, sharing.smb.create on any node will create a share that is visible in sharing.smb.query on all cluster nodes. AD and LDAP should work in clustered fashion (join one node and all nodes are automatically joined to AD). Local users and groups are going to require some work. Currently you'll have to manually create same users with same ids, and groups + ids on all cluster nodes to get reliable behavior.

Due to the way that clustered SMB works (converting samba state files to clustered variants) mixing gluster / non-gluster SMB shares on the same server is not supported. Once you're using glusterfs, SMB shares are clustered.

 

[Patrick M. Hausen]

Once you're using glusterfs, SMB shares are clustered.

Great!

Just out of curiosity and for test/development purposes: is Samba as a DC en par with Windows as far as TrueNAS SCALE/CORE as a domain member is concerned? Can I set up Samba in a jail or VM as a DC and expect to be able to "have a working domain"?

 

[anodos]

Great!

Just out of curiosity and for test/development purposes: is Samba as a DC en par with Windows as far as TrueNAS SCALE/CORE as a domain member is concerned? Can I set up Samba in a jail or VM as a DC and expect to be able to "have a working domain"?

FreeBSD AD DC role is something I'd consider somewhat less well supported. It's been a while since I worked on it, but from what I understand Timur (the upstream FreeBSD port maintainer) has been making sure that it keeps working.

You do need to be somewhat careful about creating circular dependencies in your overall environment design. Joining a host to a domain in a FreeBSD jail or VM that's a guest on the same host is never a great idea.

For most situations the Samba AD domain is sufficient. The upstream samba wiki contains detailed information about how to set up / configure a samba AD DC. My only practical guidance is to try to ensure you're using a currently supported Samba version (4.13 or 4.14). You also should make sure that your sysvol share is on ZFS and uses vfs_zfsacl.

 

[Patrick M. Hausen]

I would probably create a Linux VM on my CORE system and try to join my SCALE system to the domain. Thanks for the detailed info.

 

[fr0stbyt3]

Coming from Unraid and with no Kubernetes experience, I find myself needing to google some of the new concepts that Kubernetes introduces. Only to find out that that not exactly how things work in SCALE.

Ideally, I just want to run a few apps across all of my servers and automatically failover if one server is down. I think this would be nice for most users that host their own plex for instance. Reality seems quite a bit different.

That being said, I do like what I have seen so far. With a few good youtube videos and expanded documentation this could be an amazing upgrade from what i'm used to with unraid and docker today. But as a new user just kicking the tires, i'm a little lost.

 

[SnoppyFloppy]

I second most of what have been said here. I come from Openmediavault and Proxmox with TN Core VM for NAS capabilities and Ubuntu VM for docker-dompose deployments.

Overall I think that the concept of having a linux-based TrueNAS is great because of the better VM capabilities and container support and because I'm more familiar with linux. And idealy with TN scale, I wouldn't deen any VM as everything I need exist as docker images. But as several others have mentioned I also don't like the current implementation of Apps in TN Scale.

My main gripes are:
1) I came onboard to TN Scale at 21.06 and so far I haven't had much luck with the Truecharts apps. Traefik launched once but then I needed to change some configuration and now it won't deploy - even with everything at default. Truecharts Nextcloud deploys but the UI hangs as soon as I go on to add apps
2) I'm fairly familiar with docker-compose but I find the Kubernetes learning curve very steep and therefore I'm in the mercy of truecharts / TN. If they have the apps that I want everything is probably fine - if the apps deploys and woth, that is.
3) If the app I'm looking fore isn't in the truecharts or standard repository, my only other option is the "Launch Docker Image" GUI. This is probably fine for simple stuff but if I want to deploy something more complicated where several containers (e.g an app and a DB) need to communicate I cant get it to work.
4) To mitigate #3, I can spin up a VM for docker-compose deployments but then (at least the last time I tried) I can't reverse proxy these containers with the truecharts Traefik and therefore I will have to deploy a seckond reverse proxies which is plain dumb

I would much prefer if TN focused on providing whatever middlewares are needed for HTTPS, storage, etc. and provided the users the options (via GUI) to install tried and tested container GUI's like portainer, rancher or just use CLI (docker-compose, kube-ctl) without having to do alot of hacking and without the certainty that it will be supported long-term.

 

[Basil Hendroff]

@fr0stbyt3 @SnoppyFloppy you may be interested in following the discussions in these threads...

Requesting docker-compose example
VHS or BETA?