For those of you coming from other systems e.g. Unraid, proxmox, how are you finding TrueNAS Scale?

[Ixian]

@fr0stbyt3 @SnoppyFloppy you may be interested in following the discussions in these threads...

Requesting docker-compose example
VHS or BETA?

Both of those threads are just users talking past one another at this point. Run compose and hope over time Ix Systems doesn't break it and/or the app experience improves for smaller home NAS users, or find something else to use, is pretty much what it all boils down to.

Ideally, I just want to run a few apps across all of my servers and automatically fail-over if one server is down. I think this would be nice for most users that host their own plex for instance. Reality seems quite a bit different.

The apps also have to support that, and Plex (and most "media server" apps) don't. If you're envisioning a senario where, for example you have a Plex instance that goes down and it switches right over to one running on another SCALE box, that won't work. There are applications that would work with but Plex and the like aren't among them.

 

[fr0stbyt3]

Both of those threads are just users talking past one another at this point. Run compose and hope over time Ix Systems doesn't break it and/or the app experience improves for smaller home NAS users, or find something else to use, is pretty much what it all boils down to.



The apps also have to support that, and Plex (and most "media server" apps) don't. If you're envisioning a senario where, for example you have a Plex instance that goes down and it switches right over to one running on another SCALE box, that won't work. There are applications that would work with but Plex and the like aren't among them.

Awesome. Care to explain how/why that doesn't work? I don't mind if connections get dropped and sessions get reset. If both instances have access to the same database and the files are kept in the same storage location, it should just... work. Right?

 

[ornias]

The apps also have to support that, and Plex (and most "media server" apps) don't. If you're envisioning a senario where, for example you have a Plex instance that goes down and it switches right over to one running on another SCALE box, that won't work. There are applications that would work with but Plex and the like aren't among them.

This is not True. Plex can do this just fine, as long as the storage is clustered.
If a K8S node fails it would just start the container on another node.

Simply put:
There is "High Availability" (zero downtime on node/container failure) and "resillience" (low downtime automatic restart on node/container failure), Plex cannot be deployed high available (failure without downtime) because indeed the Application needs to support it, but it can be run resillient

 

[Ixian]

You can have two (or more) Plex servers share the same media backend, a number of folks do that, and if you copy the db from the "source" server to the second one then they will look the same and you can set them to both update media/etc. in the same way so they stay in "sync". It's two separate instances & dbs though.

What you can't do with Plex is share the same backend db (Plex uses sqlite, and while there is a branch of that called Dqlite that has HA capabilities Plex doesn't use it) and there's no way I know of to have Plex use a different db backend with HA. So you'll have two different servers that your client would need to switch between, however they could at least use the same media sources.

Of course, you can do this with almost any setup. None of the clustering/HA stuff SCALE aspires to provide will matter for this use case.

 

[ornias]

My main gripes are:
1) I came onboard to TN Scale at 21.06 and so far I haven't had much luck with the Truecharts apps. Traefik launched once but then I needed to change some configuration and now it won't deploy - even with everything at default. Truecharts Nextcloud deploys but the UI hangs as soon as I go on to add apps

We've not recieved any support request about this. How do you think things improve if these kind of failure don't get reported to the people that can fix them? "some configuration" is rather vague, no one is even going to be able to comment on it.

That being said: I suggest no one starts with Traefik and Ingress, just try setting up apps first, get a feel for the system and try to add ingress/traefik later.

 

[ornias]

Of course, you can do this with almost any setup. None of the clustering/HA stuff SCALE aspires to provide will matter for this use case.

What @fr0stbyt3 inquired about wasn't HA, it was just automatic restarts on another node... See:
"Ideally, I just want to run a few apps across all of my servers and automatically fail-over if one server is down. "

 

[fr0stbyt3]

What @fr0stbyt3 inquired about wasn't HA, it was just automatic restarts on another node... See:
"Ideally, I just want to run a few apps across all of my servers and automatically fail-over if one server is down. "

Correct. I would not want two "active" plex servers as this would almost certainly mess up the database. Failover is very much what I'm looking for.

That being said, is there a way to accomplish this now right now with SCALE?

as the storage is clustered

I'm guessing this is similar to longhorn for rancher. But I don't know how to do this in TN.

I'm thinking it might be easier to start with a simple app like sonarr and see if I can get it to failover. I'm willing to learn, I just need pointed in the right direction.

 

[ornias]

Correct. I would not want two "active" plex servers as this would almost certainly mess up the database. Failover is very much what I'm looking for.

That being said, is there a way to accomplish this now right now with SCALE?

Nope, clustered k8s is not part of the first release of TrueNAS SCALE.

I'm guessing this is similar to longhorn for rancher. But I don't know how to do this in TN.

Currently no clustering at all is supported, also: besides using hostPath the current PVC's and iX-Volumes are also unable to utilise clustering. So in short: It's not supported at all.

I'm thinking it might be easier to start with a simple app like sonarr and see if I can get it to failover. I'm willing to learn, I just need pointed in the right direction.

I would suggest starting with something simple indeed. But Sonarr and Plex are both not simple because they actively monitor all sorts of storage. Try starting with something like Jackett, once k8s clustering becomes available in 2022/2023

 

[fr0stbyt3]

Try starting with something like Jackett, once k8s clustering becomes available in 2022/2023

In the meantime, where would you suggest I go to learn some of the underlying concepts? Is it worth attempting to stand up a k8s cluster with vms to learn how its supposed to work?

Also, thanks for answering all the new guys questions.

 

[ornias]

In the meantime, where would you suggest I go to learn some of the underlying concepts? Is it worth attempting to stand up a k8s cluster with vms to learn how its supposed to work?

Also, thanks for answering all the new guys questions.

I think learning new skills like k8s is always worth it! :)
I started learning k8s last year in the summer and that lead me to creating TrueCharts, when you understand it, a lot of underlaying systems get a lot easier to understand.

It also means you can more easily figure out of SCALE eventually is even the right way to go for your usecase, because it doesn't neccisarily has to be ;-)

 

[marshalleq]

Great replies everyone. I started this thread because I wondered if others had had similar challenges /confusion to me or if I was alone and the moron in the room so to speak. I am reinstalling TrueNAS scale today and figured I could learn Kubernetes from it, though reading the most excellent discussion at the beginning of this thread started by @Ixian @Patrick M. Hausen and @inman.turbo which outlines that TrueNAS has modified this quite a bit making me reconsider.

I am extremely excited about the core of TrueNAS e.g. what it brings coming from unraid back to all the things I miss about proper enterprise focussed software. I actually posted on Unraid (last post) about all these things that I miss - that TrueNAS does provide (and with that all the things I like about Unraid - which SCALE could possibly learn a thing or two from also).

I am torn to know what to do with TrueNAS Scale. It clearly handles all those basics extremely well, but loses on it's value add and GUI. I sort of want to tear out everything Kubernetes so that I can do the basics I need to do without a clunky interface and the pain that K8s brings with it and I otherwise must contend with. I could run all my dockers in a VM - or just put unraid in a VM on SCALE, but that is pretty wonky and I'd lose the benefits of ZFS segregation of apps and create nightmare issues with putting CoW file systems on top of each other. Not to mention the potential for runaway read/writes due to misaligned FS clusters / record sizes and such.

I think a marvellous job has been done with Scale so far but if I had a magic wand, I'd ask for a one click option to remove K8s and replace it with docker. Or even just remove it and I add docker myself. And maybe some way to segregate and learn K8S on the side without having to impact the core docker system.

Lastly I find it interesting that the move from BSD to Linux kernel for IX Systems, was likely borne out of the observation that VM's and containerisation worked better on linux. And yet somehow (IMO and based on the above comments) they've messed up the containerisation. I assume this was just a punt that hasn't played out so well. Assuming IX Systems are using Agile development practices, I can only assume their customer feedback sessions are with some customers with a pretty specific use case. Or perhaps they're still doing waterfall deployments - they did have a fairly up front and comprehensive roadmap after all - not something typical of agile.

If anyone from IX Systems comes across this - I'd say there's some great information in this thread you could turn into an opportunity, if you so desired. Well done on the work so far anyway, it's an amazing achievement!

Marshalleq.

 

[headwhacker]

If you want to learn about K8s, SCALE will not be the good place to start. K8s by itself is confusing to begin with especially if you are starting to learn about containers/orchestration. SCALE at least in its current form tries to abstract a lot of the normal user interaction with K8s.

I was initially excited about SCALE because CORE is hands down the best NAS/storage platform I have used so far. The thought of integrating TN with containers,VMs,K8s and managing them in a single pane of glass so to speak I thought is the holy grail. But to be fair, it's still very early for SCALE and can evolve over time.

For now I went with proxmox in a 3-node HA cluster and a separate CORE which provide storage(via NFS, iscsi) to other VMs, containers and K8s nodes. In this setup I can do pretty much everything I can think of. The HA/live-migration in proxmox just works with little effort.

I'm watching Rancher's Harvester as well, it looks good especially if you are familiar with Rancher's K8s ecosystem. However, this is still in preview and has a long way to go. I think I prefer their take on HCI platform as far as K8s integration goes.

 

[marshalleq]

Thanks, yeah I've just installed it to have a quick look again - but I've basically decided not to use TrueNAS Scale at the moment. Full credit for what has been achieved - because getting a stable base is absolutely crucial. But right now TrueNAS is actually in the way of the things I want to do rather than supporting me to achieve them.

I hate that I'm saying that though because that probably means unraid. Or yes, ProxMox. Proxmox no docker though - have to use a VM. I'm not so keen on using NFS / iscsi either because of the connection speed and it also means I have to convert my dev box to a prod box. I may as well just go buy a QNAP or something in that case.

So close yet so far lol.

 

[truecharts]

I am reinstalling TrueNAS scale today and figured I could learn Kubernetes from it, though reading the most excellent discussion at the beginning of this thread started by @Ixian @Patrick M. Hausen and @inman.turbo which outlines that TrueNAS has modified this quite a bit making me reconsider.

Seems like a good place for us to chime in:
One of our missions is to offer close-to-native kubernetes/helm access, with our Apps.
We aim to offer very close to the same amount of configurability as "normal" k8s helm charts from, for example, the k8s-at-home community or Bitnami. But instead of a "values.yaml" file we utilise the GUI.

Even "In the back" every App we release is also released as a normal Helm-Chart, which you can also install using normal helm CLI tools or modify using kubectl.

Our, work in progress, "quick-start" guides might help you get up and running to a place where you can safely experiement with different configuration options. Thats our goal: Making kubernetes accessable for everyone using SCALE Apps!
(https://truecharts.org/manual/Quick-Start Guides/01-Open-Apps/)

Hope this puts your mind at ease!
If you've any complications during the use of our Apps, feel free to reach out to us directly!

 

[Ixian]

Seems like a good place for us to chime in:
One of our missions is to offer close-to-native kubernetes/helm access, with our Apps.
We aim to offer very close to the same amount of configurability as "normal" k8s helm charts from, for example, the k8s-at-home community or Bitnami. But instead of a "values.yaml" file we utilise the GUI.

Even "In the back" every App we release is also released as a normal Helm-Chart, which you can also install using normal helm CLI tools or modify using kubectl.

Our, work in progress, "quick-start" guides might help you get up and running to a place where you can safely experiement with different configuration options. Thats our goal: Making kubernetes accessable for everyone using SCALE Apps!
(https://truecharts.org/manual/Quick-Start Guides/01-Open-Apps/)

Hope this puts your mind at ease!
If you've any complications during the use of our Apps, feel free to reach out to us directly!

You folks are doing good work and I'm sure the Truecharts catalog will be well-appreciated by many but it's still a bespoke, curated solution run by volunteers. Any user looking for a "just works" simple click-and-save app approach will like it (until they need to draw outside the lines, so to speak, with how they want their containers to work) but "standard" it is not.

Ix still doesn't recommend using the cli, for much of anything, and kubectl isn't even available as a command until you change your shell variables to point to it. Professionals expecting to leverage a SCALE server as a "regular" K8 node as part of their cluster (unless, I suppose, all the nodes are SCALE servers) are in for disappointment.

I still can't wrap my head around Ix's approach to containers here. "Kubernetes="enterprise"=that's what Ix cares about, duh" is not an explanation, given what I've outlined.

 

[truecharts]

You folks are doing good work and I'm sure the Truecharts catalog will be well-appreciated by many but it's still a bespoke, curated solution run by volunteers. Any user looking for a "just works" simple click-and-save app approach will like it (until they need to draw outside the lines, so to speak, with how they want their containers to work) but "standard" it is not.

Because we got a lot of community requests to accept any valid App, we've stopped curating apps a few months ago.

We aim to expose any feature exposed by common Helm charts. We're actively looking for people with good input for adding more native kubernetes functionality. If you've any customisation you're missing which we could add, please let us know!

 

[marshalleq]

Hey, thanks for chiming in. Can you elaborate on this comment below?

Because we got a lot of community requests to accept any valid App, we've stopped curating apps a few months ago.

One of the things as a noob to TrueNAS Scale that's a bit frustrating is (in order of levels of pain) you either have to use the docker one off GUI thing, use truecharts, if unavailable request it at truecharts (which I recall was rejected for a few of mine) or finally learn helm charts yourself if you can't get it any other way, which may mean setting up your own repository. The level of pain here, to get a basic tried and true app you rely on can be quite significant when you get down to the last mile of 'create it yourself'. Leading to one example of why, I remember I really didn't appreciate someone in the middle acting like they're my IT department and denying my app request because it didn't meet their requirements for security.

So, I'm wondering if your above comment means one of these three things?
1. No new apps are being added to TrueCharts anymore
2. New apps for TrueCharts are no longer being 'vetted' anymore so anything can be added
3. We're leaving this up to others to decide what to make and giving them access to truecharts

Thanks.

Marshalleq

 

[truecharts]

Hey, thanks for chiming in. Can you elaborate on this comment below?

One of the things as a noob to TrueNAS Scale that's a bit frustrating is (in order of levels of pain) you either have to use the docker one off GUI thing, use truecharts, if unavailable request it at truecharts (which I recall was rejected for a few of mine)

No problem!
Your MariaDB request has since been added to our combined "App Request" list: https://github.com/truecharts/apps/issues/729
It seems we forgot to mention this on your PR, sorry for that!

Requests still need people willing to create the App.
As it's a community effort, we still cannot give any guarantee your App will actually get made :-(

or finally learn helm charts yourself if you can't get it any other way, which may mean setting up your own repository.

For simple Apps (not talking MariaDB in this case), learning Helm isn't needed at all. Mostly just altering the containers and some ports works out fine to get a sample going. We are also always willing to chime in to push you through it :)

The level of pain here, to get a basic tried and true app you rely on can be quite significant when you get down to the last mile of 'create it yourself'. Leading to one example of why, I remember I really didn't appreciate someone in the middle acting like they're my IT department and denying my app request because it didn't meet their requirements for security.

Those policies have been retracted, so we don't curate apps anymore. We also removed the seperate train for deprecated or non-free Apps.

As you can see in the request list, only a few App idea's have been "soft denied" (not likely to be added). Which is mostly things that are already available in TrueNAS itself, like SAMBA.

So, I'm wondering if your above comment means one of these three things?
1. No new apps are being added to TrueCharts anymore

We just added multiple new Apps, from multiple different creators, this week.
So we're pretty certain new Apps are still comming!

2. New apps for TrueCharts are no longer being 'vetted' anymore so anything can be added

True!
Within reason, for example: Mailcow is technically simply not possible.

3. We're leaving this up to others to decide what to make and giving them access to truecharts

We've always wanted others to submit their Apps, but we're not going to give up on them either!


If you've any more suggestions for Apps, feel free to discuss or submit them to us through the usual channels!

 

[marshalleq]

Wow thanks for the update - sounds like TrueCharts has really moved up in the world. I will take another look at it! Thanks again!

Edit - I hav some limited time where I'd be keen to help out with some of these - but I'll need a kickstart in the beginning to get the basics covered (i.e. questions) if that's OK? I do recall there were some good instructions somewhere too.

 

[inman.turbo]

I'm watching Rancher's Harvester as well, it looks good especially if you are familiar with Rancher's K8s ecosystem. However, this is still in preview and has a long way to go. I think I prefer their take on HCI platform as far as K8s integration goes.

It's like an HCI appliance race. Everyone wants to vendor lock the clientele. Meanwhile the big fish just write wrappers for the the standard libraries, (libvirt, qemu, kvm, kubernetes) and push features and bug fixes upstream.

I would much prefer an appliance that used this cross platform approach, but then why bother if your users can bail at anytime and run all their stuff on any linux OS?

TBH it was all fun and games until companies like IBM started buying companies like Red Hat and tabling projects like Centos. OpenSuse bought rancher. Who to trust? We have Red Hat running all over the place in production but our lab infra is triple our Production, and many cycles and systems alike are donated to non profit, educational and open source ... just like the upstream sources that give all of us our lifeblood.

Any any moment these vendors could choose to attempt quick profits over long term mutual gain and stem their flow of information and choke everyone downstream.

I think I'll take my chances on a bunch of Debian or Alpine (which is basically an instant embedded appliance you can run from ram -- install kvm, cronjob LBU and you have the same basic paradigm as ESXI ) boxes provisioned over a simple PXE, running k8s, libvirt, qemu and kvm, postfix and dovecot.

You can buy an HCI frontend wrapper from the vendor of your choice, then switch vendors when you need to.

All this changing all the backend stacks to satisfy the frontend unified API is a bit ridiculous.

Edit:
Harvester is awesome.