For those of you coming from other systems e.g. Unraid, proxmox, how are you finding TrueNAS Scale?

[sjieke]

The discussion is already going on for a while, but I would like to give my experience also :)

I was a Freenas user untill verion 11 I think. I started to use my 'nas' for other things also, in the beginning I used the 'plugins' but for some reason those did not fullfill my needs.
Then the bhyve virtualization was introduced and I stumbled upon 'Rancher'. I began setting up a VM (using bhyve) to get my single-noded rancher (based on k8s) setup. I started to like the whole containerized apps and orchestration, but the overhead of running it in bhyve was to heavey...

So I started looking for something else and found Proxmox. I used proxmox with a lot of LXC containers. Performance was great. Although I missed the ease of maintenance containerized apps brought and also the ease of storage management, I kept my Proxmox setup because it just worked.

But then my disks got full and I needed to extend my pool. A good opportunity to upgrade my hardware. Off course it get's paired with new round of comparing what to use.
I started looking at Freenas again to find out a lot has changed and SCALE had it's first BETA released. It looked like it was just what I needed, Freenas storage management with integrated containerized apps. So I installed it to play with it.

I must admit that my previous experience with Rancher and also playing with k3s helped a lot to understand the concepts. In no time I had my first apps running. I could even look around on how it was deployed by using the 'kubectl' commands. Just for sports I also tried installing apps using the bare metal yml files, which just worked as expected. You don't see it in the Truenas portal, but it is available and useable.

So I decided to start configuring it. I use unifi devices for my network, and since there was a unifi app available I tried that one first. It just worked on the first shot. Then I added traefik for reverse proxy but also needed a dns server. First I was going to install PiHole as a VM, but then just tried the 'launch docker' options which worked like a charm for me. I just added an additional virtual interface with its dedicated ip for my DNS server.

Now I switch to the TrueCharts apps as much as possible, because I like where it is going. Even replaced my PiHole app.

So bottom line, I like the fact that it is based on kubernetes, I also like the way it is going. Especially with the additional great job done by TrueCharts.

 

[Vertigo 7]

Just to add my 2 cents... (never used unraid or proxmox, buuuut)

I've dinked with scale off and on, but I keep going back to core. My biggest problem with what is keeping me away from scale is the inability to put specific containers, or whatever you want to call them, behind specific VLANs, like you can with jails with core. Instead, you have to put the host and containers on the same VLAN. That just doesn't sit well with me as I would then have to do a major overhaul of firewall rules, or, not use containers at all and just spin up VMs to replace all of the functionality of jails from core, which is also unappealing due to the increased maintenance and resource requirements. Not to mention, some jails I have in core are on isolated VLANs and I wouldn't be able to do that with scale, which I do not want those devices in those VLANs to be able to access other subnets.

That's a real shame to me because there are other features of scale that I really do want to take advantage of.

 

[truecharts]

Now I switch to the TrueCharts apps as much as possible, because I like where it is going. Even replaced my PiHole app.

So bottom line, I like the fact that it is based on kubernetes, I also like the way it is going. Especially with the additional great job done by TrueCharts.

Thanks for your support, it's appreciated! ^^

 

[marshalleq]

So I have Tdarr running on Unraid and a while back I gave TrueNAS a fourth attempt at getting it to work. What happened is that while there was tdarr available in truecharts, the version I used to match up with my docker in unraid wasn't available - going straight back to the problems listed previously in this thread. OK, so I thought should I just make it with docker again in TrueNAS. But then I realised the long winded GUI I'd have to run through - this is a real weakness of TrueNAS SCALE - the GUI. I am tired of having to spend so much time doing these things as it is, without having to make it longer because of a poorly designed GUI.

So I just thought back old school and decided to run up a linux VM so that I can move it around. So I did this and came across even more problems. Such as TrueNAS VM's only support ZVOL's rather than images inside datasets. Unraid supports the opposite. OK I thought, I can handle that with qemu image convert tools, but it was hard work to convert between the two. Ultimately even though I had an existing starter VM, it ended up being easier to use the very slow GUI in TrueNAS to set up the VM. Which was very frustrating. I don't know what was going wrong, or if I will actually be able to convert this back to an image later. I can certainly ZFS send it at least.

Another frustration in the above, was not being able to create sparse ZVOL's properly. There were two issues, one was you couldn't create a sparse ZVOL in the VM creation (you had to do it in the storage area) and second was you couldn't create a sparse ZVOL larger than the free space, which is really really dumb IMO.

Another issue was performance, even doing CPU passthrough, the performance of the VM was really really bad. I was surprised at just how much slower encoding was, with nothing else at all running on TrueNAS. So much so, I assume it must be a bug in TrueNAS. So ultimately I removed TrueNAS, put on a linux distro on bare metal and ran docker on that. I should have been able to copy my VM from TRUENAS, onto the disk that I used for bare metal linux, but that didn't work either (and I've successfully done that elsewhere a few times - converting between RAW images and such) - for some reason I just couldn't get it to work on TRUENAS.

Maybe I'm the edge case. But I'm not a dummy either. And I think these things are basic things. I've had 4 or 5 decent attempts at giving TrueNAS a serious shot at replacing unraid, but I can't get past the first item.

Trying to be positive here because I know a lot of great work has gone on to date, the things I'd like to see changed before I have another attempt would be:

Change the slow and painful multiple clicks required to do anything with docker, Kubernetes or VM's due to it's lists and sections style
Have a hard think about what in those lists is mandatory or not to make the process quicker
Change VM's so they don't have to be in a ZVOL and can be in an img file of various sorts
Fix sparse ZVOLS
Enable in the GUI the mounting of shares for TrueNAS to be a client
As a bonus investigate VM performance

Marshalleq.

 

[morganL]

Change the slow and painful multiple clicks required to do anything with docker, Kubernetes or VM's due to it's lists and sections style
Have a hard think about what in those lists is mandatory or not to make the process quicker
Change VM's so they don't have to be in a ZVOL and can be in an img file of various sorts
Fix sparse ZVOLS
Enable in the GUI the mounting of shares for TrueNAS to be a client
As a bonus investigate VM performance

Marshalleq.

Thanks for the list. If possible, it would be great if these can each be turned into "suggestions" on jira (Report-a-bug)
On the GUI.. it would be worth noting the clicks you thought were unnecessary. Others could make comments on whether they agree.
Performance issue will start to be addressed as we get close to the February RELEASE. We're still chasing bugs...and setting up test infrastructure.
For TrueNAS to be a client... are you suggesting mount iSCSI volumes or SMB/NFS shares. What is the specific use case?

 

[truecharts]

What happened is that while there was tdarr available in truecharts, the version I used to match up with my docker in unraid wasn't available - going straight back to the problems listed previously in this thread.

There is no stable TrueCharts Tdarr available, only a WIP incubator version. We do not, yet, think it's ready for prime time... Mostly due to Tdarr developers making some choices which make it super annoying to build a solid App for it :\

 

[marshalleq]

Thanks for the list. If possible, it would be great if these can each be turned into "suggestions" on jira (Report-a-bug)
On the GUI.. it would be worth noting the clicks you thought were unnecessary. Others could make comments on whether they agree.
Performance issue will start to be addressed as we get close to the February RELEASE. We're still chasing bugs...and setting up test infrastructure.
For TrueNAS to be a client... are you suggesting mount iSCSI volumes or SMB/NFS shares. What is the specific use case?

Ok, will do that regarding the suggestions - I was avoiding Jira because of how slow the ixsystems implementation of it was (minutes per page load), but I just checked and I think that's sped up now which is great.

The use case is for dockers to use external storage via SMB/NFS. For example, the TDARR encoding app (which is a distributed encoding app), needs all nodes to point to one storage location. In my case, these nodes exist on Unraid, but equally could be TrueNAS and the same requirement would apply. Others have commented on their desire to do the same thing around the forums. Failing the GUI option an official and recommended guide to do it in console would suffice.

Writing this all out reminds me how much I really want TrueNAS to be my NAS!

Thanks for replies.

 

[truecharts]

The use case is for dockers to use external storage via SMB/NFS. For example, the TDARR encoding app (which is a distributed encoding app), needs all nodes to point to one storage location. In my case, these nodes exist on Unraid, but equally could be TrueNAS and the same requirement would apply. Others have commented on their desire to do the same thing around the forums.

Mounting NFS storage can be done natively on kubernetes, it just requires someone to build a SCALE GUI for it.

 

[marshalleq]

That's great to know. Whenever I actually get the skills to run that for the things I use that'd be great. Presently I'd have to build my own versions of containers to do it and I find that a bit too extreme when other solutions exist where I don't have to. From the above it seems IX Systems are keen to take on board the feedback, so I am hopeful for these improvements in future versions.

 

[truecharts]

That's great to know. Whenever I actually get the skills to run that for the things I use that'd be great. Presently I'd have to build my own versions of containers to do it and I find that a bit too extreme when other solutions exist where I don't have to. From the above it seems IX Systems are keen to take on board the feedback, so I am hopeful for these improvements in future versions.

We've had many requests for it and, while skeptical, we'll just put it on the "back-of-the-head" roadmap :)

 

[morganL]

The use case is for dockers to use external storage via SMB/NFS. For example, the TDARR encoding app (which is a distributed encoding app), needs all nodes to point to one storage location. In my case, these nodes exist on Unraid, but equally could be TrueNAS and the same requirement would apply. Others have commented on their desire to do the same thing around the forums. Failing the GUI option an official and recommended guide to do it in console would suffice.

There was something similar done here with NFS.. using the democratic CSI driver.
Is this close to what you want.... enabling other CSI drivers for external storage ?

 

[TheSir]

Will this be implemented in TrueNAS Scale aswell?

ZFS fans, rejoice—RAIDz expansion will be a thing very soon

Founding OpenZFS dev Matthew Ahrens opened a pull request last week.

arstechnica.com

One of the things that urgendly needs to be done is that we can expand a pool with more disks without deleting it before. I created a 2 disk mirror and had to delete it because of a third drive. now i want to add a fourth drive later in my RAIDz1 pool and i cant, because to do so, i had to relete my 3 disk z1 pool again to build a new pool with 4 disks. thats not usable, really :D

 

[truecharts]

There was something similar done here with NFS.. using the democratic CSI driver.
Is this close to what you want.... enabling other CSI drivers for external storage ?

NFS is already natively integrated in Kubernetes.
It doesn't even need Democratic CSI, It just need a GUI for the Apps to expose it.;-)

Other CSI's can easily be added as Apps as well, in case that's needed :)

K.S.

 

[aussiejuggalo]

My 2cents, I like the UI of Scale, it's so much better than my current FreeNAS install (yes I'm still on FreeNAS, lol) but I agree with everyone else, the Apps side of things is a mess. I understand the need for Kubernetes, this is aimed at enterprise after all but If people who are experienced with Docker and Kubernetes are saying they don't understand the implementation then those of us who are inexperienced with it have no hope.

My main complaint is the same as others, not being able to simply assign an IP address for local access to a Kubernetes pod is a massive hassle and clearly a problem for a lot of home users. Truecharts Discord is full of people having networking problems. I've spent the last 3 days trying to assign an IP to Plex so I have local access instead of remote without using the Host-Networking because it apparently breaks stuff and causes issues (although no one has ever said or shown proof of what exactly it can do). I've given up for now because it's just running on a test system so if the Internet goes out and I lose connection I've still got my main NAS. The Kubernetes networking is overly complicated for home users who aren't use to it, looks like it has quite a steep learning curve in general as well.

For me personally I was excited about Scale because of Docker, thought it would mean I can easily run up to date stable apps (as opposed to the normally outdated jails, that break ever other update, especially Plex) but after using it and seeing how complicated and convoluted networking Kubernetes pods are I'm now looking at other options. I'll either go with TrueNAS Core, OMV with ZFS plugin, scratch built Linux system with ZFS or maybe Proxmox / EXSI.

I do have a question, it's probably stupid (remember I don't have much knowledge of Docker, Kubernetes etc.) Doesn't Kubernetes just run on top of Docker so you have a more advanced control system as opposed to the standard Docker system? If so couldn't IX have given us an option to use standard Docker or Kubernetes? I know Scale has been specifically built around Kubernetes but when they first started building Scale would having the option to use standard Docker or Kubernetes have been possible?

 

[truecharts]

My main complaint is the same as others, not being able to simply assign an IP address for local access to a Kubernetes pod is a massive hassle and clearly a problem for a lot of home users.

This has already been fixed in the nightlies in a colabarative effort between iX and us.

Truecharts Discord is full of people having networking problems.

You need to seperate TrueCharts from TrueNAS.
We don't have many reports of TrueNAS networking problems and the most problems people are having with our Apps are related to the more advanced features we offer. Which is to be expected: The more advanced features are offered, the heavier the load on support resources. This is also partly why some of our features aren't offered on official App ;-)

.I've spent the last 3 days trying to assign an IP to Plex so I have local access instead of remote without using the Host-Networking because it apparently breaks stuff and causes issues

You can get local access just fine, but native kubernetes per-App IP's is only available in the nightlies.

(although no one has ever said or shown proof of what exactly it can do).

If you would ask us about our advisories, you generally get an answer. ;-)

The Kubernetes networking is overly complicated for home users who aren't use to it, looks like it has quite a steep learning curve in general as well.

We've just simplified our networking setup 2 weeks ago with a completely revamped GUI.
In it's simple mode it does not ask for any kubernetes specific networking settings anymore.

after using it and seeing how complicated and convoluted networking Kubernetes pods are I'm now looking at other options.

The only option offered by default in the new UI is "port", even beter: All our Apps are expected to be reachable with default settings.
So it's not very clear what is convoluted.
But it looks a bit like you tried to do something (bind an external IP to a pod) that simple isn't supported on RC1 yet.

I do have a question, it's probably stupid (remember I don't have much knowledge of Docker, Kubernetes etc.) Doesn't Kubernetes just run on top of Docker so you have a more advanced control system as opposed to the standard Docker system?

Kubernetes is not some sort of "portainer" like docker-manager (though lots of people view it a bit that way). It manages it's own networking, security etc.
Simply put: To actually spawn the containers, it can use a multitude of "engines" behind the scenes. One of which is docker. But it's not just creating things in docker, it just uses it as a tool to create/build the actual containers.

If so couldn't IX have given us an option to use standard Docker or Kubernetes?

They could, but decided not to support that. Nor support native kubernetes CLI nor support Helm CLI.
But everything added to a product has costs and risks.

know Scale has been specifically built around Kubernetes but when they first started building Scale would having the option to use standard Docker or Kubernetes have been possible?

There was just some confusing PR in the first few months. Actually direct docker access, was never actually part of the product from pretty soon (Q3 or Q4 2020) in the developement cycle.

To be clear: there are still hacks to fully use docker(-compose) they are just not iX supported.

 

[jeyare]

I'm long time runner of Synology in home & SME level.
This year, I realized that a series of stupid decisions killed the possibility of extending the enterprise features of this vendor. I have been a longtime critic that one home and enterprise GUI is not sustainable in the long run. So I installed Beta TN Scale in my VM environment over the summer. I was amazed at how quickly this system could be used. Next, I installed TN Core in parallel. I understand their differences, and I also appreciate the logical decisions of the direction of these two platforms (Scale / Core). I even understand the use of k3s instead of k8s. However, I don't understand the strict binding of k3s without any strong support from TN Scale, including this forum. The GUI on the Scale side is unusable, and I dare say that it is not appropriate for which segment Scale wants to provide its services primarily. It took me several hours to figure out how to break up Docker on the Scale. Again - this forum does not contain a single tutorial. I don't understand because Docker and CLI / Portainer orchestration are great alternatives for microservices, especially for SMEs. For sure - I describe the use for other activities such as media server and the like. Currently, I use dozens of Docker-based microservices on my Synology NAS farm, orchestrated through Portainer. The Docker platform makes this easy.

I would also like to try k3s, but this forum itself is absolutely empty on this topic. To understand me, I don't need to figure out how orchestration works (a lot of useful sources over the internet). It easy.
What I don't need - is to solve why the kubectl command in the CLI does not work (because 'k3s kubectl' works), or why when I run the proxy for the Rancher Dashboard, my Get from the browser gets to the Scale localhost to a specific open port for the Dashboard (tcdump), but I don't get an answer. There is probably a lot of work in the project ahead.

Container Scale networking is a disaster. Using HostNetwork is inappropriate and dangerous. Creating your VLANs, masq., ports assignment and easy/complex configuration of IPTABLES is completely common even in a platform like Synology. In the containerized platform, Scale still has a shorter end of the rope and still has a long way to go.
I would like to use TN Scale as my future platform (see the advantages of OpenZFS over LVM/BTRFS in Syno). However, it will not be possible without the massive support of the containers operations.
Seems to be I'm not alone here.

I understand you embarked on a challenging project. I cross my fingers for you!

 

[truecharts]

I have been a longtime critic that one home and enterprise GUI is not sustainable in the long run.

Thats actually a good point that is not documented anywhere:
Apps don't require a GUI, you can build one just with the name field and the rest of settings in values.yaml on your own catalog :)

It's all just Helm in the background, so any enterprise person should be able to port their Helm deployements to SCALE in a blink. Even when the existing catalogs doen't offer the flexibility they want :)

However, I don't understand the strict binding of k3s without any strong support from TN Scale, including this forum.Again - this forum does not contain a single tutorial.

This forum is not an official iX Systems resource, it's a community effort. Just like our Project is a community effort with some iX employees on our discord as well.
Employees are here to "nudge things along" from time-to-time, but their only documentation is... their own documentation.
Which is not done yet, as explained by @morganL

I would also like to try k3s, but this forum itself is absolutely empty on this topic. To understand me, I don't need to figure out how orchestration works (a lot of useful sources over the internet). It easy.

SCALE offers Apps based on kubernetes/k3s.
k3s is the backend, but on SCALE only the GUI or API are the official ways of interacting with anything in the backend. (same as with Samba, iSCSI, Gluster etc).

I would also like to try k3s, but this forum itself is absolutely empty on this topic. To understand me, I don't need to figure out how orchestration works (a lot of useful sources over the internet). It easy.

There is always the chance with Hacks for non-released products that no one is interested in your specific hack enough to help you. Thats not fun, but a sad fact of running unsupported setups

Using HostNetwork is inappropriate and dangerous.

Hear Hear!
We have repeated this time and time again, happy to see someone understand :)

The clean way is loadbalancers bound to specific IP's/VLANs/Interfaces.
Which is indeed not supported on RC1. But... read along... ;-)

Creating your VLANs, masq., ports assignment and easy/complex configuration of IPTABLES is completely common even in a platform like Synology. In the containerized platform, Scale still has a shorter end of the rope and still has a long way to go.

Assigning pod LoadBalancers to specific interfaces, using MetalLB, should be possible in RC2. Our proposal/fix to allow the internal loadbalancer to be disabled, is merged and we have already a MetalLB App ready. That should also allow binding Apps to specific VLAN's for example :)


Anyway, it might be more worthwhile to work on which specific features/changes you want to see in the SCALE UI and file Jira suggestions for them. That way we, as a community, can all work towards the goal of improving SCALE.
(rather than putting it in a dumpster and using portainer)

 

[Ur-Pedant]

Nineteen hours trying to get multipath access to SAS controllers and still no joy—at least, not when a ZFS pool is imported/created in the GUI. Scale apparently wants to handle all pool & dataset management itself, but why? Why reinvent the wheel instead of using what's already there? I was so hopeful for Scale, too. Conceptually it ticks all the boxes for my use-case, but that doesn't matter if the thing straight up doesn't work. Sooooo back to Proxmox I go. Maybe I'll spin up a Scale vm in a few months to see how it's coming along.

Maybe.

 

[Patrick M. Hausen]

Nineteen hours trying to get multipath access to SAS controllers and still no joy

Is multipath supposed to be supported? In the CORE/Enterprise products it's an Enterprise feature ...

 

[Ur-Pedant]

Is multipath supposed to be supported? In the CORE/Enterprise products it's an Enterprise feature ...

Only the Shadow knows. What I do know is that it's supported by Debian. As in, install multipath-tools and it works automagically. This is true even on Scale, or rather, on the Debian that Scale runs on.