Kubernetes for homelabs

[GuyFromMars]

Does iX currently have a good solution for clustering for home labbers? ...I forced myself to migrate from OpenMediaVault to TrueNAS Scale in an effort to move to Kubernetes, however, I'm struggling. My intent is to use Scale as my storage solution and perhaps put Nextcloud, PLEX or an NVR solution for streaming on another more powerful box. Is this something currently feasible in scale? It this what TrueCommand is suppose to handle? If TrueCommand is the only way to accomplish this, does the subscription just enable the license essentially, while I can host everything on my local server without interaction from iX (for security/privacy reasons).

 

[Kris Moore]

What will the more powerful "Apps" system be running? TrueNAS or something else? Usually for home-labs you'd want to host the Apps directly on the storage, which will give you the best overall performance with your storage itself. If you need two physical machines, you'll end up with some variety of exporting SMB or NFS shares to the 2nd system and interacting with storage that way. TrueCommand isn't required for any of that.

 

[GuyFromMars]

What will the more powerful "Apps" system be running? TrueNAS or something else? Usually for home-labs you'd want to host the Apps directly on the storage, which will give you the best overall performance with your storage itself. If you need two physical machines, you'll end up with some variety of exporting SMB or NFS shares to the 2nd system and interacting with storage that way. TrueCommand isn't required for any of that.

Currently I run SCALE-23.10.0.1 on a ProLiant MicroServer Gen10 w/AMD Opteron X3421 APU, 16GB ECC RAM. That system may be well suited for HDD storage, but not so much for running Plex/Jellkyfin, NVR Software (Frigate, etc), Nextcloud, etc, all concurrently. I thought with Scale, I could presumably setup my ProLiant as my primary long-term storage with large HDD, then have other machines running Scale with SSD's, high-end graphics cards, etc to spread workloads across appropriately tuned machines using Kubernetes.

 

[Kris Moore]

TrueNAS today in general isn't well setup to act as a "client" of some storage which is being served from some other storage system. You'd have to do some manual hackery to mount the shares to the 2nd box and run apps on top of them. Here be dragons.

There are some improvements coming to SCALE down the road for better capabilities to mount an SMB share directly to an App which enables this. (We did that for our Syncthing Enterprise App already), but its not fully enabled everywhere yet, so you'd be waiting a bit before operating that way is better supported.

 

[GuyFromMars]

I guess I've misunderstood the marketing material. I also suppose that's why Trucharts is migrating away from Scale-first focus? https://truecharts.org/news/stability-tiers-and-helm/#google_vignette

Why are apps, especially using Kubernetes, part of the iX offerering if we cant actually create nodes across different systems?

 

[Kris Moore]

Yea, that circled section isn't talking about K8s / Apps, completely unrelated. Lots of our customers in that example use-case would use an enterprise clustered virtualization solution for the compute (You can probably guess what the most common ones are) backed by TrueNAS for the storage component of those many incoming video feeds.

The TrueCharts bit is complicated.. In the end they have a set of goals that don't always align with what we are doing on the TrueNAS roadmap, which isn't too unexpected.

 

[GuyFromMars]

@Kris Moore Thanks for the feedback & appologies for the delayed reaponse.

Maybe we're missing each other? Are you saying TrueNAS Scale is a great storage solution, but NOT meant to be a good platform for Kubernetes &/or VM's? I thought iX was transitioning to Linux & Kubenetes, using words like hyperconverge & scale to expand from JUST a strong storage solution to one that had the flexibility of Kubernetes/ VM's to manage computing loads across mutliple machines & hardware. Seems like there was much fanfare over "apps," especially the ones TrueCharts put a lot of energy into. However, due to conflicts with TrueCharts, it looks like iX has stepped back from emphasis on apps/kubernetes, which to me is disappointing.

I was hoping to find a software platform that I could put on all my machines, manage/scale both my computing & storage from one place, but it sounds like that is not what iX intends for Scale? Why bother with Kubernetes, VM's, PCI-E Passthrough, GPU sharing, "Single Pane Management," etc? What am I missing?

 

[GuyFromMars]

@Kris Moore Thanks for the feedback & appologies for the delayed reaponse.

Maybe we're missing each other? Are you saying TrueNAS Scale is a great storage solution, but NOT meant to be a good platform for Kubernetes &/or VM's? I thought iX was transitioning to Linux & Kubenetes, using words like hyperconverge & scale to expand from JUST a strong storage solution to one that had the flexibility of Kubernetes/ VM's to manage computing loads across mutliple machines & hardware. Seems like there was much fanfare over "apps," especially the ones TrueCharts put a lot of energy into. However, due to conflicts with TrueCharts, it looks like iX has stepped back from emphasis on apps/kubernetes, which to me is disappointing.

I was hoping to find a software platform that I could put on all my machines, manage/scale both my computing & storage from one place, but it sounds like that is not what iX intends for Scale? Why bother with Kubernetes, VM's, PCI-E Passthrough, GPU sharing, "Single Pane Management," etc? What am I missing?

@Kris Moore Happy New Years! I know you're busy, but hoping you could answer my questions here, as I know many more have similar questions.

 

[Kris Moore]

@Kris Moore Thanks for the feedback & appologies for the delayed reaponse.

Maybe we're missing each other? Are you saying TrueNAS Scale is a great storage solution, but NOT meant to be a good platform for Kubernetes &/or VM's? I thought iX was transitioning to Linux & Kubenetes, using words like hyperconverge & scale to expand from JUST a strong storage solution to one that had the flexibility of Kubernetes/ VM's to manage computing loads across mutliple machines & hardware. Seems like there was much fanfare over "apps," especially the ones TrueCharts put a lot of energy into. However, due to conflicts with TrueCharts, it looks like iX has stepped back from emphasis on apps/kubernetes, which to me is disappointing.

I was hoping to find a software platform that I could put on all my machines, manage/scale both my computing & storage from one place, but it sounds like that is not what iX intends for Scale? Why bother with Kubernetes, VM's, PCI-E Passthrough, GPU sharing, "Single Pane Management," etc? What am I missing?

Just caught on to this thread. We are by no means backing away from Apps, if anything we're working hard on making sure the support for them gets better and smoother, with more options for people to run containerized or other workloads. We've even added some preliminary community level support for systemd-nspawn "sandboxes" in Dragonfish and you should expect you'll hear more noises on the container front before too long.

Sandboxes (Jail-like Containers)

Provides advanced users information on deploying custom FreeBSD jail-like containers in SCALE.

www.truenas.com

 

[GuyFromMars]

Thanks for the update, although I wish iX would focus on managing Kubernetes well, instead of jumping to yet another containerization method, which is what your announcement sounds like. :/

 

[CountBuggula]

Thanks for the update, although I wish iX would focus on managing Kubernetes well, instead of jumping to yet another containerization method, which is what your announcement sounds like. :/

What he's not mentioning (and for some reason none of the ix-systems people will address this directly) is anything about app clustering.
Kunernetes clustering and VM HA/failover were on the original roadmap announced for Scale, and supposed to be part of the first major update after Angelfish. Ix-systems quietly removed those from roadmap/marketing/updates and those of that had decided to try Scale because of those promised features have just had to slowly realize that they're never coming.
I really wish someone officially would come out and say "hey, we ran into big problems with clustering and decided to pivot to other priorities. App and VM clusters are no longer coming to Scale, sorry for the inconvenience!"
Instead we get them being super dodgy about the whole thing which just serves to frustate us all and erode what little trust we had with them.

Also, to answer one of your other questions, TrueCharts apps support connecting NFS shares from other systems directly to your apps without messing around with the complexity of trying to mount a share on Scale to pass through to apps. It works quite well.

 

[GuyFromMars]

@CountBuggula, it does at least seem like right now Apps are on iX radar & they are deciding what to do moving forward: https://www.reddit.com/r/truenas/comments/1akgua4/container_technology_poll/

@Kris Moore SCALE users would appreciate some clarity from iX, soon, on iX's long-term app plans.

...I've migrated from my Docker-Compose stack on Open Media Vault to K3's on SCALE, specifically because a trusted NAS platform was crossing into Linux, with a focus on scaling BOTH storage and apps (app-clustering). With it being increasingly clear that app clustering is a long way off or not planned at all, we just need iX to set clear expectations.

 

[Kris Moore]

We're wrapping up some long-term planning now. Nothing I can specifically announce just yet, apart to confirm we are indeed looking seriously at Apps and how to make them more reliable for 95%+ of our user-base. Right now I can say that app clustering is *not* on our short or mid-term road map, long-term is TBD.