Core 12.0-U7 to SCALE 22.02-RC.2 homelab migration experience

dirtyfreebooter

Explorer
Joined
Oct 3, 2020
Messages
72
i have been testing SCALE on a separate test box off and on since ALPHA. really before RC.2, SCALE definitely had some basic usability issues, especially the Web UI, but after RC.2 came out, stuff was starting to seem real :) i moved my Dell R7515 running TrueNAS Core 12.0-U7 to SCALE 22.02 RC.2. i wasn't running any VMs, i had a separate proxmox 1u server for that, since FreeBSD hypervisor leaves a lot to be desired, IMHO. just some jails on Core. other apps that i was running on proxmox where either as VMs, LXCs, or in docker (in a ubuntu VM).

i did a fresh install of SCALE and just imported my pools. definitely had some weird permission quirks, but nothing that wasn't easy to fix. one by one i setup my previous config, cloud syncs, etc. not too bad, would have upgraded, but wanted to go through all of SCALE menus anyways.

i was able to get all my homelab services in docker, no more VMs, no more LXCs, no more proxmox, was able to completely turn off that machine, which is great, since the Dell R7515 is already super overkill for NAS.

my experience so far has been great. just what i expect from iX and man when they say release candidate, they mean release candidate.. today in software, that seems rare, as full releases these days barely seem to qualify as alpha, let your customers do the testing / soak!!! hah.

the only thing i found strange was what seems like the overkill with docker / k3s / helm charts. being a homelab i certainly have a small use-case, but i found that TrueNAS charts and Truecharts do things differently, so mixing and matching official apps, Truecharts apps, and straight docker containers is a mixed bag, feels a bit disconnected. sometimes, apps i wanted to run weren't available in either. finally, all the chart apps wrap the actual docker image, if the chart app doesn't expose some environment variable or setting or clamps down a setting, you are basically screwed.

if you want to stay on the latest container as soon as its released, with charts you have to wait until the chart updates, there is no option to specify the docker image with a chart (which would also be great for testing, etc)

so for my services, i just ran everything as plain out docker images, "Launch Docker Image" option, then i am able to follow any README or whatnot that comes with the docker image and stuff just works and it easy 1-2-3. services i am running are
  • Plex
  • Syncthing
  • Ghost Blog (2x)
  • deluge (torrents)
  • Handbrake
  • PiHole
  • NGINX Proxy Manager
Everything is reversed proxied through NGINX Proxy Manager with Let's Encrypt SSL certs. Some services only communicate over the cluster network, 172.17.x.y, some i assigned static IPs (things like PiHole each need to bind to port 53 are not possible with the k3s restriction of ports < 9000.. not sure how TrueCharts catalogs get around that). Could have also used "host networking" but liked the separate IPs and the multi-homed NIC isn't a performance issue in a homelab (network is 10g SFP+ anyways)

Only issues I have ran into are
  • Containers launched via "Launch Docker Image" do not get assigned a cluster DNS name unless a port forward is setup, even if all you want to do is internal k3s cluster level communication between containers. Seems like BUG
  • Containers launched via "Launch Docker Image", the "upgrade" does not work. When a new image is available, it correctly detects the new image, the "upgrade" menu item restarts the container never pulls in the updated image.
    • Workaround is to stop the container, go to the images page, update the image from there, go back to the installed apps page and start the container
    • This also seems like a BUG
  • Cannot bind to ports < 9000. This is a dumb restriction IMHO
    • mostly worked around that by either assigned static IP or using host networking
  • There needs to be some design done around load balancers and ingresses with TrueNAS apps, Truecharts, docker images, and other 3rd party catalogs
  • Because ZFS is now a Linux kernel module, you cannot use the tunables to adjust things like the arc size, so you have to make initscript jobs that echo <size> > /sys/modules/zfs/parameters/<blah> .. definitely easy enough to workaround, but the differences between Linux and FreeBSD are obvious here and a proper UI solution is yet to done for SCALE

1640581924491.png


1640581497810.png
 
Last edited:
Top