k3s-server uses 10% CPU for no reason

[winnielinnie]

"It's a feature!"

 

[nasBuilder]

Yeah, a good joke :( Any other way to get photoprism up and running without this pathetic process?

 

[dirtyfreebooter]

no, this is not been solved and likely will not, as the k3s authors closed the github issue, more or less saying its working as intended. works great for google in their datacenter, but really falls short for home users with a single NAS just looking to maximize powersavings IMHO

at one point, it was mentioned that TrueNAS might offer different app backends, K3 vs docker, etc, but it seems like reality set in and now it seems like an official docker backend will never happen and next release will go further into k3s ecosystem. (i know k3s on truenas right now is backed by docker currently but seems likely to change in the next release?)

 

[nasBuilder]

10% of the CPU for nothing and it's working as intended. What are they smoking...
Thanks for the clarification anyway

 

[dirtyfreebooter]

i really want to like truenas and really like having an open source appliance that just works and for that, you basically pay 10% of idle cpu lol.

for my use-case, i could probably install debian, portainer (business license for home users up to 5 nodes is now free), zfs, samba, nfs and have nearly the same features. i know truenas does a bit of work with ZFS and permission handling, but for my home use-case, i could get by without it for sure. ..

could maybe run a VM on scale, but running VMs cuts your ARC cache to 50% of memory due to linux's zfs implementation not counting zfs arc as buffer cache, but as in-use memory -> https://github.com/openzfs/zfs/issues/10255

so various possibilities to try and workaround, but each seems like it has its own issues. computers. they are the worst, basically.

 

[winnielinnie]

What are they smoking...

Burgers and hotdogs.

Use the exhaust fans on your case as heating elements to cook and smoke your own foods, which saves on time and money; rather than being wasteful and buying extra propane for your outdoor griddle.

 

[nasBuilder]

Yeah, I'm actually seriously considering this. It's probably gonna take me 3 months to get zfs, samba and docker going on debian with some scheduled snapshots but having some stupid process not let my machine go to C10 after all the struggle of finding the right mainboard to save electricity is just really painful :(

 

[Etorix]

What about running CORE with your favourite Linux server distro in a VM?

 

[dirtyfreebooter]

one of the main things i would want to do in a VM is PCI passthrough either an intel iGPU or NVIDIA GPU for plex, which i don't believe is supported with bhyve.

bhyve/pci_passthru - FreeBSD Wiki

 

[nasBuilder]

What about running CORE with your favourite Linux server distro in a VM?

Kinda intimidated by core and I dunno, does it even support "powertop" and someone was writnig about cache limitations with VMs, does it not apply to core? Sorry if these questions are dumb

 

[dirtyfreebooter]

Kinda intimidated by core and I dunno, does it even support "powertop" and someone was writnig about cache limitations with VMs, does it not apply to core? Sorry if these questions are dumb

no the ZFS arc cache memory not classifying itself as buffer cache and thus always in-use is linux only

 

[nasBuilder]

So, powertop isn't available for freebsd and apparently you can't passthrough the iGPU to the VM so truenas core is out of the question.

 

[Etorix]

Plex in a jail can use an Intel iGPU for hardware transcoding (requires C2x6 chipset).
As far as I understand, powertop is an interface collecting various power statistics from specific Linux kernel calls. So, no, it is not available anywhere else than on Linux—though it should be possible to find equivalent data under FreeBSD using suitable kernel calls.
SCALE comes with k3s, which takes 10% CPU even when it has nothing to do. The next release, Cobia, may come with a fix for the 50% limitation on ARC, and should come with a switch to containerd and the removal of docker.

If you end up picking up another NAS solution (OMV, with or without the ZFS plugin, maybe?), please do not hesitate to come back here and report how it worked for you.

 

[nasBuilder]

Sure, so I've been playing with exactly OMV and ZFS-plugin for the past 24 hours. It's been a rather annoying experience because it didn't quite work out of the box or not quite as easily as I thought it would. One needs to install proxmox kernel to get ZFS-plugin going and I don't even quite understand what proxmox is :-D Anyway, I did it and it's working now, the ZFS plugin in OMV is absolutely not needed though, it's very basic and I found it much easier to read up on ZFS commands and do the basic stuff on the command line. As far as I understand the ZFS-plugin is maintained by one guy (kudos to him of course) who does not even use it and would rather stop doing so. Unfortunately there is no GUI for replication and keeping snapshots for a specific amount of time. I'm currently not sure how to organise this for my needs, probably with bash scripts and cron but it's gonna take me a long time to get it going, truenas is really quite comfortable when it comes to that.

Other than that, my photoprism library from truenas didn't not get picked up by photoprism plugin on OMV either. So that's another problem which I have no idea how to approach at all at the moment. Next I'm probably going to try unraid, the've started implementing ZFS spread over two releases and some basic operations are available already. It's not free though.

All in all, I'm still not sure what path to go, I was absolutely ready to go with truenas scale but having spent so much time trying to find the right components in terms of power consumption, will not do it due to the k3s-rubbish even if it seems petty. I think that a clean debian or ubuntu server with full control over everything is not entirely out of the question. OMV without any plugins seems like an option though making it easy to set up users and samba shares.

On the plus side, I've learnt some stuff over the last few days, specifically I've learnt about LUKS and it doesn't seem too hard to set up. Now it seems that zfs native encryption is entirely unnecessary for my use case (especially since I do not understand what stuff is still visible, for example can you read file names on the encrypted zfs dataset?), I'm probably going to set up LUKS plus ZFS without encryption on top of it (and the same for the external drive onto which I'm replicating) and go with that. Although I seem to lose the ability to only encrypt one dataset with LUKS.

Most important takeaway: zfs console commands are not as scary as I thought :-D

UPDATE
Seems like LUKS can't quite solve my usecase because it can only properly encrypt an entire device. Encrypting folders or datasets would mean specifying their size in advance which does not work for me. Judging by the information at https://blog.heckel.io/2017/01/08/zfs-encryption-openzfs-zfs-on-linux/#What-s-encrypted file names are not revealed, only dataset and snapshot names which is oooooook I suppose :-D

 

[Etorix]

Thanks for the report, and best luck in your endeavour!
Not to try your luck too much, avoid encryption: Any mistake with key management can lock you out of your data. And always have a backup…

 

[nasBuilder]

Thanks for the report, and best luck in your endeavour!
Not to try your luck too much, avoid encryption: Any mistake with key management can lock you out of your data. And always have a backup…

If you are interested, I ditched all of this entirely. Instead, I installed a fresh Debian 12 (you can also take ubuntu server) with zfs, docker and samba. The boot NVME is encrypted via LUKS, the zpool is using zfs native encryption. It works perfectly well, zfs automatic snapshots can be done with zfs-auto-snapshot, I wrote a script for incremental replication to an external drive. It's all working and I have full control, I'm actually using ffmpeg and exiftool from photoprism docker container to do some transcoding and image metadata parsing before importing it all to photoprism.

All in all I have a raid-z1 pool with 3 SSDs and it's idling nicely in C10 consuming 9 Watt. With "fancontrol" you can program how fans behave, the CPU fan is not moving unless the CPU reaches a certain temperature. But we're getting too much into detail here. So it's a dead silent system which was my requirement.

Very happy! It requires some IT-knowledge though and bash scripting is just absolute shit, it never works they way I expect it to. I've never had to google as much before :-D The only thing that is left is some kind of automatic regular pool scrubbing and smart monitoring. Can be easly done with cronjob but maybe there is a tool like "zfs-auto-snapshot" that does the cron jobs .

 

[Whattteva]

Yeah, a good joke :( Any other way to get photoprism up and running without this pathetic process?

It sounds like a joke, but it really isn't. There have been multiple issues like this one being reported by people and k3s people just keep closing the issues, essentially saying it's working as designed/normally and it's supposed to use that much idle.

 

[Etorix]

Very much interested in knowing how it played out—even, and especially, if it turns out that TrueNAS was not the best fit for your requirements—, so thanks for the report!

 

[R.Heaton]

I was also annoyed by k3s constantly using about 10% CPU. Kubernetes is overkill for my single-node personal home NAS. I'd rather use vanilla docker.

Here's the trick I use to get rid of k3s while still having containers running.
* Set up containers with the built-in Apps web UI.
* Stop the k3s service with command 'systemctl stop k3s.service'.
Containers continue to work without k3s server, though the Apps web UI doesn't.
Whenever I need to set up new containers, I run 'systemctl start k3s.service' to enable the Apps web UI again.

What's more, I added a Post Init script (in System Settings -> Advanced -> Init/Shutdown Scripts) to stop k3s service automatically after boot:
'sleep 10m && systemctl stop k3s.service'.

Thanks, running this command fixes my issue with k3S
Rusty

 

[truecharts]

Thanks, running this command fixes my issue with k3S
Rusty

This is generally a bad advice, as it can and will lead to numerous issues.
One of which is, for example, nextcloud cron not working. But there are many other potential area's issues can and will show themselves.