Citadel maintenance and upgrade log - TrueNAS SCALE edition

[ctag]

This thread is a spiritual successor to my FreeNAS build thread.

In middle school I would visit my local thrift store and pick apart donated PCs to buy components, eventually assembling my own desktop from the bits. Not knowing any better I housed it in a shoebox:

I enjoyed the DIY, but it came with some challenges. In highschool I started the typical home photo and video collection for school trips and projects I tinkered on. When it was almost entirely lost in a hard drive failure I felt the unexpected punch in the gut that comes with real loss, and a new door to hobby adventures was opened: how do I keep these things that have only value to me intact?

The solutions varied for a while. But in early 2018 I settled on FreeNAS, and I've been benefiting from the hard work that goes into FreeNAS/TrueNAS and the support of this community ever since. So as I start this next chapter of my not-quite-a-NAS-blog, I'd like to highlight how grateful I am to the various forum members and iX contributors that have made my DIY data backup journey possible.

Ever since the scrounging around in middle school, my computer hostnames have been tributes to fictional spacecraft. And while it seems almost too corny to continue now, I'm still sticking with it. My TrueNAS powered machine is called Citadel after the fictional hub world in Mass Effect.

Citadel started out as a repurposed Dell T7500 tower server with FreeNAS installed and 4 8TB hard drives for 20TB of usable space. Today it's the same Dell server with TrueNAS Cobia and has 8 8TB drives in a 41TB pool, along with an SSD pool and secondary CPU card for virtualized services.

 

[ctag]

A nonprofit I volunteer with has their website hosted through a CPanel, and for a while now I've wanted to keep scheduled offsite backups in case we need to revert. This should have been a pretty simple ask (like, the bare simplest ask of a hosting system designed for casual users), but for whatever reason it turned out not to be. Instead, CPanel will gladly hand over an account backup... if you sign in and click a button. No automation to be had.

So I eventually worked out (googled) a cpanel cron job that would make SQL backups of the website database and drop them in the user home directory. And then I learned that I could upload a SSH public key and use that for SFTP. So the plan became to just vacuum up the whole user directory with sftp and pray I'd never need to restore from it.

Logging in with Filezilla worked just fine, but I ran into some trouble with setting up cloud credentials in TN. So for the time being I'm using cron as a stop-gap. Steps taken are as follows.

First, generate a ssh keypair at Credentials -> Backup Credentials -> SSH Keypairs -> Add

When done generating, click the download button and then scp the private key into the home directory on TN.

Copy the public key into CPanel: Security -> SSH Access -> Manage SSH Keys -> Import Key.
Then go to Manage SSH Keys, click Manage next to the new key and then Authorize to allow it to be used for your CPanel account.

I ran sftp manually and recorded the FTP-style commands I needed in a script file:

Pretty simple.

Next go to System Settings -> Advanced -> Cron Jobs and add:

The cron command is:

sftp -4 -i /root/cpanel_private_key_rsa -b /root/cpanel_sftp_script user@nonprofit.org

 

[ctag]

Setting up NFS share to Truecharts app on Truenas Cobia​

Solution is the link in last paragraph of this post.

I recently attempted installing a Truecharts app that I wanted to have access to the storage pool on my system (rather than the default applications pool, which is SSD based). I recall being warned against using hostPath a while back, and so the default option to accomplish my goal seemed to be NFS shares. But things didn't go smoothly, here is what I missed.

At first I created a fully-stock NFS share from the web-ui. Set the path and left everything else alone. But when I attempted to install and start the app, I got errors complaining about the nginx configuration:

I believed there was a problem with the app and not with my understanding. After toiling a little longer on different options during install, I turned to the Truecharts support channel on Discord. They began offering some troubleshooting suggestions, and notably one of the helpers tried installing the app with all-stock settings and claimed it worked fine. Interesting! I set about deleting my failed app, and following along to install again without changing any of the default options. It worked for me too! So the app - at least in its default config - does work.

When I changed the app's volume back to NFS share, it went back to failing to deploy. So now I had a likely culprit, something with the NFS share was at fault. The Truecharts support users heard that and figured out the issue: You have to set up the NFS share with maproot configured, apparently. I went back and added root as the maproot user (and group), and that solved the issue.

 

[anodos]

Setting up NFS share to Truecharts app on Truenas Cobia​

Solution is the link in last paragraph of this post.

I recently attempted installing a Truecharts app that I wanted to have access to the storage pool on my system (rather than the default applications pool, which is SSD based). I recall being warned against using hostPath a while back, and so the default option to accomplish my goal seemed to be NFS shares. But things didn't go smoothly, here is what I missed.

At first I created a fully-stock NFS share from the web-ui. Set the path and left everything else alone. But when I attempted to install and start the app, I got errors complaining about the nginx configuration:
View attachment 73644

I believed there was a problem with the app and not with my understanding. After toiling a little longer on different options during install, I turned to the Truecharts support channel on Discord. They began offering some troubleshooting suggestions, and notably one of the helpers tried installing the app with all-stock settings and claimed it worked fine. Interesting! I set about deleting my failed app, and following along to install again without changing any of the default options. It worked for me too! So the app - at least in its default config - does work.

When I changed the app's volume back to NFS share, it went back to failing to deploy. So now I had a likely culprit, something with the NFS share was at fault. The Truecharts support users heard that and figured out the issue: You have to set up the NFS share with maproot configured, apparently. I went back and added root as the maproot user (and group), and that solved the issue.

Do note that depending on the situation allowing NFS clients to access an export as root could be a significant security issue.

 

[ctag]

Do note that depending on the situation allowing NFS clients to access an export as root could be a significant security issue.

Thanks anodos. I should mention that I still have some confusion here. The app is given PUID 568, the apps user. I guess that the runAsUser directive is what controls the access mechanism then?

 

[ctag]

All of my internet-facing apps are kaput, with timeout errors through cloudflare. Including the one that gives me access to the truenas UI.

I don't know if this was Colonel Mustard with the candlestick or Professor Plum with the revolver, so I'm going to jot down a list of events from the past day.

1. Yesterday afternoon.
   1. Notice that heavyscript is skipping app updates because almost all of them have major version changes.
   2. I start tinkering with Storj after reading that they partnered with Internet Archive.
   3. I try installing the Truecharts Storj-node app. It won't finish deploying. It looks like the app needs manual intervention with a shell to generate the identity files, and I can't locate any documentation for the Truecharts app online.
   4. I do find documentation for a Truenas community app - Storj, which includes installation instructions.
   5. I add firewall rules in my router to try and get tcp+udp port 28967 forwarded.
2. Yesterday around midnight to 12:30am.
   1. I get around to deleting the Truecharts app that's been hung up in Deploying state.
   2. I install the Truenas community app for Storj. Configure it to use datasets in the storage pool. Same forwarded port.
3. This morning
   1. I woke up to a text from someone who has a colocated server at my house. Asking WTF the internet is down and has been down since ~1:30am.
   2. I check. The internet connection is down.
   3. I reboot the router, and the connection comes back.
   4. But none of the Truenas apps' ingress comes back. I log into Truenas with the LAN ip address.
   5. Expecting that the apps just need to be restarted, I try upgrading them starting with Traefik and clusterissuer. Then I read the announcement that Truecharts just put out major changes and recommend waiting a few weeks to update.
   6. I roll back the two apps. Still no good.
   7. I feel like I've seen this before. I try deleting and re-installing (the 10.x release) external service app for Truenas web-ui access. Still no good.
   8. Since I'm planning to reboot TN anyway, I go ahead and run the system update to 23.10.1. The system reboots and the apps are started. Still no good.

Logs for Traefik show it complaining that it can't find the service. I'll get the exact error and post it when I can.

 

[ctag]

Hey, ctag. Here's something to remember next time this happens:

Google Fiber will change your public IP if your router goes offline for a while​

That was the issue. The Truecharts Discord support users were kind enough to spend an hour getting me to "port 443 isn't being forwarded through your router" at which point I realized the IP had likely changed.

Apps are accessible from the internet again, and I have set up cloudflareddns app to manage automatically updating the IP address next time it changes.

 

[ctag]

Choosing a torrent client for seeding​

I want to get back into leveraging my symmetric fiber internet connection for seeding Archlinux and openstreetmap files. Back in the old Freenas days I used Deluge in a jail. It was a mess, with custom-compiled python eggs to get RSS plugins working and whatnot. But I did get pretty familiar with Deluge, and found myself liking it well enough. This time there are a few apps to choose from, and I thought it would be a good moment to revisit my assumptions and pick something to go forward with.

Transmission​

I didn't test Transmission. I had a bad time with it a decade ago, and despite my claim of clear-eyed decision making I'm holding a grudge here.

Flood UI apps​

I tried flood+rtorrent and while I like the pretty interface, I think I'd rather prioritize stability over it, and I was turned off by claims on their github that you couldn't pass env variables to the rtorrent container when using flood.

rtorrent​

rtorrent is billed as the cut-throat performance and stability champion, and uses its own branch of libtorrent (known as libtorrent-rakshasa, whereas deluge uses libtorrent-rasterbar). That means it doesn't support web-seeding, which Archlinux ISOs appear to support, so rtorrent is mostly disqualified on that point.
I did install the rtorrent-rutorrent truecharts app though, and found that it required some surgery to even download the three Archlinux isos in the RSS feed to begin with. Not great, and I'm much more skiddish of mucking with these apps than I was with a jail I had control over.

• https://bbs.archlinux.org/viewtopic.php?id=260359
• http://blog.asiantuntijakaveri.fi/2012/06/manually-adding-tracker-to-torrent-with.html

qBittorrent​

I actually haven't used qBittorrent before, their website design was off-putting enough that I lumped it in with uTorrent originally.
The truecharts qBittorrent app came with the recommendation of one of the TC Discord moderators, and the app configuration screen is more fleshed out than others, which is nice.
What's not nice is that the app requires a trick to get the custom-generated password, which did not work out-of-the-box for me. Rather, what doesn't work is logging in with Firefox. I was able to log in with Chromium and then whitelist to remove log-ins from my computer entirely.
With that out of the way, the qBittorrent UI is probably the most impressive web one I've seen, clean and functional.

Deluge​

Ah, familiar ground. The truecharts deluge app is bare-bones, but works. The docker container ships with python 3.11, so I had to do the python rain dance again to build the plugins, since packaged ones are only available up to 3.10.
Deluge's settings page shows an (incredibly tiny) exclamation point next to the inbound port, which I assumed mean UDP forwarding wasn't working, since - you know - the truecharts UI explicitly calls it the TCP port. But that turns out to not be the case, and both ports are forwarded, so I'm not sure what that warning is for.

At this point it's a toss up between the last two. The main difference is qBittorrent being built in C++ and Deluge using Python. Both seem usable but still littered with dings against them.

 

[ctag]

Increasing disk size on VM​

I have a Debian VM that has burned through its entire 10GB virtual hard disk. Rather than muck with logrotate settings, I'm just going to increase the disk size. Buckle up and follow along!

Step 1: Increase the virt disk capacity​

Editor's note: Maybe backup the zvol before doing this? If I knew what I was doing I would tell you how.
Shut down the VM and log into TrueNAS UI. Go to Virtualization -> VM -> Devices -> Disk -> Edit. Take note of the disks path:

Go to the Datasets page, Select the VM disk dataset, and click Edit Zvol and change the size as desired. I did not select "force size"

Click Save.
The VM should still boot here, but it will show the root partition as still the same size.

Step 2: Use Live ISO to increase the partition​

Back at the VM, click Devices, Add, and then add a CD with your chosen Linux live-image ISO. Save it with a device order below your hard disk device. Boot and use TrueNAS's awesome "Display" VNC connection to navigate to the live-boot shell.
There are many guides explaining how to delete your root partition and re-create it larger, but I decided to keep searching and eventually found instructions to use growpart instead.

• The live disk didn't include gdisk (seriously?!) so it had to be installed with `apt install cloud-utils`.
• Then use gdisk to delete the swap partition (if swap is the last partition, blocking your main ext4 from growing).
• If there are superblock warnings in gdisk, fix them now.
• Use growpart to make your root partition fill the disk.
• If you want a swap partition, go back and increase the zvol size again. Growpart doesn't seem to be capable of nuance for leaving a gig or two for swap. If you follow this method remember to use 'mkswap' again, and edit /etc/fstab and /etc/initramfs-tools/conf.d/resume with the new UUID.
• Otherwise set up a swapfile.
• Use resize2fs to grow the underlying filesystem
• Cross your fingers and reboot!

 

[ctag]

Checking which ports apps are using​

Instead of manually crawling through the status page for each app, here's a one liner to check if a given port is free before putting it to use:

Code:

k3s kubectl get svc -A | grep -i 8801

 

[ctag]

Comparison of VM and App NICs​

Google Fiber came out and hooked up 5gbps symmetric FTTH.

TL;DR​

The Intel VM card caps out at 600mbps. The VirtIO VM card can pull 1500mbps. And the default K8S (apps) connection goes the distance with the full 5200mbps.

Host FreeNAS Scale​

Citadel is the only computer here with a NIC that can support the full connection speed, so I turned to it for initial shakedown testing of the new hookup:

That result is courtesy of speedtest.net's CLI shell script, run from the TrueNAS web-UI shell.
This is great! Citadel's host OS can see the full bandwidth, but what about the various VMs and K8S-based apps?

VM with Intel e82585 (e1000) NIC​

You can tell a VM has this NIC by looking at the details on the web-ui, or by running `lspci | grep -i ethernet` from a shell:

Results:

VM with VirtIO NIC​

Here's the VirtIO device from lspci:

And results:

Truecharts custom-app​

And here's why it's so hard to give up Truecharts and go back to VMs:

Well, that along with the hindsight that I don't want to be in charge of setting up manual nginx reverse proxies anymore.

 

[ctag]

Over the past few days I've been receiving an increasingly frequent spat of these errors. Restarting middlewared with `systemctl restart middlewared` didn't seem to help.

Code:

Failed to check for alert Smartd:
Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/plugins/alert.py", line 808, in __run_source
    alerts = (await alert_source.check()) or []
              ^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/alert/base.py", line 335, in check
    return await self.middleware.run_in_thread(self.check_sync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1254, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/alert/source/smartd.py", line 22, in check_sync
    if not self.middleware.call_sync("service.started", "smartd"):
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1421, in call_sync
    return self.run_coroutine(methodobj(*prepared_call.args))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1461, in run_coroutine
    return fut.result()
           ^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 449, in result
    return self.__get_result()
           ^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/_base.py", line 401, in __get_result
    raise self._exception
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 177, in nf
    return await func(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/schema/processor.py", line 44, in nf
    res = await f(*args, **kwargs)
          ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service.py", line 201, in started
    state = await service_object.get_state()
            ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/base.py", line 37, in get_state
    return await self.middleware.run_in_thread(self._get_state_sync)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1254, in run_in_thread
    return await self.run_in_executor(self.thread_pool_executor, method, *args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1251, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.11/concurrent/futures/thread.py", line 58, in run
    result = self.fn(*self.args, **self.kwargs)
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/base.py", line 40, in _get_state_sync
    unit = self._get_systemd_unit()
           ^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/middlewared/plugins/service_/services/base.py", line 68, in _get_systemd_unit
    unit.load()
  File "/usr/lib/python3/dist-packages/pystemd/base.py", line 90, in load
    unit_xml = self.get_introspect_xml()
               ^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3/dist-packages/pystemd/base.py", line 75, in get_introspect_xml
    bus.call_method(
  File "pystemd/dbuslib.pyx", line 446, in pystemd.dbuslib.DBus.call_method
pystemd.dbusexc.DBusTimeoutError: [err -110]: b'Connection timed out'

 

[ctag]

Migrating to the new TrueCharts tt-rss app​

The truecharts tt-rss app is working again, after a little less than a year spent defunct. I had duct-taped a solution together in the meantime, so now the task is to bring the data over to the new app!
If you already have a database backup, and just want to restore: see the Truecharts app migration guide.

Creating an SQL dump​

The existing setup was running in the docker-compose app, so there will be two layers to getting at our SQL data. First is to get into the main app:

Code:

root@bns-citadel:/mnt/application/dockerfiles/ttrss-docker# k3s kubectl get pods -A | grep -i rss
ix-docker-compose-tt-rss           docker-compose-tt-rss-6bf4494964-q44lm                    1/1     Running            1 (22d ago)       22d
ix-docker-compose-tt-rss           docker-compose-tt-rss-9b4cd5c7f-bt594                     0/1     CrashLoopBackOff   5809 (116s ago)   22d

root@bns-citadel:/mnt/application/dockerfiles/ttrss-docker# k3s kubectl -n ix-docker-compose-tt-rss exec -it docker-compose-tt-rss-6bf4494964-q44lm /bin/bash

docker-compose-tt-rss-6bf4494964-q44lm:/# docker ps                                                              
CONTAINER ID   IMAGE                                    COMMAND                  CREATED        STATUS                 PORTS                                   NAMES                                                              
6b0071ac2d37   cthulhoo/ttrss-fpm-pgsql-static:latest   "/opt/tt-rss/updater…"   6 months ago   Up 3 weeks             9000/tcp                                ttrss-docker-updater-1
0d0a73af55d0   cthulhoo/ttrss-web-nginx:latest          "/docker-entrypoint.…"   6 months ago   Up 3 weeks (healthy)   0.0.0.0:8280->80/tcp, :::8280->80/tcp   ttrss-docker-web-nginx-1
d1337ae8c443   cthulhoo/ttrss-fpm-pgsql-static:latest   "/bin/sh -c ${SCRIPT…"   6 months ago   Up 3 weeks             9000/tcp                                ttrss-docker-app-1
5340a73a5ec0   cthulhoo/ttrss-fpm-pgsql-static:latest   "/opt/tt-rss/dcron.s…"   6 months ago   Up 3 weeks             9000/tcp                                ttrss-docker-backups-1
a146e8b797e7   postgres:12-alpine                       "docker-entrypoint.s…"   6 months ago   Up 3 weeks             5432/tcp                                ttrss-docker-db-1

docker-compose-tt-rss-6bf4494964-q44lm:/dockerfiles/ttrss-docker# docker exec -it a146e8b797e7 /bin/bash

a146e8b797e7:/sql-files# pg_dump -U $POSTGRES_USER tt-rss > tt-rss-backup-02_01_2024.sql

If I could go back and do this over, I would have used the -Fc flag on pg_dump to create a custom-format dump file. But once I have extracted that sql file and spun down the docker-compose app, it wouldn't come back when I tried restarting it.

Restoring the SQL file​

If I had the -Fc version of the sql dump, I could have used it with PgAdmin as in the migration guide linked above. Instead, I wound up having to copy the sql file into the new app, and then exec into it and restore manually:

Code:

root@bns-citadel:/mnt/application/dockerfiles/ttrss-docker# k3s kubectl cp sql-files/tt-rss-backup-02_01_2024.sql ix-tt-rss/tt-rss-updater-744896d97b-t5vml:/tmp/
root@bns-citadel:/mnt/application/dockerfiles/ttrss-docker# k3s kubectl exec -it -n ix-tt-rss tt-rss-updater-744896d97b-t5vml -- /bin/sh
/ # cat /tmp/tt-rss-backup-02_01_2024.sql | psql -U tt-rss -h $TTRSS_DB_HOST -d tt-rss