AMD Ryzen build: An update after ~3 months

[andrewzah]

Hello everyone, I made a post here a few months ago about a prospective AMD build. Now that I've been using the server for a while I thought I'd give an update!

Two things I'd like to note here:

1) I specifically did not go with a typical server rack because of space and noise requirements.
2) Going with a more modern setup (and DDR4 ram) made the project more expensive. Due to my requirements this was fine.
3) I have other requirements than simply using plex. I spin up multiple VMs for testing Ansible scripts, I have a setup to automatically compile Rust and Nix binaries whenever I push code to my repo, and I run multiple games like minecraft from time to time. Etc.

---

Resources that I used:

• This community
• The FreeNAS Manual
• This github gist
  -> I learned a lot about how to structure a jail, and what iocage commands to use
• The man pages for rc.d
  -> I had to write rc.d scripts for python, nodejs, and java services

---

I set up 6 drives, 4TB, in a zpool with 3 vdevs. Each vdev has a drive and a mirror.

I chose this setup because I plan on getting more drives in the future (I have a spare one now), and I didn't want long rebuilding times when a drive failed.

---

Services that I run in jails:

• backblaze
  -> data backup using rclone
  -> app configs & data directories are mounted in the jail
• bazarr
  -> grabs subtitles in multiple languages, hooks into radarr/sonarr
• caddy
  -> reverse proxy to some of these services
  -> automatically handles certificate encryption with Let's Encrypt
• calendar
  -> runs Radicale
• kitana
  -> ui for plex plugins, iirc plex is deprecating its internal plugin menu/settings systems
• mailbackup
  -> runs Offlineimap with Cron to get all of my mail
  -> I use NotMuch to browse/search the downloaded mail
• node_exporter
  -> runs NodeExporter to feed server stats to Prometheus, which runs on my raspberry pi
  -> runs Caddy to host other generated metrics files for Prometheus
• paperless
  -> digital document store, I'm still playing around with this
• plex
  -> media catalogue + streaming to our apple tv
• radarr
  -> monitors local shows for for bazarr subtitle grabbing
• sonarr
  -> monitors local movies for for bazarr subtitle grabbing
• syncthing
  -> seamlessly sync files between computers
  -> nextcloud was really annoying to set up, it broke when I did it manually -and- when I installed it as a plugin
• thelounge
  -> modern irc client. I used to use znc+weechat, but I got tired of weechat's ux. thelounge is simple and pretty.
• drone
  -> builds software automatically in docker containers and runs tests on them when I push code
• mysql (a database)
  -> some services like matomo use this for backend storage
  -> a lot of minecraft plugins use mysql for backend storage
• postgres (a database)
• mc_main
  -> the main minecraft server instance, which is our Hub world.
  -> also runs a proxy (Waterfall, forked from Bungeecord) to let users connect to the other servers
• mc_survival
  -> our main minecraft server, in survival mode
• mc_creative
  -> our other mc server, in creative mode

If you're interested in the minecraft setup, I wrote about it here.

I also use RancherOS in a VM to run docker services, of which I have 5:

• andrewzah/gollum
  -> a wiki with changes automatically tracked in Git
  -> I forked it to add user logins and automatic git pushes every 5 minutes
• insekticid/docker-piwiki (matomo)
  -> self-hosted analytics (tracking my websites like andrewzah.com)
  -> respects requests to not track users who have DNT/etc enabled
• radhifadlillah/shiori
  -> self-hosted website backup, similar to archive.web
• huginn/huginn
  -> self-hosted, more powerful version of IFTTT
• cwspear/docker-local-persist-volume-plugin
  -> allows local volume mounts in portainer/rancheros
• portainer/portainer

---

My favorite services so far are Huginn and Syncthing. I was using Sync before, but it was quite slow. I keep personal files synced between my server, my main desktop, and my thinkpads.

Huginn is way more powerful than IFTTT. I set up email and pushover alerts for things like rainy weather, if any of my websites goes down, etc.

The only thing I -don't- use the server for is monitoring metrics, because if something happens to the server, then I can't view the metrics... So I put Prometheus and Grafana on a fresh install of raspbian, on my pi4. Now I feed it metrics from node_exporter and my minecraft instances:

Node Exporter exports a fair bit of metrics, so I'm still figuring out dashboard items in grafana.

---

I haven't had any real issues with using AMD other than not being able to see disk temperatures with smartctl. I can see them in the GUI, but I needed a way to get those programmatically within a jail. So I wrote this script:

Code:

#!/bin/bash

# sets our output file and clears it
outfile=/root/temps/disk_temps.txt
echo '' > $outfile

# grabs our disk drives and cuts out nvd because
# it doesn't show temperature when queried
drives=$(sysctl kern.disks | sed 's/kern.disks: //' | sed 's/nvd0//')

# iterates over each drive (ada0, ada1, etc) and
# -appends- the output to our target file
for drive in $drives; do
  temp=$(smartctl -A -i -v 7,hex48 /dev/$drive | grep Temperature | awk '{print $10}')
  echo "freebsd_hdd_temp{drive=\"$drive\"} $temp" >> $outfile
done

# sample output:
# freebsd_hdd_temp{drive="ada5"} 30
# freebsd_hdd_temp{drive="ada4"} 30
# freebsd_hdd_temp{drive="ada3"} 30
# freebsd_hdd_temp{drive="ada2"} 28
# freebsd_hdd_temp{drive="ada1"} 32
# freebsd_hdd_temp{drive="ada0"} 30

I run it every 15 seconds with a staggered crontab:

Code:

* * * * * bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1
* * * * * sleep 15 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1
* * * * * sleep 30 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1
* * * * * sleep 45 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1

When I feed it into Grafana from Prometheus, it looks like so:

Pretty nice, if I do say so myself.

---

My jail setup is pretty simple. I create configuration folders in /mnt/lily/apps/xxxx, and data folders in /mnt/lily/data/xxxx. So when I make a jail, it'll roughly look like this:

Code:

iocage create -n "jailname" -r 11.2-RELEASE \
  ip4_addr="vnet0|192.168.1.xxx" \
  defaultrouter="192.168.1.1" \
  boot="on" \
  vnet="on"

iocage exec jailname mkdir -p /config
iocage exec jailname mkdir -p /data

mkdir /mnt/lily/apps/jailname
mkdir /mnt/lily/data/jailname

iocage fstab -a jailname /mnt/lily/apps/jailname /config nullfs rw 0 0
iocage fstab -a jailname /mnt/lily/data/jailname /data nullfs rw 0 0

iocage console jailname
<proceed to set up jail>

I -should- have saved the commands into shell scripts for posterity, but oh well. Configuring jails is easy enough. With plex/bazarr I created a system user, and in the jails I created a user/group with the same ID to avoid r/w conflicts.

---

That's my build. I've been really happy with it. When I have more space, I'd like to build a proper server rack.

Let me know if you have any questions!

tagging @maniyer since they requested it

 

[r0nski2000]

Thanks for the update!
Would you happen to have any power draw measurements for the system?