Register for the iXsystems Community to get an ad-free experience

AMD Ryzen build: An update after ~3 months

andrewzah

Cadet
Joined
Aug 11, 2019
Messages
9
Hello everyone, I made a post here a few months ago about a prospective AMD build. Now that I've been using the server for a while I thought I'd give an update!

Two things I'd like to note here:

1) I specifically did not go with a typical server rack because of space and noise requirements.
2) Going with a more modern setup (and DDR4 ram) made the project more expensive. Due to my requirements this was fine.
3) I have other requirements than simply using plex. I spin up multiple VMs for testing Ansible scripts, I have a setup to automatically compile Rust and Nix binaries whenever I push code to my repo, and I run multiple games like minecraft from time to time. Etc.

---

Resources that I used:
  • This community
  • The FreeNAS Manual
  • This github gist
    -> I learned a lot about how to structure a jail, and what iocage commands to use
  • The man pages for rc.d
    -> I had to write rc.d scripts for python, nodejs, and java services
---

I set up 6 drives, 4TB, in a zpool with 3 vdevs. Each vdev has a drive and a mirror.

I chose this setup because I plan on getting more drives in the future (I have a spare one now), and I didn't want long rebuilding times when a drive failed.

---

Services that I run in jails:
  • backblaze
    -> data backup using rclone
    -> app configs & data directories are mounted in the jail
  • bazarr
    -> grabs subtitles in multiple languages, hooks into radarr/sonarr
  • caddy
    -> reverse proxy to some of these services
    -> automatically handles certificate encryption with Let's Encrypt
  • calendar
    -> runs Radicale
  • kitana
    -> ui for plex plugins, iirc plex is deprecating its internal plugin menu/settings systems
  • mailbackup
    -> runs Offlineimap with Cron to get all of my mail
    -> I use NotMuch to browse/search the downloaded mail
  • node_exporter
    -> runs NodeExporter to feed server stats to Prometheus, which runs on my raspberry pi
    -> runs Caddy to host other generated metrics files for Prometheus
  • paperless
    -> digital document store, I'm still playing around with this
  • plex
    -> media catalogue + streaming to our apple tv
  • radarr
    -> monitors local shows for for bazarr subtitle grabbing
  • sonarr
    -> monitors local movies for for bazarr subtitle grabbing
  • syncthing
    -> seamlessly sync files between computers
    -> nextcloud was really annoying to set up, it broke when I did it manually -and- when I installed it as a plugin
  • thelounge
    -> modern irc client. I used to use znc+weechat, but I got tired of weechat's ux. thelounge is simple and pretty.
  • drone
    -> builds software automatically in docker containers and runs tests on them when I push code
  • mysql (a database)
    -> some services like matomo use this for backend storage
    -> a lot of minecraft plugins use mysql for backend storage
  • postgres (a database)
  • mc_main
    -> the main minecraft server instance, which is our Hub world.
    -> also runs a proxy (Waterfall, forked from Bungeecord) to let users connect to the other servers
  • mc_survival
    -> our main minecraft server, in survival mode
  • mc_creative
    -> our other mc server, in creative mode

If you're interested in the minecraft setup, I wrote about it here.

I also use RancherOS in a VM to run docker services, of which I have 5:
  • andrewzah/gollum
    -> a wiki with changes automatically tracked in Git
    -> I forked it to add user logins and automatic git pushes every 5 minutes
  • insekticid/docker-piwiki (matomo)
    -> self-hosted analytics (tracking my websites like andrewzah.com)
    -> respects requests to not track users who have DNT/etc enabled
  • radhifadlillah/shiori
    -> self-hosted website backup, similar to archive.web
  • huginn/huginn
    -> self-hosted, more powerful version of IFTTT
  • cwspear/docker-local-persist-volume-plugin
    -> allows local volume mounts in portainer/rancheros
  • portainer/portainer
---

My favorite services so far are Huginn and Syncthing. I was using Sync before, but it was quite slow. I keep personal files synced between my server, my main desktop, and my thinkpads.

Huginn is way more powerful than IFTTT. I set up email and pushover alerts for things like rainy weather, if any of my websites goes down, etc.

The only thing I -don't- use the server for is monitoring metrics, because if something happens to the server, then I can't view the metrics... So I put Prometheus and Grafana on a fresh install of raspbian, on my pi4. Now I feed it metrics from node_exporter and my minecraft instances:

2019-12-24_16-23.png


Node Exporter exports a fair bit of metrics, so I'm still figuring out dashboard items in grafana.

---

I haven't had any real issues with using AMD other than not being able to see disk temperatures with smartctl. I can see them in the GUI, but I needed a way to get those programmatically within a jail. So I wrote this script:

Code:
#!/bin/bash

# sets our output file and clears it
outfile=/root/temps/disk_temps.txt
echo '' > $outfile

# grabs our disk drives and cuts out nvd because
# it doesn't show temperature when queried
drives=$(sysctl kern.disks | sed 's/kern.disks: //' | sed 's/nvd0//')

# iterates over each drive (ada0, ada1, etc) and
# -appends- the output to our target file
for drive in $drives; do
  temp=$(smartctl -A -i -v 7,hex48 /dev/$drive | grep Temperature | awk '{print $10}')
  echo "freebsd_hdd_temp{drive=\"$drive\"} $temp" >> $outfile
done

# sample output:
# freebsd_hdd_temp{drive="ada5"} 30
# freebsd_hdd_temp{drive="ada4"} 30
# freebsd_hdd_temp{drive="ada3"} 30
# freebsd_hdd_temp{drive="ada2"} 28
# freebsd_hdd_temp{drive="ada1"} 32
# freebsd_hdd_temp{drive="ada0"} 30


I run it every 15 seconds with a staggered crontab:
Code:
* * * * * bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1
* * * * * sleep 15 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1
* * * * * sleep 30 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1
* * * * * sleep 45 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1


When I feed it into Grafana from Prometheus, it looks like so:

2019-12-24_16-13.png


Pretty nice, if I do say so myself.

---

My jail setup is pretty simple. I create configuration folders in /mnt/lily/apps/xxxx, and data folders in /mnt/lily/data/xxxx. So when I make a jail, it'll roughly look like this:

Code:
iocage create -n "jailname" -r 11.2-RELEASE \
  ip4_addr="vnet0|192.168.1.xxx" \
  defaultrouter="192.168.1.1" \
  boot="on" \
  vnet="on"

iocage exec jailname mkdir -p /config
iocage exec jailname mkdir -p /data

mkdir /mnt/lily/apps/jailname
mkdir /mnt/lily/data/jailname

iocage fstab -a jailname /mnt/lily/apps/jailname /config nullfs rw 0 0
iocage fstab -a jailname /mnt/lily/data/jailname /data nullfs rw 0 0

iocage console jailname
<proceed to set up jail>


I -should- have saved the commands into shell scripts for posterity, but oh well. Configuring jails is easy enough. With plex/bazarr I created a system user, and in the jails I created a user/group with the same ID to avoid r/w conflicts.

---

That's my build. I've been really happy with it. When I have more space, I'd like to build a proper server rack.

Let me know if you have any questions!

tagging @maniyer since they requested it
 

r0nski2000

Dabbler
Joined
Jun 20, 2017
Messages
20
Thanks for the update!
Would you happen to have any power draw measurements for the system?
 
Top