andrewzah
Cadet
- Joined
- Aug 11, 2019
- Messages
- 9
Hello everyone, I made a post here a few months ago about a prospective AMD build. Now that I've been using the server for a while I thought I'd give an update!
Two things I'd like to note here:
1) I specifically did not go with a typical server rack because of space and noise requirements.
2) Going with a more modern setup (and DDR4 ram) made the project more expensive. Due to my requirements this was fine.
3) I have other requirements than simply using plex. I spin up multiple VMs for testing Ansible scripts, I have a setup to automatically compile Rust and Nix binaries whenever I push code to my repo, and I run multiple games like minecraft from time to time. Etc.
---
Resources that I used:
I set up 6 drives, 4TB, in a zpool with 3 vdevs. Each vdev has a drive and a mirror.
I chose this setup because I plan on getting more drives in the future (I have a spare one now), and I didn't want long rebuilding times when a drive failed.
---
Services that I run in jails:
If you're interested in the minecraft setup, I wrote about it here.
I also use RancherOS in a VM to run docker services, of which I have 5:
My favorite services so far are Huginn and Syncthing. I was using Sync before, but it was quite slow. I keep personal files synced between my server, my main desktop, and my thinkpads.
Huginn is way more powerful than IFTTT. I set up email and pushover alerts for things like rainy weather, if any of my websites goes down, etc.
The only thing I -don't- use the server for is monitoring metrics, because if something happens to the server, then I can't view the metrics... So I put Prometheus and Grafana on a fresh install of raspbian, on my pi4. Now I feed it metrics from node_exporter and my minecraft instances:
Node Exporter exports a fair bit of metrics, so I'm still figuring out dashboard items in grafana.
---
I haven't had any real issues with using AMD other than not being able to see disk temperatures with smartctl. I can see them in the GUI, but I needed a way to get those programmatically within a jail. So I wrote this script:
I run it every 15 seconds with a staggered crontab:
When I feed it into Grafana from Prometheus, it looks like so:
Pretty nice, if I do say so myself.
---
My jail setup is pretty simple. I create configuration folders in /mnt/lily/apps/xxxx, and data folders in /mnt/lily/data/xxxx. So when I make a jail, it'll roughly look like this:
I -should- have saved the commands into shell scripts for posterity, but oh well. Configuring jails is easy enough. With plex/bazarr I created a system user, and in the jails I created a user/group with the same ID to avoid r/w conflicts.
---
That's my build. I've been really happy with it. When I have more space, I'd like to build a proper server rack.
Let me know if you have any questions!
tagging @maniyer since they requested it
Two things I'd like to note here:
1) I specifically did not go with a typical server rack because of space and noise requirements.
2) Going with a more modern setup (and DDR4 ram) made the project more expensive. Due to my requirements this was fine.
3) I have other requirements than simply using plex. I spin up multiple VMs for testing Ansible scripts, I have a setup to automatically compile Rust and Nix binaries whenever I push code to my repo, and I run multiple games like minecraft from time to time. Etc.
---
Resources that I used:
- This community
- The FreeNAS Manual
- This github gist
-> I learned a lot about how to structure a jail, and what iocage commands to use - The man pages for rc.d
-> I had to write rc.d scripts for python, nodejs, and java services
I set up 6 drives, 4TB, in a zpool with 3 vdevs. Each vdev has a drive and a mirror.
I chose this setup because I plan on getting more drives in the future (I have a spare one now), and I didn't want long rebuilding times when a drive failed.
---
Services that I run in jails:
- backblaze
-> data backup using rclone
-> app configs & data directories are mounted in the jail - bazarr
-> grabs subtitles in multiple languages, hooks into radarr/sonarr - caddy
-> reverse proxy to some of these services
-> automatically handles certificate encryption with Let's Encrypt - calendar
-> runs Radicale - kitana
-> ui for plex plugins, iirc plex is deprecating its internal plugin menu/settings systems - mailbackup
-> runs Offlineimap with Cron to get all of my mail
-> I use NotMuch to browse/search the downloaded mail - node_exporter
-> runs NodeExporter to feed server stats to Prometheus, which runs on my raspberry pi
-> runs Caddy to host other generated metrics files for Prometheus - paperless
-> digital document store, I'm still playing around with this - plex
-> media catalogue + streaming to our apple tv - radarr
-> monitors local shows for for bazarr subtitle grabbing - sonarr
-> monitors local movies for for bazarr subtitle grabbing - syncthing
-> seamlessly sync files between computers
-> nextcloud was really annoying to set up, it broke when I did it manually -and- when I installed it as a plugin - thelounge
-> modern irc client. I used to use znc+weechat, but I got tired of weechat's ux. thelounge is simple and pretty. - drone
-> builds software automatically in docker containers and runs tests on them when I push code - mysql (a database)
-> some services like matomo use this for backend storage
-> a lot of minecraft plugins use mysql for backend storage - postgres (a database)
- mc_main
-> the main minecraft server instance, which is our Hub world.
-> also runs a proxy (Waterfall, forked from Bungeecord) to let users connect to the other servers - mc_survival
-> our main minecraft server, in survival mode - mc_creative
-> our other mc server, in creative mode
If you're interested in the minecraft setup, I wrote about it here.
I also use RancherOS in a VM to run docker services, of which I have 5:
- andrewzah/gollum
-> a wiki with changes automatically tracked in Git
-> I forked it to add user logins and automatic git pushes every 5 minutes - insekticid/docker-piwiki (matomo)
-> self-hosted analytics (tracking my websites like andrewzah.com)
-> respects requests to not track users who have DNT/etc enabled - radhifadlillah/shiori
-> self-hosted website backup, similar to archive.web - huginn/huginn
-> self-hosted, more powerful version of IFTTT - cwspear/docker-local-persist-volume-plugin
-> allows local volume mounts in portainer/rancheros - portainer/portainer
My favorite services so far are Huginn and Syncthing. I was using Sync before, but it was quite slow. I keep personal files synced between my server, my main desktop, and my thinkpads.
Huginn is way more powerful than IFTTT. I set up email and pushover alerts for things like rainy weather, if any of my websites goes down, etc.
The only thing I -don't- use the server for is monitoring metrics, because if something happens to the server, then I can't view the metrics... So I put Prometheus and Grafana on a fresh install of raspbian, on my pi4. Now I feed it metrics from node_exporter and my minecraft instances:
Node Exporter exports a fair bit of metrics, so I'm still figuring out dashboard items in grafana.
---
I haven't had any real issues with using AMD other than not being able to see disk temperatures with smartctl. I can see them in the GUI, but I needed a way to get those programmatically within a jail. So I wrote this script:
Code:
#!/bin/bash # sets our output file and clears it outfile=/root/temps/disk_temps.txt echo '' > $outfile # grabs our disk drives and cuts out nvd because # it doesn't show temperature when queried drives=$(sysctl kern.disks | sed 's/kern.disks: //' | sed 's/nvd0//') # iterates over each drive (ada0, ada1, etc) and # -appends- the output to our target file for drive in $drives; do temp=$(smartctl -A -i -v 7,hex48 /dev/$drive | grep Temperature | awk '{print $10}') echo "freebsd_hdd_temp{drive=\"$drive\"} $temp" >> $outfile done # sample output: # freebsd_hdd_temp{drive="ada5"} 30 # freebsd_hdd_temp{drive="ada4"} 30 # freebsd_hdd_temp{drive="ada3"} 30 # freebsd_hdd_temp{drive="ada2"} 28 # freebsd_hdd_temp{drive="ada1"} 32 # freebsd_hdd_temp{drive="ada0"} 30
I run it every 15 seconds with a staggered crontab:
Code:
* * * * * bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1 * * * * * sleep 15 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1 * * * * * sleep 30 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1 * * * * * sleep 45 && bash /root/temps/disk_temps.sh >> /var/log/temps.log 2>&1
When I feed it into Grafana from Prometheus, it looks like so:
Pretty nice, if I do say so myself.
---
My jail setup is pretty simple. I create configuration folders in /mnt/lily/apps/xxxx, and data folders in /mnt/lily/data/xxxx. So when I make a jail, it'll roughly look like this:
Code:
iocage create -n "jailname" -r 11.2-RELEASE \ ip4_addr="vnet0|192.168.1.xxx" \ defaultrouter="192.168.1.1" \ boot="on" \ vnet="on" iocage exec jailname mkdir -p /config iocage exec jailname mkdir -p /data mkdir /mnt/lily/apps/jailname mkdir /mnt/lily/data/jailname iocage fstab -a jailname /mnt/lily/apps/jailname /config nullfs rw 0 0 iocage fstab -a jailname /mnt/lily/data/jailname /data nullfs rw 0 0 iocage console jailname <proceed to set up jail>
I -should- have saved the commands into shell scripts for posterity, but oh well. Configuring jails is easy enough. With plex/bazarr I created a system user, and in the jails I created a user/group with the same ID to avoid r/w conflicts.
---
That's my build. I've been really happy with it. When I have more space, I'd like to build a proper server rack.
Let me know if you have any questions!
tagging @maniyer since they requested it