My TrueNAS Scale build and automatization

[Daisuke]

I've been a FreeNAS/TrueNAS user since 2011, check my first setup. This is my 3rd build and I'm going to share my findings, so others can get an idea of the components used.

Server Specs
Picking the server base canvas was easy, I'm a firm believer that using true enterprise server hardware pays off in the long term. Not to mention it is way cheaper, versus building your own server. I've had great results with my 2nd Dell C2100 build and its excellent reliability, so I decided to purchase a Dell R720xd from buildmyserver.com with following specs:

• 2x E5-2620 v2 @ 2.10GHz CPU
• 16x 16GB 2Rx4 PC3-12800R DDR3-1600 ECC, for a total of 256GB RAM
• 2x 750W Platinum Power Supply
• 1x 2-Bay 2.5" Rear FlexBay, for SATA SSDs
• 1x 2 10GB RJ-45 + 2 1GB RJ-45 Network Daughter Card, for 10GB planned connectivity
• 1x PERC H710 Mini, flashed to LSI 9207i firmware
• 1x PERC H810 PCIe, flashed to LSI 9207e firmware
• 1x NVIDIA Tesla P4 GPU, for Plex transcoding
• 1x DRAC7 Enterprise License, from eBay

Show : Flashed PERCs

Flashed PERC cards output in BIOS:

I did a lot of research, related to server sellers and decided to go with buildmyserver.com. If you call them directly, they will definitely go an extra length to accommodate you by offering a better overall price and additional server customizations not available on their website. When I went to pickup the server in NYC, I was impressed by the warehouse size and how well they test the products you purchase from them.

Server Power Consumption
Picking CPUs and other server components with low power consumption was important, since I run my NAS 24/7, with no disk spin downs or other "power saving" settings which actually hurt overall server performance.

Show : Server Power Configuration

To lower the power consumption further, I changed the System BIOS Settings > System Profile Settings > System Profile to Performance Per Watt (OS) and iDRAC Settings > Thermal > Thermal Base Algorithm to Minimum power (Performance per watt optimized).



These changes resulted in a power consumption drop from 350 Watts average, to 225 Watts average (the spike you see is when I restarted the server, to take the BIOS screenshots):

Server Storage
I used CMR enterprise grade hard drives and consumer grade SSDs. I will probably upgrade to Samsung PM883 SSD series, in the future.

• 1x Samsung 870 EVO 250GB SATA SSD for Scale OS, connected to internal USB port with a SKL Tech SATA to USB connector
• 12x HGST Ultrastar He8 Helium (HUH728080ALE601) 8TB for default pool hosting media files
• 2x Samsung 870 EVO 500GB SATA SSD for software pool hosting Kubernetes applications

I linked the SKL Tech SATA to USB connector because it was the only connector working perfectly with the Raspberry Pi attached SSDs, detailed below.

JBOD Storage
I also did a lot of research, related to future storage expandability. Originally I was looking at Dell PowerVault and Compellent but decided to go with NetApp because their excellent reliability, low price and very low noise level. I was also pleasantly surprised to notice the DS4246 do not increase the noise level on R720xd, which is already very quiet.

• 2x NetApp DS4246 enclosure
• 12x HGST Ultrastar He8 Helium (HUH728080ALE601) 8TB for default pool expandability
• 1x SFF-8436 to SFF-8088 cable, connecting the external PERC H810 to first DS4246 enclosure
• 1x SFF-8436 to SFF-8436 cable, connecting the two DS4246 enclosures

Honestly, I decided to purchase two DS4246 enclosures because I'm convinced the prices will go up exponentially, once people know about their compatibility with TrueNAS. To give you an idea, I paid $230 USD for each enclosure, which included the caddies and interposers. I originally purchased also an extra set of 24 interposers for $45 on eBay, but decided to refund them based on findings how reliable NetApp products are.

Network Integration
I'm currently using Unifi products for my home network, eventually I will upgrade to 10GB. I also use the free Cloudflare service as frontend combined with several Raspberry Pi's (DietPi OS), allowing SSL encrypted world access to my homelab and local DNS servers.

• 1x UDM SE all-in-one router and security gateway
• 1x USW-Pro-24-Poe switch
• 1x Raspberry Pi 4B running Certbot + Nginx Proxy in upstream mode linked to a Cloudflare SSL domain, for easy access to all applications running on TrueNAS
• 2x Raspberry Pi 4B running Pihole + Unbound, for network-wide protection and local network ad-free experience or through Unifi's integrated VPN

I prefer using separate Raspberry Pi's, to isolate possible network failures that might arise during a TrueNAS upgrade, hardware failures, etc. If I run Traefik or Pihole on the server, my entire home network will not be operational during this time.

Scale Configuration
I use the default Scale settings, with no customizations. The only service configuration change I applied is the server signing = mandatory into SMB service auxiliary parameters. The ix-applications dataset is deployed into software pool, which has Auto TRIM pool option enabled due to SSDs usage.

Show : Docker Prune Configuration

I have a cron job pruning all Docker leftovers:

Angelfish release:

Code:

# cat /mnt/software/uranus/prune.tool
#!/usr/bin/env bash

if [ $EUID -gt 0 ]; then
    echo 'Run prune.tool as root.'
    exit 1
fi

docker container prune -f --filter 'until=5m'
docker image prune -af --filter 'until=1h'
docker system prune -af --volumes
docker volume prune -f

Bluefin release:

Code:

# cat /mnt/software/uranus/prune.tool
#!/usr/bin/env bash

if [ $EUID -gt 0 ]; then
    echo 'Run prune.tool as root.'
    exit 1
fi

midclt call container.prune '
    {
        "remove_stopped_containers": true,
        "remove_unused_images": true
    }
'

Applications Backup and Updates
I'm using the @truecharts applications instead of official ones, because of their consistency related to initial application settings. I like being welcomed by the same setup process, for any new application I decide to deploy. I set all application names to lower-case alphanumeric values to avoid possible backup issues, explained below (e.g. sabnzbd).

As we all know, there is no current application backup process implemented in Scale, due to the nature of Kubernetes. So I created my own backup system, using a shell script I run manually, for now. Basically, the script stops the application, copies the files from a legacy mount point, then triggers a git push to a private GitHub repository. The restore process is similar. I also created a script which allows me to update all TrueCharts applications, every night, through a cron job.

Show : Datasets Configuration

In order to implement the scripts, I created two datasets into software pool:

• opt - stores temporary files used by several applications (e.g. Sabnzbd)
• uranus - stores the application backups, linked to a private GitHub repo (named from my Scale server hostname, to avoid insanities)

Disclaimer
IN NO EVENT SHALL "MYSELF (Daisuke)" BE LIABLE TO ANY PARTY FOR DIRECT, INDIRECT, SPECIAL, INCIDENTAL, OR CONSEQUENTIAL DAMAGES, INCLUDING LOST PROFITS, ARISING OUT OF THE USE OF THIS SOFTWARE AND ITS DOCUMENTATION, EVEN IF "MYSELF (Daisuke)" HAVE BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGE. THE SOFTWARE AND ACCOMPANYING DOCUMENTATION, IF ANY, PROVIDED HEREUNDER IS PROVIDED "AS IS". "MYSELF (Daisuke)" HAVE NO OBLIGATION TO PROVIDE MAINTENANCE, SUPPORT, UPDATES, ENHANCEMENTS, OR MODIFICATIONS.

The scripts are attached, below. If there is enough interest, I will create a public GitHub repo, allowing scripts being imported as submodule. This will also allow collaborators to provide script enhancements for the community. What I provided here covers my homelab needs.

backup.tool
Is important you initially backup and restore a test application, just to get a gist of the overall process. Once you are comfortable and you validated your application is restored and operates as expected, then you can proceed backing/restoring up another application. Always validate your backup/restore process is successful, for each application. Kubernetes is very sensitive to volume permissions, etc.

Show : backup.tool Configuration

Execute all commands as root user. You need to create your GitHub repo and import into it a new SSH key used for passwordless connectivity, then you can clone your new repo into uranus dataset:

Code:

$ sudo -i
# pwd
/root
# ssh-keygen -t ed25519 -C 'youremail@example.com'
# ls -lh /root/.ssh
total 16K
-rw------- 1 root root  411 Jun 26 13:58 id_ed25519
-rw-r--r--  1 root root  103 Jun 26 13:58 id_ed25519.pub

With the SSH key created and imported into GitHub repo, you can now clone the repo locally. If you don't get any errors, that means you imported properly your locally created SSH key.

Code:

# cd /mnt/software
# git clone git@github.com:yourusername/uranus.git
# cd /mnt/software/uranus
# git status

Add your SSH keys and backup.tool file into repo (yes, it should not be an executable file):

Code:

# pwd
/mnt/software/uranus
# install -dm 0700 /mnt/software/uranus/root/.ssh
# cp -a /root/.ssh/* /mnt/software/uranus/root/.ssh/
# ls -lh /mnt/software/uranus/root/.ssh/id_ed25519*
-rw------- 1 root root  411 Jun 26 13:58 id_ed25519
-rw-r--r--  1 root root  103 Jun 26 13:58 id_ed25519.pub
# ls -lh backup.tool
-rw-r--r-- 1 root root 3.8K Jul 22 10:21 backup.tool

Configure .gitignore to avoid backing useless files, and push the changes:

Code:

# cat > /mnt/software/uranus/.gitignore << 'EOF'
.DS_Store
*.pid
logs/
EOF
# git push

Script Usage
Familiarize yourself with the script options, by running bash backup.tool --help.

To create a backup for sabnzbd application, run:

Code:

# pwd
/mnt/software/uranus
# bash backup.tool --help
# bash backup.tool -d uranus -p software -u apps -a sabnzbd

To restore an application backup from your local repo, run:

Code:

# pwd
/mnt/software/uranus
# bash backup.tool -d uranus -p software -u apps -a sabnzbd -s

You can create multiple backup versions, by using git branches.

update.tool (simple automation alternative to truetool)
To leverage an application API as example, the script presumes you have sabnzbd installed, amongst other TrueCharts applications. If you don't use this application, you can ignore any file customizations or simply edit the script to use another application API. Once the script is executed either manually or with a cron job, it will start updating all applications with updates available and if sabnzbd application has any downloads in progress, it will wait until they are finished, then update the application. The script also deletes any file leftovers present into /mnt/software/opt/downloads directories.

Show : update.tool Configuration

Update the domain and apikey settings, then copy update.tool file into repo (yes, it should not be an executable file):

Code:

# pwd
/mnt/software/uranus
# ls -lh update.tool
-rw-r--r-- 1 root root 1.2K Jul 22 11:38 update.tool
# git push

Script Usage
Manually test the update process:

Code:

# bash /mnt/software/uranus/update.tool

To automate the update process, create a cron job in Scale:

update.tool Update
I released a small update to the script, which fixes the updates failing for some TrueCharts apps.

Show : update.tool Update

Applied changes:

Code:

$ diff -Naur update-old.tool update.tool
--- update-old.tool    2022-12-02 09:42:15.000000000 -0500
+++ update.tool    2022-12-02 09:49:34.000000000 -0500
@@ -22,6 +22,7 @@
 mapfile -t apps < <(cli -m csv -c 'app chart_release query name,update_available' | grep true | cut -d ',' -f 1)
 if [ ${#apps[@]} -gt 0 ]; then
     for i in "${apps[@]}"; do
+        i="${i%%[[:cntrl:]]}"
         echo -n "Updating ${i}... "
         if [ "${i,,}" == "sabnzbd" ]; then
             while true; do
@@ -32,7 +33,7 @@
                 sleep 10
             done
         fi
-        cli -c "app chart_release upgrade release_name=$i" > /dev/null 2>&1
+        cli -c 'app chart_release upgrade release_name="'$i'"' > /dev/null 2>&1
         echo 'OK'
     done
 fi

Re-download the update.tool.zip package.

 

[indivision]

Next level system there. Is it built more to learn and enjoy the process? Or, do you have a personal usage for that much storage/HP?

A continuation of my pi-hole questions from the other thread:

I've ended up creating redundancy for pi-hole with only one pi by having one install on the pi and another as a TrueNAS app. It seems then that I should always be able to maintain one or the other without down-time. Only down side is that the app doesn't use unbound. But, I suppose that isn't so bad since its typically only fielding a few random queries here and there.

I did have some trouble with DietPi initially. When I first set it up to use Apache as the web-server pi-hole just would not work. It would receive DNS requests. But, then not respond to the clients. I re-installed and selected nginx as the web server, used all the same settings and it just worked.

The whole set-up seems to be working pretty well now. I did have one strange issue come up. At a few unpredictable times I got errors due to the router IP getting rate-limited by pihole (1000 queries within 60 seconds). These were isolated spikes in queries. So, not just regular traffic building up. To me, that suggests some kind of loop.

Any ideas on what that could be and how to avoid? Or, maybe a better question would be what your pi-hole+routerDNS+dnsmasq settings are in that redundancy setup?

 

[Daisuke]

A continuation of my pi-hole questions from the other thread

You should not ask questions into multiple threads, people will label this as highjacking threads. Stick with your thread, people will answer there.

 

[indivision]

You should not ask questions into multiple threads, people will label this as highjacking threads. Stick with your thread, people will answer there.

Actually, I moved the questions here for that reason. They were not really relevant to the other thread. But, they are relevant to the use of redundant pi-hole servers that you outline here. You also directly linked to this thread when I brought it up.

That said, don't worry about it if you don't want to clarify. I'll figure it out.

 

[Daisuke]

Actually, I moved the questions here for that reason.

I apologize, I realized that was not your thread.

Or, maybe a better question would be what your pi-hole+routerDNS+dnsmasq settings are in that redundancy setup?

I use default DietPi settings, with zero customizations. DietPi process installs for me Pihole and asks me if I want to install the complement Unbound pre-configured with all correct settings, which is exactly how I intended to use Pihole. The installer will take care of everything and provide at the end the IP address you will use in your router, as well the Pihole web interface address.

Next, all I did was adding the two Pi IP addresses (which are made static through Unifi DHCP server) into Unifi Network DNS interface and it made all ads vanish instantly for my entire network. DNS requests are round-robin, whatever is the first server being hit, it will serve the request. I don't have any sync between Pihole databases, I don't care about the stats being perfectly identical. I care about keeping things simple and having the ads blocked, while performing secure DNS requests.

Pihole Local DNS takes care of all my LAN records, so I don't have to use IP addresses. Gravity service updates the black/whitelists.

 

[indivision]

I apologize, I realized that was not your thread.

No problem. Thank you for these extra details!

I use default DietPi settings, with zero customizations. DietPi process installs for me Pihole and asks me if I want to install the complement Unbound pre-configured with all correct settings, which is exactly how I intended to use Pihole. The installer will take care of everything and provide at the end the IP address you will use in your router, as well the Pihole web interface address.

View attachment 58416

Interesting. The process was slightly different for me. I had to "browse" software to add and select both 93 pihole and 95 unbound. But, I think the result was the same as for you.

But, after this selection, it installed the dependencies like a web server. I had to choose from apache, nginx or lighttpd. You didn't have to make that choice?

Next, all I did was adding the two Pi IP addresses (which are made static through Unifi DHCP server) into Unifi Network DNS interface and it made all ads vanish instantly for my entire network.

I think my setup is also working like this. I had to make some adjustments for it to play nice with the router. But, error free now.

One thing that seems like it stopped working (was working at one point for me) is the ad image replace. It used to consistently show a placeholder graphic for ads. But, now it sometimes shows broken link messages where ads used to be. Which behavior are you getting for that?

 

[Daisuke]

You didn't have to make that choice?

No, DietPi will install Lighttpd, as is the recommended server by Pihole software.

Which behavior are you getting for that?

I have none of the issues you mention. That being said, this is getting totally off track of intended discussion, which is TrueNAS Scale. Best place for you to ask these related questions is DietPi forums.

 

[indivision]

No, DietPi will install Lighttpd, as is the recommended server by Pihole software.

Actually, the pihole installer recommended Apache. Which didn't work out of the box in my case. But, nginx did.

I have none of the issues you mention. That being said, this is getting totally off track of intended discussion, which is TrueNAS Scale. Best place for you to ask these related questions is DietPi forums.

Hm. I don't know. I feel like this information would be considered helpful/relevant to many users here. And you detailed it out yourself in the OP. But, I don't want to use your thread in a way you don't like. So...

 

[Daisuke]

I feel like this information would be considered helpful/relevant to many users here. And you detailed it out yourself in the OP.

I understand. Please note the OP has these details, so others get a better idea of my current Scale build and how it integrates with my homelab. This does not mean this thread is dedicated for troubleshooting my Unifi router or other non-related TrueNAS components. If you have questions related to my TrueNAS Scale build or scripts, you are welcome to ask them here, otherwise is best you ask for support to the related product forums or wiki. I replied to your questions out of courtesy, but Pihole setup on a Raspberry Pi has nothing to do with TrueNAS Scale, I'm sure you agree. Cheers.

 

[Daisuke]

I released a small update to update.tool script, which fixes the updates failing for some TrueCharts apps. Applied changes:

Code:

$ diff -Naur update-old.tool update.tool
--- update-old.tool    2022-12-02 09:42:15.000000000 -0500
+++ update.tool    2022-12-02 09:49:34.000000000 -0500
@@ -22,6 +22,7 @@
 mapfile -t apps < <(cli -m csv -c 'app chart_release query name,update_available' | grep true | cut -d ',' -f 1)
 if [ ${#apps[@]} -gt 0 ]; then
     for i in "${apps[@]}"; do
+        i="${i%%[[:cntrl:]]}"
         echo -n "Updating ${i}... "
         if [ "${i,,}" == "sabnzbd" ]; then
             while true; do
@@ -32,7 +33,7 @@
                 sleep 10
             done
         fi
-        cli -c "app chart_release upgrade release_name=$i" > /dev/null 2>&1
+        cli -c 'app chart_release upgrade release_name="'$i'"' > /dev/null 2>&1
         echo 'OK'
     done
 fi

Re-download the update.tool.zip package.

 

[Daisuke]

I upgraded successfully to Bluefin 22.12.0, without any issues. The only hiccup was the server was stuck at GRUB for 20 minutes, I fixed it with a full shutdown and start.

 

[jhp612]

@Daisuke really enjoyed this post and your build. Im about to build a TrueNas scale system and was wondering if you had more feedback now youre a few months in. A few questions I had

1. Would you still go with buildyourserver?
2. Are you happy with the R720xd or would you go with the R730xd or R740xd now?
3. Any specs that you would change (memory amount etc)? And why?
4. Where did you procure your ds4246 enclosures?
5. Any limitations you are aware of in those enclosures supporting large HDD?

Again appreciate your post!

 

[Daisuke]

Are you happy with the R720xd or would you go with the R730xd or R740xd now?

R730 and R740 are not fully supported by TrueNAS, that's the reason why I purchased a R720. Plus the savings can be passed on other components, memory, SSDs, drives, etc.

Any specs that you would change (memory amount etc)? And why?

I upgraded the SSDs to Samsung PM883.

Where did you procure your ds4246 enclosures?

eBay.

Any limitations you are aware of in those enclosures supporting large HDD?

I'm using 8TB drives and should not be any limitations for larger CMR drives. I purchased the interposers, for my SATA drives. They are not needed but adding them offer an additional channel and enables certain missing SAS features. @HoneyBadger helped with this, he could comment more.

Would you still go with buildyourserver?

It was the cheapest price I found at that time, you can negotiate the prices over phone. I plan to keep my R720 for at least 10 years.