Bluefin Recommended Settings and Optimizations

[Daisuke]

My Bluefin 22.12.0 upgrade from Angelfish 22.02.4 was flawless. I'm going to post all steps I took during my TrueNAS Scale Build upgrade, as well other improvements and optimizations I implemented, as reference for other users.

For related inquiries or questions involving formatted code, please post the output using [CODE]command[/CODE]. See more details into Linux Terminal Conventions section, below.

My Server Specs

• Dell R720xd with 12x HGST Ultrastar He8 Helium (HUH728080ALE601) 8TB for default pool expandability
• 2x E5-2620 v2 @ 2.10GHz CPU
• 16x 16GB 2Rx4 PC3-12800R DDR3-1600 ECC, for a total of 256GB RAM
• 1x PERC H710 Mini, flashed to LSI 9207i firmware
• 1x PERC H810 PCIe, flashed to LSI 9207e firmware
• 1x NVIDIA Tesla P4 GPU, for transcoding
• 2x Samsung 870 EVO 500GB SATA SSD for software pool hosting Kubernetes applications
• 1x NetApp DS4246 enclosure with 12x HGST Ultrastar He8 Helium (HUH728080ALE601) 8TB for default pool expandability

Linux Terminal Conventions
You will find various Linux terminal commands present into this guide. Since Scale is a Debian Linux based OS, it is recommended you become familiar with the Linux command-line environment and terminal. It will help you troubleshoot several issues, a lot easier.

Show : Linux Terminal Conventions

In a Linux command-line environment, there are two types of prompts at the beginning of an executed command:

• Dollar sign ($) means you are a normal non-priviledged user
• Hash (#) means you are the system administrator (root)

When an user posts into forums a command starting with the hash #, everyone understands that command was executed as root user:

Code:

$ ssh -l user uranus.lan
$ id
uid=3000(user) gid=3000(user) groups=3000(user),544(builtin_administrators),568(apps)
$ sudo su -
[sudo] password for user:
# id
uid=0(root) gid=0(root) groups=0(root),544(builtin_administrators)

When you post a terminal command into forums, it is important to use the [CODE]command[/CODE], for readability:

The code segment should contain both the executed command and outputted result:

Code:

$ sudo sg_scan -i /dev/sda
[sudo] password for user:
/dev/sda: scsi0 channel=0 id=0 lun=0 [em]
    ATA       HUH728080ALE601   0003 [rmb=0 cmdq=1 pqual=0 pdev=0x0]

In the above example, the command was executed as non-root user and escalated the execution as root user with sudo.

Pools and Datasets
It is strongly recommended to separate your media storage, from your applications storage. Also, is preferable that root defined pool dataset names follow the Linux standard naming conventions. Users familiar with FHS, will find this beneficial.

Show : Pools and Datasets Configuration

My pools and datasets setup:

• default pool, using RaidZ2 CMR disks:
  • media dataset used for SMB share access
    • center dataset used for storage of media related files and accessed by multiple Kubernetes applications
  • opt dataset used for shared data
    • backup dataset used for backup files

• software pool, using Mirror SSDs:
  • home dataset used for non-root user home directories
  • ix-applications dataset used for Kubernetes applications
  • opt dataset used for SMB share access
    • downloads dataset used for storage of temporary media related files and accessed by multiple Kubernetes applications
    • uranus dataset used for a private Github repo where automated application backups are stored

If you use SSDs for your pool, make sure you enable Auto TRIM pool option in Storage > ZFS Health:

TRIM is a command that helps Scale operating system know precisely which data blocks are no longer needed and can be deleted, or marked as free for rewriting, on SSDs. Whenever a delete command is issued, TRIM immediately wipes the pages or blocks where files are stored. Next time the operating system tries to write new data in that area, it does not have to wait first to delete it.

To take advantage of your SSDs speed, set the System Dataset Pool to software pool:

Datasets should have the recordsize set according your specific dataset usage. For example, if media files larger than 5MB are stored on a dataset, recordsize should be set to 1M:

On the other hand, a MySQL InnoDB database stored on a dataset will greatly benefit from a recordsize set to 16K. That is because InnoDB defaults to a 16KB page size, so most operations will be done in individual 16K chunks of data.

If recordsize is a lot smaller than the size of the typical storage operation within the dataset, fragmentation will be greatly increased, leading to unnecessary performance. If recordsize is much higher than the size of the typical storage operation within the dataset, latency will be greatly increased. See openzfs performance tuning, for reference.

To determine the correct recordsize, you can calculate the average file size for a specific dataset with this command:

Code:

# find /mnt/software/ix-applications -type f -ls | awk '{s+=$7} END {printf "%.0f\n", s/NR}' | numfmt --to=iec
43K

Note that changing the recordsize affects only newly created files, existing files are unaffected. Therefore, is important to move your data to a new dataset with the correct recordsize. See man zfs, for reference.

Pool Disks Swap Partition
In Bluefin, each pool storage device has a predefined Linux swap partition of 2GB, used mainly to allow the use of slightly-different disk capacities in a physical setup. For example, if a replacement disk capacity is slightly smaller than other pool disks, the swap partition can be used to allocate additional space to ZFS partition. Thank you @Arwen and @winnielinnie for the useful information.

Show : Pool Disks Swap Configuration

In certain cases, it might be necessary to change the default Linux swap partition size. Default setting in Bluefin:

Code:

# midclt call system.advanced.config | jq '.swapondrive'
2

Set the default Linux swap partition, to 4GB:

Code:

# midclt call system.advanced.update '{"swapondrive": 4}' | jq '.swapondrive'
4

Once the Linux swap partition size setting changed, you need to wipe each disk and resilver it.

In UI, go to Storage > Pool Name > Manage Devices and take the disk offline:

Login into shell as root user and wipe the disks signatures, this process will take only few seconds:

Code:

# wipefs -af /dev/sda

Back in UI, forcibly Replace the disk:

The replacement will trigger an automatic resilvering, this process will take several hours.

Pool Disks or Reporting DIF Warnings
Is really important to follow this detailed guide, in today's world, reliable data storage is the most valuable asset. Recent versions of Linux kernel properly detect badly formatted disks and warn you about it.

Non-Root Local User
In Bluefin, non-root local user UID starts with 3000, versus Angelfish UID 1000. It is recommended after Bluefin upgrade to delete old non-local users and re-create them with the new UID structure, to avoid future permission conflicts. Prior upgrade, make sure your root user has the password enabled into Angelfish UI. Once upgraded to Bluefin, create a non-root local user to use it as UI login. On a clean install, admin user is created with UID 950 and root user password is disabled.

Show : Non-Root Local User Configuration

Standard Linux user creation defines the same userid and groupid, then you can add additional groupids to your user. A standard Linux user also requires a /home directory defined, in your case /mnt/pool/dataset/username. Auxiliary groupid 41 is builtin_administrators and auxiliary groupid 96 is apps (needed for Plex), names showing as ids will be fixed in Bluefin 22.12.1.

Create a /mnt/pool/dataset/username dataset (e.g. /mnt/software/home/user), where your user private information will be stored, then add a new user with following settings:

Make sure you set a valid email address, needed for testing the email alerts (see below Email Alerts section). The new UID 3000 can be used into runAsUser setting for all Scale applications, see below Kubernetes Apps Security section.

You should also generate a ssh key, login into your server as user and run the following command:

Code:

$ pwd
/mnt/software/home/user

$ ssh-keygen -t ed25519 -C 'user@icloud.com'
Generating public/private ed25519 key pair.
Enter file in which to save the key (/mnt/software/home/user/.ssh/id_ed25519): [press Enter]
Enter passphrase (empty for no passphrase): [press Enter]
Enter same passphrase again: [press Enter]
Your identification has been saved in /mnt/software/home/user/.ssh/id_ed25519
Your public key has been saved in /mnt/software/home/user/.ssh/id_ed25519.pub
The key fingerprint is:
SHA256:kpfu/eLIwWkXyujcDQfUDX/FqAX0UoJUnabz8XR9esc user@icloud.com
The key's randomart image is:
+--[ED25519 256]--+
|         .o+=o.+.|
|         ..+ +*..|
|        . . +=o .|
|       o .  +o. =|
|      o S .  o *o|
|       B + .  o E|
|      . X o    ..|
|     o = O.      |
|      o =.+o.    |
+----[SHA256]-----+
$ mv .ssh/id_ed25519.pub .ssh/authorized_keys

Use your new id_ed25519 private key to access the server, the authorized_keys file content will automatically show into Authorized Keys UI field.

Next, recursively re-apply the correct user/group permissions to your /mnt/pool/dataset/username dataset:

Once upgraded to Bluefin, disable password for root user (see caveat mentioned below into Kubernetes Apps Shell):

Data Protection
Configure S.M.A.R.T self-tests to measure the storage devices health, predict possible device failures and provide notifications on unsafe values. S.M.A.R.T. (Self-Monitoring, Analysis, and Reporting Technology) is a supplementary component built into modern storage devices through which devices monitor, store, and analyze the health of their operation.

Show : S.M.A.R.T Tests Configuration

There are four types of self-tests that a device can execute, all safe to user data:

• SHORT - runs tests that have a high probability of detecting device problems
• LONG - same as SHORT check but with no time limit and with complete device surface examination
• CONVEYANCE - identifies if damage incurred during transportation of the device
• OFFLINE - runs an immediate offline Short self-test, in foreground

To configure a daily short SHORT self-test, go to Data Protection > S.M.A.R.T Tests:

It is recommended to run a daily SHORT self-test and a bi-monthly LONG self-test, on all your pool disks.

Email Alerts
It is extremely important to have functional mail alerts in Scale, in order to receive notifications of a degraded pool or other critical errors. I personally use the Apple iCloud SMTP servers. You cannot send iCloud emails without an application password with software other than Apple products, is a two-factor security feature.

Show : Email Alerts Configuration

Go to Alerts and select Email from Settings dropdown menu:

iCloud settings example:

The Username must be your true AppleID email account or the linked iCloud alias (which you cannot modify or delete). From Email can be any @icloud.com alias you can create, modify or delete, but must be part of your list of linked emails to main AppleID. For example, you can use user@icloud.com, which is an account alias you created into username@icloud.com AppleID account.

Next, go to Credentials > Local Users > root > Edit and add the email address you used into previous From Email field:

See also NAS-119145 related to wrong hostname, fixed in Bluefin 22.12-RC.1.

System Failed Jobs
In certain rare cases, you might experience certain cron jobs being stuck for days or other failed tasks:

Show : System Failed Jobs Fix

You can address the issue easy, by restarting the middlewared service:

Code:

# systemctl restart middlewared.service

Kubernetes Service
If you get an error message telling you the Kubernetes service is not running or Unable to connect to kubernetes cluster in Bluefin, go to Apps > Settings > Unset Pool. Reboot the server and Set/Choose Pool. Once the pool set, it can take up to 30 minutes until all services are reconfigured, please be patient.

Kubernetes Applications
Is recommended to use the TrueCharts applications, their development team is specialized in Kubernetes charts, the same way iXsystems team is specialized in Scale OS. Beside robust charts, the main advantage TrueCharts applications offer is an uniform installer process. You will find the same familiar settings in each application and the team offers stellar support on their Discord channel, if you have any questions. All related application settings reflected in this guide are taken from their apps, thank you @truecharts for your community support.

Show : Kubernetes Apps Configuration

Make sure your apps have the following storage types set accordingly.

Required default app storage:

• PVC, the simple type is deprecated, never change this type of storage, this is the main cause for application failures

Example of additional storage, in TrueCharts Plex application:

• Host Path set to /mnt/pool/dataset/child-dataset, in my example /mnt/default/media/center
• Mount Path set to internal container directory /media, standard Linux directory where media is stored

As per Kubernetes documentation, the hostPath volume must be mounted as Read Only. This will also prevent any SMB share mounted on /mnt/default/media dataset to change the file permissions inside containers.

List of directories present into internal /media container directory, which are also present inside /mnt/default/media/center dataset:

Code:

$ ls -lh /media
drwxrwxr-x   8 3000 kah   8 Mar  8  2022  Books
drwxrwxr-x   4 3000 kah   4 Aug  2 00:00  Courses
drwxrwxr-x   4 3000 kah  12 Oct 31 00:00  Documents
drwxrwxr-x   6 3000 kah   6 Dec 15 00:00  Photos
drwxrwxr-x  54 3000 kah  54 Dec 15 20:49  Trash
drwxrwxr-x   4 3000 kah   4 Dec 14 17:57 'TV Recordings'

Bluefin introduces a new non-root admin user, disabling the password for root user. This prohibits the access to application logs and shell, which will be fixed into a future Bluefin release. You can easily access to logs and shell through an ssh terminal, with root user. If you are a Kubernetes administrator, see the Kubernetes External Access section.

List all Kubernetes pods:

Code:

# k3s kubectl get pods -A

Access the TrueCharts Plex application logs:

Code:

# k3s kubectl get pods -n ix-plex
NAME                    READY   STATUS    RESTARTS   AGE
plex-7d47c9cc4f-dprph   1/1     Running   0          42h

# k3s kubectl logs plex-7d47c9cc4f-dprph -n ix-plex | tail -n 3
udc input failed with 1001
udc input failed with 1011
ERROR: Error decoding audio frame (Invalid data found when processing input)

Access the TrueCharts Plex application shell and execute a command, never perform any changes through shell:

Code:

# k3s kubectl exec -it plex-7d47c9cc4f-dprph -n ix-plex -- /bin/sh
$ df -ahT /tmp
Filesystem     Type   Size  Used Avail Use% Mounted on
tmpfs          tmpfs   32G  425M   32G   2% /tmp

Kubernetes Applications Security
To avoid any file permission issues while sharing the same hostPath with multiple applications and a SMB share, is recommended to set the runAsUser to the non-root local user you use to access the SMB share, in my case UID 3000:

You can always use the default apps (568) user as runAsUser, but you will end-up with mixed permissions, if you manually create a directory or file through a SMB share (logged as non-root local user), which will break your apps file access. Proceed, based on your own needs.

Show : App Permissions Fix

In certain cases, the runAsUser UID change can result in failure to properly apply the UID for all container files:

Container logs:

Code:

2022-12-19 19:15:27.197191+00:00 Permission denied: /config/Library/Application Support/Plex Media Server/Preferences.xml

With PVC mounted locally, old UID 1000 is still assigned, instead of UID 3000:

Code:

# ls -lh /mnt/software/tmp-plex
total 512
drwxrwsr-x 3 1000 apps 3 Aug  9 16:33 Library

You have the choice to delete the app and start fresh, or execute the script listed below, to fix the permissions.

Adjust the pool (where ix-application dataset belongs to), app and username script variables (where user should be the name of your UID 3000):

Code:

# id user
uid=3000(user) gid=3000(user) groups=3000(user),568(apps),544(builtin_administrators)

The script has all required failsafe checks in place and is set to use the apps (568) usergroup, which is the default runAsGroup setting into all apps. I recommend to never change the usergroup name, into your applications. Create the script as root:

Code:

cat > ./perms.tool << 'EOF'
#!/usr/bin/env bash

pool='software'
app='plex'
usergroup='apps'
username='user'

if [ $EUID -gt 0 ]; then
    echo 'Run perms.tool as root.'
    exit 1
fi
if [ ! -x "$(command -v k3s)" ]; then
    echo 'k3s is not installed.'
    exit 1
fi
if [ ! -x "$(command -v zfs)" ]; then
    echo 'zfs is not installed.'
    exit 1
fi
if [ ! -d /mnt/$pool/ix-applications ]; then
    echo 'Invalid pool name or ix-applications dataset not part of pool.'
    exit 1
fi
if [ "$(id -g $usergroup 2>/dev/null)" == "" ]; then
    echo 'Invalid user group.'
    exit 1
fi
if [ "$(id -u $username 2>/dev/null)" == "" ]; then
    echo 'Invalid user name.'
    exit 1
fi

volume="$(k3s kubectl get pvc -n ix-$app -o json | jq -r '.items[0].spec.volumeName')"
if [ "$volume" == "null" ]; then
    echo 'Invalid application name or not installed.'
    exit 1
fi
if [ -d /mnt/$pool/tmp-$app ]; then
    rm -rf /mnt/$pool/tmp-$app
fi
name="$pool/ix-applications/releases/$app/volumes/$volume"
zfs set mountpoint=/$pool/tmp-$app $name
if [ $? -ne 0 ]; then
    echo "Failed to set local mountpoint."
    exit 1
fi
echo -n "Applying $username:$usergroup permissions to $app application... "
find /mnt/$pool/tmp-$app -type d -exec chmod a+rwx,g-w,o-w,ugo-s,-t {} +
find /mnt/$pool/tmp-$app -type f -exec chmod a+rwx,u-x,g-wx,o-wx,ugo-s,-t {} +
chown -R $username:$usergroup /mnt/$pool/tmp-$app
echo 'OK'
zfs set mountpoint=legacy $name
if [ $? -ne 0 ]; then
    echo "Failed to set legacy mountpoint."
    exit 1
fi
rm -rf /mnt/$pool/tmp-$app
EOF

In UI, stop the application if is still showing as Deploying and execute the script:

Code:

# bash perms.tool
Applying user:apps permissions to plex application... OK

Start the application in UI.

Kubernetes Applications Shell
The UI allows you to have access to any application shell, however, you will be able to access the shell only if you login into UI with root user. This should be fixed into a future Bluefin release. Example of denied shell access into container, when logged into UI with a non-root user:

WARN[0000] Unable to read /etc/rancher/k3s/k3s.yaml, please start server with --write-kubeconfig-mode to modify kube config permissions error: error loading config file "/etc/rancher/k3s/k3s.yaml": open /etc/rancher/k3s/k3s.yaml: permission denied

Kubernetes Cluster External Access (for Advanced Users)
If you are an experienced Kubernetes cluster administrator, you can access the Scale Kubernetes cluster remotely and troubleshoot quite easy any issues you might encounter. Thank you @blacktide for the useful information.

Show : Kubernetes External Access Configuration

You can temporarily modify iptables, allowing the Kubernetes cluster to accept connections from a wider private subnet. My current iptables output:

Code:

# iptables -L INPUT -n --line-numbers
# Warning: iptables-legacy tables present, use iptables-legacy to see them
Chain INPUT (policy ACCEPT)
num  target     prot opt source               destination
1    KUBE-ROUTER-INPUT  all  --  0.0.0.0/0            0.0.0.0/0            /* kube-router netpol - 4IA2OSFRMVNDXBVV */
2    KUBE-ROUTER-SERVICES  all  --  0.0.0.0/0            0.0.0.0/0            /* handle traffic to IPVS service IPs in custom chain */ match-set kube-router-service-ips dst
3    KUBE-FIREWALL  all  --  0.0.0.0/0            0.0.0.0/0
4    ACCEPT     tcp  --  192.168.1.8          0.0.0.0/0            tcp dpt:6443 /* iX Custom Rule to allow access to k8s cluster from internal TrueNAS connections */
5    ACCEPT     tcp  --  127.0.0.1            0.0.0.0/0            tcp dpt:6443 /* iX Custom Rule to allow access to k8s cluster from internal TrueNAS connections */
6    DROP       tcp  --  0.0.0.0/0            0.0.0.0/0            tcp dpt:6443 /* iX Custom Rule to drop connection requests to k8s cluster from external sources */

Modify INPUT 4 (your line number might be different) to accept a wider private subnet, instead of just the assigned Scale 192.168.1.8 private IP address:

Code:

# iptables -R INPUT 4 -s 192.168.0.0/16 -j ACCEPT

Next, copy the /etc/rancher/k3s/k3s.yaml file contents to your local ~/.kube/config and change the localhost IP address, to your required private IP address:

Code:

$ sed -i 's|127.0.0.1|192.168.1.12|' ~/.kube/config

You can now access the Kubernetes cluster with Lens:

Rebooting the server will revert iptables to original configuration.

SMB Shares
New in Bluefin, shared hostPaths are considered insecure and are not recommended. That means, if you setup a dataset as SMB share and share that directory with multiple Kubernetes apps, you will get conflict warnings. This is actually a very good approach, for your data integrity and this is where dataset permissions play an important role.

iXsystems mentioned into their Release Notes: "As a last resort that can result in system and app instability, Host Path Safety Checks can be disabled in Apps > Settings > Advanced Settings." Do not disable that setting.

In simple terms, the hostPath validation prevents the use of same pool dataset for both apps and shares. From my understanding, there are a lot of apps changing file permissions without any warnings, or they simply don't work with SMB share ACLs. That's the main reason why you should not disable that setting and instead, re-organize your pool datasets properly. This also will be beneficial for future Bluefin upgrades. As a side note, @truecharts team do not offer storage related support for their applications, if you disable the hostPath validation.

Datasets Structure Organization
You can easily organize your pool datasets structure with zfs rename command, thank you @winnielinnie for the useful information.

Show : Datasets Structure Configuration

The goal is to move all data from media dataset:

Into media/center child dataset:

➊ Use zfs command as root user, to rename the dataset you plan to move into a child dataset:

Code:

# zfs rename default/media default/media-old

➋ Into UI, create a new media dataset:

➌ Rename the media-old dataset to center child dataset:

Code:

# zfs rename default/media-old default/media/center

The rename process is instant and recordsize setting will be preserved.

New dataset usage in TrueCharts Plex application, change only the hostPath and your Plex media files will not be affected in any way:

SMB Shares Configuration
Once your datasets structure is properly organized, proceed with the SMB shares configuration.

Show : SMB Shares Configuration

The main concern is having ix-applications dataset isolated, it should not be accessed by anyone except root user. Currently, the ix-applications dataset is allowed to be read by anyone and permissions cannot be changed. For example, a non-root local user should:

• not be able to access ix-applications dataset through an SMB share, see thread
• be able to read/write data using a hostPath volume assigned to a child dataset, accessible by multiple pods

Prior setting a new SMB share, configure the service:

• Enable Apple SMB2/3 Protocol Extensions, if you connect the shares to a Mac
• Set the Administrators Group to user UID 3000 you created earlier

I currently have two SMB shares set:

• media - mounted on media dataset, where center child dataset is used for shared files between multiple applications
• opt - mounted on opt dataset, where downloads child dataset is used for shared files between multiple applications

For any SMB connectivity issues, you can troubleshoot with following command:

Code:

# midclt call smb.status AUTH_LOG | jq

Organize your pool datasets, preferably following Linux standard naming conventions (read the Pools and Datasets section, for important dataset optimizations):

To avoid any hostPath conflicts, set your SMB share to /mnt/software/opt and your application hostPath to /mnt/software/opt/downloads. Assigning an SMB share directly to /mnt/software pool is not considered good practice.

To access your SMB shares, use the previously created UID 3000, which must have proper permissions to all your datasets (except ix-applications dataset, which should never be exposed or accessed by an end-user).

/mnt/software/opt/downloads dataset permissions, see documentation for setting POSIX ACLs:

You will not encounter any issues or warnings if you make sure the /mnt/software/opt SMB share is not directly accessed by multiple applications, the SMB share user matches the runAsUser UID and the dataset ACL Type is set to POSIX (default option), which is supported by Linux, Mac and Windows:

Changing the dataset ACL Type to SMB/NFSv4 will result in breaking the file permissions and cannot be reversed, as detailed into warning:

As per Kubernetes documentation, the hostPath volume must be mounted as Read Only. Since we are using POSIX ACLs and a single non-root local user controls the SMB share, as well all applications, an exception can be made. Example of /mnt/software/opt/downloads dataset shared with multiple applications who access the internal pod /opt/downloads directory to read/write data:

Example of re-used internal directory, into an application:

Plex Transcoding
Media that is incompatible with your device will be transcoded to a playable format. If you own a Plex compatible GPU, you can turn on Hardware-Accelerated Streaming in Plex Media Server. It is strongly recommended to create a Memory medium storage type, used specifically for transcoding.

Show : Plex Transcoding Configuration

Using a Memory medium will significantly improve the transcoding performance. In your TrueCharts Plex application, add:

In Plex Media Server, go to Settings > Transcoder and set the temporary directory:

Uninterruptible Power Supply
It is recommended to use an UPS, in order to avoid brief power failures, which will abruptly shutdown your server. These interruptions are known to corrupt your pool data, therefore is worth the investment.

Show : UPS Configuration

Identify the UPS model and driver:

Code:

# nut-scanner
Neon library not found. XML search disabled.
Scanning USB bus.
No start IP, skipping SNMP
No start IP, skipping NUT bus (old connect method)
Scanning IPMI bus.
[nutdev1]
    driver = "usbhid-ups"
    port = "auto"
    vendorid = "051D"
    productid = "0002"
    product = "Back-UPS NS 1500M2 FW:957.e4 .D USB FW:e4"
    serial = "5B2150T61261"
    vendor = "American Power Conversion"
    bus = "001"
[nutdev2]
    driver = "nut-ipmipsu"
    port = "id1"
[nutdev3]
    driver = "nut-ipmipsu"
    port = "id2"

In my case, the scanner detected an UPS unit connected to USB port. The only important information you will need is the driver usbhid-ups, the product name might vary. Into UPS service, configure the settings as listed below and pay attention to:

• Driver - match the found driver, in our example usbhid-ups
• Port - match the found port, in our example auto
• Monitor Password - set a random complex password, you will not use it as end-user

If you configured correctly your email alerts, in case of a power failure, you will receive an alert similar to:

 

[danb35]

I'm seeing a lot of "it is strongly recommended" here--by whom? Is this just a matter of your own opinion (and if so, on what basis), do these reflect things iX recommends (and if so, where, because they don't appear in the release notes), or do they come from somewhere else (and again, if so, where)?

Just for one example: you recommend to create a non-root local user, and using that user to admin the web UI. Is this really what iX now wants? Because if so, it represents a complete 180 on something people have been asking for, very frequently, for many years, and to which their answer has always been a hard no. As recently as last year, their answer was that if you wanted to log in as someone other than root, you should use TrueCommand.

I appreciate the obvious work you've put into this, but a lot of these recommendations represent a radical departure from what iX have consistently recommended--for many years in the product now known as TrueNAS CORE, and for as long as it's been around in the case of SCALE.

 

[winnielinnie]

Just for one example: you recommend to create a non-root local user, and using that user to admin the web UI. Is this really what iX now wants? Because if so, it represents a complete 180 on something people have been asking for, very frequently, for many years, and to which their answer has always been a hard no. As recently as last year, their answer was that if you wanted to log in as someone other than root, you should use TrueCommand.

I remember.

Why does the login page even require a "username"?

If one of them is malicious, it's a matter of who can change the root password first, or who can revoke access to the other three users first. But if one of them is fired, or leaves of his own accord, his account can be terminated without affecting the others (and I think it's common practice...

www.truenas.com

 

[Daisuke]

Just for one example: you recommend to create a non-root local user, and using that user to admin the web UI. Is this really what iX now wants?

There is definitely change related, see post. Also, these are my recommendations, I don’t work for iX and nobody is forcing you to apply any listed suggestions. From my perspective, is common sense to have the same non-root local user login to either UI or ssh with a key to server. Also, many people have no idea how to properly set an user securely, so is beneficial to learn something new.

I appreciate the obvious work you've put into this, but a lot of these recommendations represent a radical departure from what iX have consistently recommended

I honestly think you nickel and dime for no reason. Most of details I explained are related to features little known to users and specific to Scale. Or troubleshooting issues never present prior Bluefin. And for the two times I mention strongly recommended, it is. Running Kubernetes on SSDs is beneficial for several aspects, not just faster deployments. And Plex needs a ramdisk for smooth transcoding.

 

[mgoulet65]

I created a non-root user in anticipation of upgrading to Bluefin once we get the next dot release. Is it the case that this user WILL NOT work for UI login under Angelfish?

 

[Daisuke]

Is it the case that this user WILL NOT work for UI login under Angelfish?

I already have upgraded to Bluefin, when I wrote the guide. What happens when you try to login into Angelfish UI with the non-root local user? I believe you cannot, this was enabled in Bluefin. Your user should have assigned builtin_administrators auxiliary usergroup.

 

[mgoulet65]

I already have upgraded to Bluefin, when I wrote the guide. What happens when you try to login into Angelfish UI with the non-root local user? I believe you cannot, this was enabled in Bluefin. Your user should have assigned builtin_administrators auxiliary usergroup.

It says the credentials are invalid, which fits with what you are saying. I was just doing a sanity check on myself. Thx.

 

[Daisuke]

I decided to perform a Bluefin 22.12.0 clean install, just to compare everything with the previous upgrade from Angelfish 22.02.4. I'm personally happy I did a clean install, to make sure everything works as intended for future upgrades. If I find other changes, I will update this post. See below the found differences.

General Settings
I cannot assign anymore port 8080 as HTTP port, which was previously defined in Angelfish and ported properly in Bluefin upgrade. When I enter the new port, I get an error that is being used by Webdav Service and the setting is reverted back to port 80. I'm wondering how was running that service on previous Bluefin upgrade?

Network
A new Inherit domain from DHCP option is now present, not sure if this was showing also into previous upgrade, as I never checked. I have this option now enabled, since my Unifi UDM SE router provides this information.

Users
Two new users are created with the clean install, admin and media. As explained earlier, new non-root local users now start with UID 3000, see new user I created:

New admin user has the wrong permissions and cannot be changed, should be identical to root permissions, see NAS-119570:

root user permissions are OK, but the user has now the UID 41, instead of previously upgraded UID 93:

New media user has a nologin shell, with adjustable permissions, see NAS-119571. No idea what it serves for, I will not delete it for now. Confirmed with iXsystems, this user can be safely deleted, this will be fixed in a new release.

The ssh MOTD changed on clean install, versus upgraded Bluefin:

Code:

$ sudo -i
[sudo] password for user:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI, CLI, and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

Pools
I imported without issues the two pools, all settings were unchanged.

Kubernetes Apps
I restored all apps from my backups, without issues.

 

[morganL]

I'm seeing a lot of "it is strongly recommended" here--by whom? Is this just a matter of your own opinion (and if so, on what basis), do these reflect things iX recommends (and if so, where, because they don't appear in the release notes), or do they come from somewhere else (and again, if so, where)?

Just for one example: you recommend to create a non-root local user, and using that user to admin the web UI. Is this really what iX now wants? Because if so, it represents a complete 180 on something people have been asking for, very frequently, for many years, and to which their answer has always been a hard no. As recently as last year, their answer was that if you wanted to log in as someone other than root, you should use TrueCommand.

I appreciate the obvious work you've put into this, but a lot of these recommendations represent a radical departure from what iX have consistently recommended--for many years in the product now known as TrueNAS CORE, and for as long as it's been around in the case of SCALE.

For some of these issues, iX doesn't have strong recommendations, we are just providing the tools. Different users will have different objectives.

Providing the option for non-root user is important for environments with multiple administrators.. and provides a bit of a safety belt to minimize accidents. We could not do it easily with FreeBSD, but have found a path with linux. We are not requiring it.

Separating pools for Apps and media make sense for larger systems with more budget.... but reality is that many users have 2-5 drives. That separation is not feasible or worthwhile.

From iX's perspective, we see the Community as the leads in developing and recommending good deployment practices. If something is critical to reliability, we will make our position known, but otherwise, we appreciate that flexibility is important. Lets be aware that many users have a tight budget.....

 

[Daisuke]

Separating pools for Apps and media make sense for larger systems with more budget

First, thank you for the added clarifications. If your system does not have the space to add two SSDs, yes. Purchasing a pair of Samsung 128Gb SSDs is like $60-$70. Worth every penny to have an additional fast pool for Scale.

The only clarification I would like to have from iX Systems is how do we treat the pool upgrades, Example for boot-pool:

Code:

# zpool status boot-pool
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      sdo3      ONLINE       0     0     0

By default, boot-pool is not upgraded on a clean Bluefin 22.12.0 install. Is it recommended to upgrade that pool? If not, can you please explain why? Pool upgrades are one way, you cannot downgrade. Since 2011 when I started using FreeNAS/TrueNAS, I always upgraded all pools, without any incidents. Once I get an official answer, I will update the OP with this information.

Also, why boot-pool is using uuid, instead of partuuid? I would prefer to have it converted to a partuuid, following the standards. uuid is not always visible, specially in VM envs.

 

[morganL]

First, thank you for the added clarifications. If your system does not have the space to add two SSDs, yes. Purchasing a pair of Samsung 128Gb SSDs is like $60-$70. Worth every penny to have an additional fast pool for Scale.

The only clarification I would like to have from iX Systems is how do we treat the pool upgrades, Example for boot-pool:

Code:

# zpool status boot-pool
  pool: boot-pool
 state: ONLINE
status: Some supported and requested features are not enabled on the pool.
    The pool can still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
    the pool may no longer be accessible by software that does not support
    the features. See zpool-features(7) for details.
config:

    NAME        STATE     READ WRITE CKSUM
    boot-pool   ONLINE       0     0     0
      sdo3      ONLINE       0     0     0

By default, boot-pool is not upgraded on a clean Bluefin 22.12.0 install. Is it recommended to upgrade that pool? If not, can you please explain why? Pool upgrades are one way, you cannot downgrade. Since 2011 when I started using FreeNAS/TrueNAS, I always upgraded all pools, without any incidents. Once I get an official answer, I will update the OP with this information.

Also, why boot-pool is using uuid, instead of partuuid? I would prefer to have it converted to a partuuid, following the standards. uuid is not always visible, specially in VM envs.

Because pool upgrades are one way, we don't recommend upgrading them until either (or preferably both):

1. Its clear that all required functions are working for the new release. System is stable.
2. new pool functionality is required for the use case

 

[Daisuke]

we don't recommend upgrading them

Thank you Morgan, when point 1 applies? I presume the boot-pool is automatically upgraded with a new Scale software release.

 

[Daisuke]

Several sections of the guide have been revamped, including the iX Systems recommended approach to use properly use SMB shares with datasets shared by multiple applications, which is no. 1 issue with certain pools and datasets configurations. I also added into Pools and Datasets section my research done related to recordsize, which I'm sure many users ignore.

 

[mistermanko]

Thank you for the write-up. There is lots of useful information here one wouldn't happen to find in - let's say - the official documentation maybe.

 

[Daisuke]

I updated the guide:

• added into Polls and Datasets section a command that allows you easily determine the correct recordsize for a dataset
• added Pool Disks Swap Partition section with details how to change the swapondrive setting
• added into Kubernetes Apps section how to access the logs and shell from a terminal, with root password disabled, you cannot do this anymore from UI (will be fixed in a future Bluefin release)
• added Data Protection section with S.M.A.R.T self-tests configuration

 

[avggeek]

I want to mention that the screenshot provided for Kubernetes App Security where one can specify the RunAs user etc does not seem to be accurate for most (if not all) the apps available via the Official Catalog.

I upgraded from Angelfish to Bluefin and the "recommendation" seems to be to delete that User (with UID 1000) and create a new User (with UID 3000). It's not clear to me in this scenario, what happens to apps deployed via the official catalog. Will they stop working once the UID 1000 user disappears? Do I need to redeploy the apps?

I did some more checks and I believe that if one is upgrading from Angelfish to Bluefin, the situation is as follows:

• In Angelfish, the only option for the UI was to login as root.
• Thus when you try to install an app for the first time (say from the official catalog), you are prompted to select a pool to create the ix-applications dataset. It appears as though the dataset thus gets created under the root account(?) {This is my guess as I have upgraded to Bluefin now and have no way to confirm]
• The Plex app itself seems to be running under UID 1000.

Given that the official Plex app does not have "RunAs" parameters or indeed Storage UID/Storage GID (which some official apps seem to provide), what's the recommendation here?

I also want to mention that the suggestion to create a emptyDir Memory path for Plex transcoding is possibly based on a TrueCharts deployment? I certainly do not see any such option with the official catalog Plex app. It would be good to be more explicit - especially when you mention "the rules should apply to official apps" as well in the Kube Apps section, which would lead one to think this is some sort of standard option in TrueNAS Scale.

 

[Daisuke]

I want to mention that the screenshot provided for Kubernetes App Security where one can specify the RunAs user etc does not seem to be accurate for most (if not all) the apps available via the Official Catalog.

Is mentioned into guide that TrueCharts apps are used. @truecharts are truly specialized in apps, while iXsystems are specialized in building the OS. I strongly recommended anyone to use them, instead of official apps.

I’ll update the section based on your details, thank you. I never installed the official apps, for the simple reason TrueCharts apps have an identical installer process for all their apps, which is not the case with official apps.

Edit: I updated the Kubernetes Applications section intro, with the correct information.

 

[Daisuke]

I updated the SMB Shares section with information on how to easily organize your pool datasets structure with zfs rename command. I temporarily removed the Plex Transcoding section, until @wsoteros increases the message capacity.

 

[jace92]

Hi @Daisuke! Thank you for the very informative write up of yours! I did read through what you were saying about the SMB Shares/hostPath issue but I have a question, though. As someone who doesn't have a solid grasp on Linux yet, exactly what your reasonings were, and the fixes to implement for avoiding the issues mentioned, would you be able to "dumb down" a little on why it's bad to disable the hostPath safety checks? I too was having the same issue with Synching on an SMB share where it was just stuck on "Deploying" and disabling the checks instantly allowed it to work. I also have my data a certain way and having to reorganize something would be very difficult!

Thanks again for everyone's chime-ins!

 

[wheelerx]

I was trying to send a very similar response yesterday, but I keep getting an error when I try to reply. I am also a newbie and don't have a clue how to run your scripts as instructed. so dumbing it way down with with some additional instructions would be helpful. I threw in the towel for now and disabled the SMB share which solve the issue, with obvious downsides