SOLVED Hardened backup repository with TrueNAS

blanchet

Guru
Joined
Apr 17, 2018
Messages
515
Update:
The latest version of this guide is available in the User recommendation section of the TrueNAS SCALE documentation.

Abstract
Veeam has published a guide to make a hardened repository that will survive to any remote attack. It is basically a Linux server that operates without remote management. Starting from this idea, I would like to propose a variant, based on TrueNAS and compatible with any backup software. My idea relies on ZFS snapshots to guarantee data immutability and disabling all the management interfaces to avoid someone delete the snapshots after stealing my credentials.

Note:
In this guide, I will use TrueNAS Scale, but it can be easily adapted with TrueNAS Core. Nevertheless, TrueNAS Scale has a slight advantage over TrueNAS Core because it has a more powerful command line interface. that will avoid restarting the webui to modify the configuration.
In the same way, I use Veeam as an example for the backup account, but you can use any other backup software that is compatible with CIFS/SMB.

Installation
  • Install TrueNAS Scale 22.02RC1.2
  • Create a ZFS pool
  • Configure mail notification (very important because it will be the only way to know that something goes wrong)
    • Click on the bell icon on the top right corner
    • Click on the gear, select Email
    • Fill the web form
  • Create a local group for the backup software (Credentials | Local Groups)
    • Add Group
      • Group ID: 10000
      • Group Name: veeam
      • [ ] Permit Sudo
      • [x] Samba authentication
      • [ ] Allow Duplicated GIDs
  • Create a user with a strong password for the backup software (Credentials | Local Users)
    • Add User
      • Full Name: Veeam B&R
      • Username: veeam
      • Password: <use a strong password>
      • Password confirmation:
      • Email: blank
      • User ID: 10000
      • [ ] new primary group
      • Primary Group: veeam
      • Auxiliary group: empty
      • Create Home Directory in: /nonexistent
      • Home directory permission: clear all, except user permissions
      • SSH Public key: blank
      • Disable password: no
      • Shell: nologin
      • [ ] lock user
      • [ ] permit sudo
      • [ ] microsoft account
      • [x] samba authentication
  • Create a dataset for the backup data
Code:
zfs create tank1/veeam
zfs set compression=off org.freenas:description="veeam hardened repo" tank1/veeam
chown veeam:veeam /mnt/tank1/veeam
chmod 700 /mnt/tank1/veeam

If the dataset is empty, you can create an empty snapshot and lock it to prevent deleting it by mistake the dataset from the webui or with the zfs destroy
Code:
zfs snap tank1/veeam@LOCKED
zfs hold LOCKED tank1/veeam@LOCKED

  • Create 3 periodic ZFS snapshots to recover the data if the backup software is compromised
    • Data Protection | Periodic Snapshot Tasks
      • one hourly snapshot with 1 day lifetime
        • Dataset: tank1/veeam
        • Exclude: <empty>
        • [x] Recursive
        • Snapshot lifetime: 1 day
        • Naming Schema: auto-%Y%m%d_%H%M-hourly
        • Schedule: Hourly (0 * * * * ) at the start of each hour
        • Begin; 00:00:00
        • End: 23:59:00
        • [x] Allow Taking Empty Snapshots
        • [x] Enabled
        • SAVE
      • one daily snapshot with 2 weeks lifetime
        • Dataset: tank1/veeam
        • Exclude: <empty>
        • [x] Recursive
        • Snapshot lifetime: 2 weeks
        • Naming Schema: auto-%Y%m%d_%H%M-daily
        • Schedule: Daily (0 0 * * * ) at 00:00 (12:00 AM)
        • [x] Allow Taking Empty Snapshots
        • [x] Enabled
        • SAVE
      • one weekly snapshot with 1 month lifetime
        • Dataset: tank1/veeam
        • Exclude: <empty>
        • [x] Recursive
        • Snapshot lifetime: 1 month
        • Naming Schema: auto-%Y%m%d_%H%M-weekly
        • Schedule: Weekly (0 0 * * sun ) on Sundays at 00:00 (12:00 AM)
        • [x] Allow Taking Empty Snapshots
        • [x] Enabled
        • SAVE
If you have enough disk space you can even have longer retention periods

  • Configure SMB Service (System Settings | Services | SMB)
    • netbiosname: hardenedserver
    • netbios alias: <blank>
    • workgroup: WORKGROUP
    • Description: hardened TrueNAS
    • [ ] Enable SMB1 support
    • [ ] NTLMv1 Auth
    • enable autostart, and start smb immediately
    • If you have several network interface you can bind the SMB service to a specific interface in the Advanced Options

  • Setup a SMB share to be used by the backup software (veeam) (Share | Windows (SMB) Shares) |
    • Add
      • path: /mnt/tank1/veeam
      • name: veeam
      • purpose: multi-protocol (NFSv3/SMB) shares
      • description: hardened veeam repo
      • [x] enabled
      • [x] Enable ACL
      • [ ] Export Read Only
      • [x] Browseable to Network client
      • [ ] allow guest access
      • [ ] allow based shared enumeration
      • host allow: put the IP of the Veeam Software server here
      • host deny: <blank>
      • [ ] Use as home share
      • [ ] Timemachine
      • [ ] legacy AFP compatibility
      • [x] Enable shadow copy
      • [ ] Export Recycle bin
      • [ ] Use Apple-Stlye Character Encoding
      • [ ] Enable alternate data streams
      • [ ] Enable SMB2/3 Durable handles
      • [ ] Enable FSRVP
      • Path suffix: <blank>
      • Auxiliary parameters: <blank>
      • SAVE
now add this repository to veeam software


Hardened the repository
  • If you wish to use (temporary) SSH, install your public SSH key on the root account
    • Credentials | Local Users | root
  • Enable password for console access
    • System Settings | Advanced | Console | Configure
      • [ ] Show Text Console without Password Prompt
      • MOTD Banner: Hardened repository without remote management, to enable temporary the web interface type "systemctl start nginx"
  • Enable two factor authentication (2FA) for web access
    • Credentials | 2FA
      • Enable Two-Factor Authentication
  • Disconnect the IPMI
  • Disable the SSH service for normal operation
    • Disable the SSH service with the TrueNAS services web page
      • Services | SSH | Disable
    • Add a startup script to stop SSH in case we have enabled SSH by mistake
      • System Settings | Advanced | Init/Shutdown Scripts | Add
        • Description: Stop SSH at startup
        • type: COMMAND
        • when=POSTINIT
        • command="/usr/bin/systemctl stop ssh"
        • enabled=true
        • timeout= 10
        • SAVE
    • Add a cron job to stop ssh at midnight (in case we forget to stop ssh after a maintenance operation)
      • System Settings | Advanded | Cron Job | Add
        • description="stop ssh at midnight"
        • command="/usr/bin/systemctl stop ssh"
        • run as user=root
        • Schedule: daily (0 0 * * *) at 00:00 (12:AM)
        • [X] hide standard output
        • [ ] hide standard error
        • enabled=true
        • SAVE
  • Edit message of the day to explain how to restart the webui
    • This is a hardened repository without remote management web interface. To restart the webui type "systemctl start nginx"
  • Disable the web interface
    • System Settings | Advanced | Init/Shutdown Scripts | Add
      • Description: Stop webui at startup
      • type: COMMAND
      • when=POSTINIT
      • command="/usr/bin/systemctl stop nginx"
      • enabled=true
      • timeout= 10
      • SAVE
    • Add a cron job to webui at midnight (in case we forget to stop webui after a maintenance operation)
    • Add a cron job to stop nginx at midnight (in case we forget to stop nginx after a maintenance operation)
      • System Settings | Advanded | Cron Job | Add
        • description="stop webui at midnight"
        • command="/usr/bin/systemctl stop nginx"
        • run as user=root
        • Schedule: daily (0 0 * * *) at 00:00 (12:AM)
        • [X] hide standard output
        • [ ] hide standard error
        • enabled=true
        • SAVE

At the end reboot the TrueNAS server to check that everything works as expected.

Daily management
  • If something goes wrong, TrueNAS will send a mail notification.
  • If you need to temporary enable the web interface
    • Login to the console (a password is required)
    • Start either the webui (nginx) or the SSH service
Code:
systemctl start nginx
systemctl start ssh

  • When you have finished your maintenance operation, switch off the web and ssh services
Code:
systemctl stop nginx
systemctl stop ssh

  • if you forgot to stop the service, the cron job will do if for you at midnight
Enjoy your hardened repository, and sleep more peacefully at night.
 
Last edited by a moderator:

Kris Moore

SVP of Engineering
Administrator
Moderator
iXsystems
Joined
Nov 12, 2015
Messages
1,448
Wow, this is a fantastic walkthrough / recipe for Veeam users! Would you perhaps be opposed to contributing this to our community guides in the handbook? So it doesn't get lost on the forums over time?

 

blanchet

Guru
Joined
Apr 17, 2018
Messages
515
Indeed, I have just written this guide to help people sleeping peacefully at night without worry to much about ransomware.

I did not know that there is a special section in the official documentation for such document.
Ok I will try to convert into a community guide during the Christmas holidays, so that the document could be easily improved / updated over the time by the TrueNAS community.
 
Top