A script to run rclone on the FreeNAS server to backup NAS data to Backblaze B2 cloud storage. Can easily be adapted for other cloud providers. The script is available from my GitHub site.
Script was adapted from the one posted in the FreeNAS community forums by Martin Aspeli: https://www.ixsystems.com/community...2-backup-solution-instead-of-crashplan.58423/
Why? Freenas (v11.1 onwards) now supports rclone nativetly and also has GUI entries for Cloudsync, however there is no option to use additional parameters within the GUI, thus I have adapted a script to be run instead via a cron job.
Note: The rclone config file saved via the CLI is different to the one used in the FreeNAS GUI, so its safe to update this configuration file via rclone config without affecting anything you have configured in the FreeNAS GUI.
### Change Log ###
Update(1): The log file rclone produces is not user friendly, therefore this script will also create a more user friendly log (to use as the email body). This shows the stats, and for "sync" a list of files Copied (new), Copied (replaced existing), and Deleted - as well as any errors/notices; for "cryptcheck" it only lists errors or notices. (see my github site for examples)
29-02-2020: I've noticed that since upgrading to FreeNAS v11.3 that this script was no longer working. When you create the "rclone.conf" file from a shell prompt (or SSH connection) with the command "rclone config" it is stored at "/root/.config/rclone/rclone.conf" (you can check this by running the command "rclone config file").
However, when you then run this bash script as a cron task, for some reason it is looking for the config file at "/.config/rclone/rclone.conf"; a quick fix was to copy the "rclone.conf" file to this folder. However, it's not a good idea to have multiple copies of the "rclone.conf" config file as this could become confusing when you make changes. Therefore I decided to update the script to utilise the "--config" parameter of rclone to point to the config file created at "/root/.config/rclone/rclone.conf". I did log this as a bug (NAS-105088), but it was closed by IX as a script issue - however, I don't believe that is correct!
12-03-2020: I have updated the script to be dual purpose, with regard to running a "rclone sync" to backup, and "rclone cryptcheck" to do a verification of your cloud based files and see if any files are missing as well as confirming the checksums of all encrypted files (for the paranoid out there!). You now need to run this script with an parameter, for example: "rclonebackup.sh sync" or "rclonebackup.sh cryptcheck", any other parameters or no parameter will result in an error email being sent. As this reuses code dynamically it means that any configuration changes in this script are only entered once and used for both sync and cryptcheck.
Further information about cryptcheck can be found at: https://rclone.org/commands/rclone_cryptcheck/
02-04-2020: I have updated it as follows:
Step 1
First off you need to configure rclone for Backblaze B2 as per: https://rclone.org/b2/, and if encrypting your data with rclone you then need to configure as per: https://rclone.org/crypt/.
Step 2
Download the rclonebackup.sh shell script and rclone_excludes.txt file from my github site.
Step 3
Save the two files within a dataset on your FreeNAS server.
Step 4
You now need to edit the rclonebackup.sh script, to update some of the user configurations, as per your specific requirements:
Basically the default settings above will:
src:
backup all data within your pool (/mnt/tank in my case).
dest:
store them in the root of your remote:bucket on Backblaze B2 (secret:/ in my case).
exclude_list:
The path where you have saved the "rclone_excludes.txt" file (include the filename as well).
compressLog:
Do you want the log compressed with gzip?
keepLog:
Do you want to keep the log?
backupDestination:
Where to keep the log?
amountBackups
How many logs to keep?
email:
Your email address, for backup results and log file. Can be left as root
cfg_file:
The location of the rclone.conf file.
log_level:
for your first run I recommend this is set to NOTICE, as this will only report errors, otherwise the log file will get very large. Once you have completed your first backup, you can change this it INFO, which will let you know what files were backed-up or deleted.
min_age:
set a minimal age of the file before it is backed, handy for ignoring files currently in use.
transfer:
the amount of simultaneous transfer, Backblaze recommends a lot, so I have set at 16, but you can tweak this depending on your broadband connection.
You then may want to edit the rclone_excludes.txt file and and add/remove exclusions as per your requirements.
Step 4a
Just a note on other parameters used:
--fast-list:
Use recursive list if available, uses more memory but fewer Backblaze B2 Class C transactions (i.e. cheaper). Not supported by all cloud providers though.
--copy-links:
Follow symlinks and copy the pointed to item. (delete this line if not wanted)
--b2-hard-delete:
Permanently delete files on remote removal, otherwise hide files. This means no version control of files on Backblaze (saves space & thus costs). This is a Backblaze B2 specific command. (delete this line if not wanted)
--exclude:
These lines will exclude certain files and folders from being backed-up. (delete this if not wanted, or edit, or add additional lines as per your needs)
Step 5
You now want to create a new cron job task via the FreeNAS GUI, remembering the command you want to run is the full path to the rclonebackup.sh script, with the parameter of sync or cryptcheck or check:
I recommend that for the first time you run this, that you do not enable the task. This is because the first run could take some time, many days if you have a few TB to backup. You do not want the server starting a new task while the old one is still running, as this will only confuse the backup process and slow down your server.
So, for your first time, just select the task and select Run Now, i.e.
Note: If the FreeNAS server reboots at any time during this first run, then you will need to restart the process by selecting Run Now again. This will re-scan the files and continue from where it last got to.
Step 6
Once the backup task has completed you should receive an email, along with a log file (of any errors encountered).
You can now edit the cron job and tick the Enabled box, so that the task will run automatically at the times you want. You may also want to edit the rclonebackup.sh script to change the log-level to INFO at this stage.
I hope this helps?
Yours
Jonathan
Script was adapted from the one posted in the FreeNAS community forums by Martin Aspeli: https://www.ixsystems.com/community...2-backup-solution-instead-of-crashplan.58423/
Why? Freenas (v11.1 onwards) now supports rclone nativetly and also has GUI entries for Cloudsync, however there is no option to use additional parameters within the GUI, thus I have adapted a script to be run instead via a cron job.
Note: The rclone config file saved via the CLI is different to the one used in the FreeNAS GUI, so its safe to update this configuration file via rclone config without affecting anything you have configured in the FreeNAS GUI.
### Change Log ###
Update(1): The log file rclone produces is not user friendly, therefore this script will also create a more user friendly log (to use as the email body). This shows the stats, and for "sync" a list of files Copied (new), Copied (replaced existing), and Deleted - as well as any errors/notices; for "cryptcheck" it only lists errors or notices. (see my github site for examples)
29-02-2020: I've noticed that since upgrading to FreeNAS v11.3 that this script was no longer working. When you create the "rclone.conf" file from a shell prompt (or SSH connection) with the command "rclone config" it is stored at "/root/.config/rclone/rclone.conf" (you can check this by running the command "rclone config file").
However, when you then run this bash script as a cron task, for some reason it is looking for the config file at "/.config/rclone/rclone.conf"; a quick fix was to copy the "rclone.conf" file to this folder. However, it's not a good idea to have multiple copies of the "rclone.conf" config file as this could become confusing when you make changes. Therefore I decided to update the script to utilise the "--config" parameter of rclone to point to the config file created at "/root/.config/rclone/rclone.conf". I did log this as a bug (NAS-105088), but it was closed by IX as a script issue - however, I don't believe that is correct!
12-03-2020: I have updated the script to be dual purpose, with regard to running a "rclone sync" to backup, and "rclone cryptcheck" to do a verification of your cloud based files and see if any files are missing as well as confirming the checksums of all encrypted files (for the paranoid out there!). You now need to run this script with an parameter, for example: "rclonebackup.sh sync" or "rclonebackup.sh cryptcheck", any other parameters or no parameter will result in an error email being sent. As this reuses code dynamically it means that any configuration changes in this script are only entered once and used for both sync and cryptcheck.
Further information about cryptcheck can be found at: https://rclone.org/commands/rclone_cryptcheck/
02-04-2020: I have updated it as follows:
- Tidied up the script;
- Separated out user and system defined variables (to make editing easier);
- Have moved from multiple "--exclude" statements to using "--exclude-from" and a separate file to hold excludes, which will make managing excludes much easier and saves editing this script.
- Have changed the default email to root.
- Have added in a third parameter option "check", which checks the files in the source and destination match.
- Rewrote the email generation aspect of the script to remove the need for the separate email_attachments.sh script - all managed within the one script now; as well as moving from "echo" to "printf" commands for more control.
- Tidied up the script some more;
- Have added another option to compress (gzip) the log file before being attached to the email (compressLog="yes"), this is set to "yes" by default.
- You can also now request the script to keep a local copy (backup) of the unformatted log file (keepLog="no"), this is set to "no" by default, as well as how many logs to keep (amountBackups=31). Keeping the log was code I used to cut-n-paste in for my debugging, but decided to leave it in as an option now.
- Have added in some checks that will verify that some of the user defined variables are valid.
Step 1
First off you need to configure rclone for Backblaze B2 as per: https://rclone.org/b2/, and if encrypting your data with rclone you then need to configure as per: https://rclone.org/crypt/.
Step 2
Download the rclonebackup.sh shell script and rclone_excludes.txt file from my github site.
Step 3
Save the two files within a dataset on your FreeNAS server.
Step 4
You now need to edit the rclonebackup.sh script, to update some of the user configurations, as per your specific requirements:
Code:
src="/mnt/tank" dest="secret:/" exclude_list="/mnt/tank/Sysadmin/scripts/rclone_excludes.txt" compressLog="yes" keepLog="no" backupDestination="/location/for/log/file/save" amountBackups=31 email=root cfg_file="/root/.config/rclone/rclone.conf" log_level="NOTICE" min_age="15m" transfers="16"
Basically the default settings above will:
src:
backup all data within your pool (/mnt/tank in my case).
dest:
store them in the root of your remote:bucket on Backblaze B2 (secret:/ in my case).
exclude_list:
The path where you have saved the "rclone_excludes.txt" file (include the filename as well).
compressLog:
Do you want the log compressed with gzip?
keepLog:
Do you want to keep the log?
backupDestination:
Where to keep the log?
amountBackups
How many logs to keep?
email:
Your email address, for backup results and log file. Can be left as root
cfg_file:
The location of the rclone.conf file.
log_level:
for your first run I recommend this is set to NOTICE, as this will only report errors, otherwise the log file will get very large. Once you have completed your first backup, you can change this it INFO, which will let you know what files were backed-up or deleted.
min_age:
set a minimal age of the file before it is backed, handy for ignoring files currently in use.
transfer:
the amount of simultaneous transfer, Backblaze recommends a lot, so I have set at 16, but you can tweak this depending on your broadband connection.
You then may want to edit the rclone_excludes.txt file and and add/remove exclusions as per your requirements.
Step 4a
Just a note on other parameters used:
--fast-list:
Use recursive list if available, uses more memory but fewer Backblaze B2 Class C transactions (i.e. cheaper). Not supported by all cloud providers though.
--copy-links:
Follow symlinks and copy the pointed to item. (delete this line if not wanted)
--b2-hard-delete:
Permanently delete files on remote removal, otherwise hide files. This means no version control of files on Backblaze (saves space & thus costs). This is a Backblaze B2 specific command. (delete this line if not wanted)
--exclude:
These lines will exclude certain files and folders from being backed-up. (delete this if not wanted, or edit, or add additional lines as per your needs)
Step 5
You now want to create a new cron job task via the FreeNAS GUI, remembering the command you want to run is the full path to the rclonebackup.sh script, with the parameter of sync or cryptcheck or check:
I recommend that for the first time you run this, that you do not enable the task. This is because the first run could take some time, many days if you have a few TB to backup. You do not want the server starting a new task while the old one is still running, as this will only confuse the backup process and slow down your server.
So, for your first time, just select the task and select Run Now, i.e.
Note: If the FreeNAS server reboots at any time during this first run, then you will need to restart the process by selecting Run Now again. This will re-scan the files and continue from where it last got to.
Step 6
Once the backup task has completed you should receive an email, along with a log file (of any errors encountered).
You can now edit the cron job and tick the Enabled box, so that the task will run automatically at the times you want. You may also want to edit the rclonebackup.sh script to change the log-level to INFO at this stage.
I hope this helps?
Yours
Jonathan