winnielinnie
MVP
- Joined
- Oct 22, 2019
- Messages
- 3,641
Disclaimer: I use bold text to highlight important points for visitors who quickly scroll through forum posts. I don't mean for it to appear as if I am yelling at the reader.
Currently, as of FreeNAS 11.2 U6, there exists no means to manually replicate your entire main pool to an external USB drive using only the official web interface buttons and menus; let alone having to create a redundant Periodic Snapshot Task. This would be useful in dealing with an emergency situation in the event you cannot access your main pool anymore (corruption, loss of disks, lost encryption keys, etc), yet you cannot afford nor have the capability or time to purchase, build, and configure a separate FreeNAS system for routine remote replication tasks.
Until such a feature exists in the GUI, here is the method I am using, which comes very close to achieving the same level of convenience. If anyone sees anything wrong with this method (even a potential risk), please let me know! I have tested it several times and it appears to work, *yet it lacks any status or progress indicator, so I am left to "guess" when the task is complete. I can also check my email, since the task is configured to email me when it finishes. *(You can always see if it's currently running by clicking on the Task Manager icon in the upper-right.)
Here are the steps I took, including my initial replication, followed by the script I created under Tasks > Cron Jobs.
---
One-Time Initial Replication to USB Drive
Before any scripts or incremental backups, I need to replicate my entire main pool to the USB drive. This is done only once. All subsequent backups will be done with the "manual, on-demand" cron job explained later in this forum post.
1) First I make sure no other tasks or data transfers are taking place. This initial backup can take a few hours or more, depending on how much data is involved.
2) In the main menu of the FreeNAS web interface, I click on Shell.
Next I do the following commands in order.
3) Physically plug in the prepared USB drive, assuming it has been initialized with a fresh empty pool named usbdrivepool of equal or greater capacity as your main pool.
4) Import the pool, usbdrivepool, under Storage > Pools > Add.
5) Create a snapshot of the entire main pool, recursively, including all child datasets.
6) Backup this entire snapshot (mainpool@offsite-backup-new) of the main pool created just now, to the USB drive containing the fresh empty pool named usbdrivepool.
This can take a very long time, so make sure to keep the Shell window open to monitor its progress.
---
Creating the Task
Now that the one-time initial backup is done, the following is for recurrent, on-demand manual incremental backups, which should yield a perfect clone of your main pool to the external USB drive. It will transfer only the changes since the previous backup.
1) First, I create a cron job under Tasks > Cron Jobs > Add, and configure it in such a way that it will never run on its own: it can only be run on demand, manually.
Under Schedule a Cron Job, I select the drop-down menu entry Custom. I then select Monthly under Presets. I then set Minutes to 0, Hours to 0, and Days to 1. I then select only Jan under Months. I click Done. (This simply is used as an arbitrary place-holder, since all tasks require a date. January 1 at midnight is nothing special.) The task will never run on its own, since I will next "disable" it.
2) I uncheck Enabled.
3) I uncheck Hide Standard Error.
4) I uncheck Hide Standard Output.
5) Under Run as User I select root from the drop-down menu.
6) Under Description I write: Replicate entire main pool to external USB drive for offsite safe-keeping
7) Under Command I write out the following several commands (in a single line), which will be run in sequential order (separated by semicolons), and finally emailed to me when it completes. I will explain what I assume it does. Yes, the parentheses are intentional, as it runs in a subshell prior to emailing me.
---
Manually Running the Backup to the USB Drive
Whenever I wish to clone my main pool to my emergency backup USB drive, I do the following.
1) Physically plug in the USB drive.
2) Import usbdrivepool through Storage > Pools > Add.
3) Go to Tasks > Cron Jobs > click the three-dot menu drop-down to the right of the "Replicate entire main pool to external USB drive for offsite safe-keeping" task > click on Run Now.
4) Check my email until I see a confirmation that the task has completed. (Or keep checking with Task Manager, which will perpetually display 20% completion until the task is truly finished.)
5) Optionally, do any verifications to make sure everything went smoothly.
6) Disconnect usbdrivepool through Storage > Pools > click the cogwheel icon next to its name > Export / Disconnect.
7) Physically unplug the USB drive.
8) Place the USB drive somewhere safe, offsite. It can be retrieved in case of an emergency, or when another manual incremental backup needs to be created, or stored in secure briefcase for security and archive purposes.
---
What the Manual On-Demand Cron Job Supposedly Does
The cron job does the following in the specific order.
Rename the current all-inclusive snapshot ("-new"), used for the previous backup, to an "-old" snapshot, to prepare it for the next incremental backup.
Make a new all-inclusive snapshot ("-new"), which will next be used to transfer only the changes since the previous all-inclusive snapshot ("-old").
Replicate the main pool (mainpool) to the USB pool (usbdrivepool), using an incremental backup of the changes from the -old to the -new snapshot.
Delete the "-old" all-inclusive snapshots, which are no longer needed anymore. (The "-new" all-inclusive snapshots will be used for the next incremental backup, obviously being renamed to "-old", since a fresh all-inclusive "-new" snapshot is made immediately before the next incremental backup.)
Everything within the subshell on the cron job is then emailed with the subject "FreeNAS Replication to USB Drive". Hopefully, any errors will be printed in the email as well.
---
Questions, Comments, Criticisms?
I need to know if anything is wrong, risky, or could be done better. I don't want myself or anyone to risk any data loss if anything I wrote could possibly cause irreversible damage!
---
UPDATE June 17, 2020:
Currently, as of FreeNAS 11.2 U6, there exists no means to manually replicate your entire main pool to an external USB drive using only the official web interface buttons and menus; let alone having to create a redundant Periodic Snapshot Task. This would be useful in dealing with an emergency situation in the event you cannot access your main pool anymore (corruption, loss of disks, lost encryption keys, etc), yet you cannot afford nor have the capability or time to purchase, build, and configure a separate FreeNAS system for routine remote replication tasks.
Until such a feature exists in the GUI, here is the method I am using, which comes very close to achieving the same level of convenience. If anyone sees anything wrong with this method (even a potential risk), please let me know! I have tested it several times and it appears to work, *yet it lacks any status or progress indicator, so I am left to "guess" when the task is complete. I can also check my email, since the task is configured to email me when it finishes. *(You can always see if it's currently running by clicking on the Task Manager icon in the upper-right.)
Here are the steps I took, including my initial replication, followed by the script I created under Tasks > Cron Jobs.
---
One-Time Initial Replication to USB Drive
Before any scripts or incremental backups, I need to replicate my entire main pool to the USB drive. This is done only once. All subsequent backups will be done with the "manual, on-demand" cron job explained later in this forum post.
1) First I make sure no other tasks or data transfers are taking place. This initial backup can take a few hours or more, depending on how much data is involved.
2) In the main menu of the FreeNAS web interface, I click on Shell.
Next I do the following commands in order.
3) Physically plug in the prepared USB drive, assuming it has been initialized with a fresh empty pool named usbdrivepool of equal or greater capacity as your main pool.
4) Import the pool, usbdrivepool, under Storage > Pools > Add.
5) Create a snapshot of the entire main pool, recursively, including all child datasets.
zfs snapshot -r mainpool@offsite-backup-new
6) Backup this entire snapshot (mainpool@offsite-backup-new) of the main pool created just now, to the USB drive containing the fresh empty pool named usbdrivepool.
zfs send -R mainpool@offsite-backup-new | zfs recv -vFd usbdrivepool
This can take a very long time, so make sure to keep the Shell window open to monitor its progress.
---
Creating the Task
Now that the one-time initial backup is done, the following is for recurrent, on-demand manual incremental backups, which should yield a perfect clone of your main pool to the external USB drive. It will transfer only the changes since the previous backup.
1) First, I create a cron job under Tasks > Cron Jobs > Add, and configure it in such a way that it will never run on its own: it can only be run on demand, manually.
Under Schedule a Cron Job, I select the drop-down menu entry Custom. I then select Monthly under Presets. I then set Minutes to 0, Hours to 0, and Days to 1. I then select only Jan under Months. I click Done. (This simply is used as an arbitrary place-holder, since all tasks require a date. January 1 at midnight is nothing special.) The task will never run on its own, since I will next "disable" it.
2) I uncheck Enabled.
3) I uncheck Hide Standard Error.
4) I uncheck Hide Standard Output.
5) Under Run as User I select root from the drop-down menu.
6) Under Description I write: Replicate entire main pool to external USB drive for offsite safe-keeping
7) Under Command I write out the following several commands (in a single line), which will be run in sequential order (separated by semicolons), and finally emailed to me when it completes. I will explain what I assume it does. Yes, the parentheses are intentional, as it runs in a subshell prior to emailing me.
( zfs rename -r mainpool@offsite-backup-new mainpool@offsite-backup-old; zfs snapshot -r mainpool@offsite-backup-new; zfs send -RI mainpool@offsite-backup-old mainpool@offsite-backup-new | zfs recv -vFd usbdrivepool; zfs destroy -r mainpool@offsite-backup-old; zfs destroy -r usbdrivepool@offsite-backup-old; ) | mail -s "FreeNAS Replication to USB Drive" "myemail@example.com"
---
Manually Running the Backup to the USB Drive
Whenever I wish to clone my main pool to my emergency backup USB drive, I do the following.
1) Physically plug in the USB drive.
2) Import usbdrivepool through Storage > Pools > Add.
3) Go to Tasks > Cron Jobs > click the three-dot menu drop-down to the right of the "Replicate entire main pool to external USB drive for offsite safe-keeping" task > click on Run Now.
4) Check my email until I see a confirmation that the task has completed. (Or keep checking with Task Manager, which will perpetually display 20% completion until the task is truly finished.)
5) Optionally, do any verifications to make sure everything went smoothly.
6) Disconnect usbdrivepool through Storage > Pools > click the cogwheel icon next to its name > Export / Disconnect.
7) Physically unplug the USB drive.
8) Place the USB drive somewhere safe, offsite. It can be retrieved in case of an emergency, or when another manual incremental backup needs to be created, or stored in secure briefcase for security and archive purposes.
---
What the Manual On-Demand Cron Job Supposedly Does
The cron job does the following in the specific order.
Rename the current all-inclusive snapshot ("-new"), used for the previous backup, to an "-old" snapshot, to prepare it for the next incremental backup.
zfs rename -r mainpool@offsite-backup-new mainpool@offsite-backup-old
Make a new all-inclusive snapshot ("-new"), which will next be used to transfer only the changes since the previous all-inclusive snapshot ("-old").
zfs snapshot -r mainpool@offsite-backup-new
Replicate the main pool (mainpool) to the USB pool (usbdrivepool), using an incremental backup of the changes from the -old to the -new snapshot.
zfs send -RI mainpool@offsite-backup-old mainpool@offsite-backup-new | zfs recv -vFd usbdrivepool
Delete the "-old" all-inclusive snapshots, which are no longer needed anymore. (The "-new" all-inclusive snapshots will be used for the next incremental backup, obviously being renamed to "-old", since a fresh all-inclusive "-new" snapshot is made immediately before the next incremental backup.)
zfs destroy -r mainpool@offsite-backup-old; zfs destroy -r usbdrivepool@offsite-backup-old
Everything within the subshell on the cron job is then emailed with the subject "FreeNAS Replication to USB Drive". Hopefully, any errors will be printed in the email as well.
mail -s "FreeNAS Replication to USB Drive" "myemail@example.com"
---
Questions, Comments, Criticisms?
I need to know if anything is wrong, risky, or could be done better. I don't want myself or anyone to risk any data loss if anything I wrote could possibly cause irreversible damage!
---
UPDATE June 17, 2020:
- I cleaned up the formatting to more easily follow through.
- This is meant as a means to fill in a user's need to make a quick and simple "entire pool" backup to an easily accessible USB drive. Many of us don't have the money, time, nor convenience of a second location to configure a dedicated and fully automated server-to-server replication task. I believe that something is better than nothing. I have used this method many times and it has worked exactly as expected. This is basically the equivalent of copying everything to a USB drive, while retaining the ZFS pool/dataset structure.
- There still exists no means of achieving this from the GUI menus, as it still requires the creation of a Periodic Snapshot Task, which is hard to overlay for something done manually (i.e., plugging and unplugging a USB drive.)
Last edited: