Home user backup stratergy help

[Geneleco]

Hello all
I hope i am posting in the right place and you can help me.
I am trying to set up a backup strategy for me and my family, I will be backing up 7 win10 pcs, and a few phones

Here is my current plan, please if you have feed back or suggestions let me know.
I have 4 different criteria for what my backup is to protect my data from
1: Accidental file deletion
2: Hardware breaking (hdd ,ssd etc)
3: Ransomware
4: Property damage (flood, fire etc)

My plan is to,
-Use a sync software (syncback, bvckup2), and sync files and folders to my nas (custom truenas build), this is to protect me from Accidental file deletion. And i will be able to access the files directly on the nas without having to do a lengthy restore.
-Use a backup software (macrium reflect, or any other suggestions) to image windows install and specific folders and place the image on the nas
-Use m disc blue rays to back up compressed very important documents and photos (very little data), and store offsite and offline, this should help if there is a flood/fire,
or if my network gets ransomware, apparently ransomware now will look for nas servers on the network and encrypt them.

Now here comes a part im really not sure about,
Should i be wanting to backup from the nas to a cloud service (back blaze b2), or should i use a on device cloud backup such as idrive, backblaze, etc
I travel a lot and am sometimes away for 2 months at a time, so i don't always have direct access to my nas with my laptop, in that case maybe on device cloud backup is better?
Using on device would also let me use the same service to back up from my phone. So maybe it is more cost effective.

Am i completely wrong in my thinking?
Should i be doing something else?
Any suggestions are welcome.
Thank you

 

[elvisimprsntr]

Implement a 3-2-1 backup strategy. https://en.wikipedia.org/wiki/Backup#3-2-1_rule

• 3 copies of your data (original + 2 backups)
• 2 different types of media (NAS + 1 another media type, i.e. external drive)
• 1 encrypted offsite (only highly/mission critical data to prevent loss from disaster, i.e. financial, medical, legal, etc. data)

To protect against accidental file deletion and/or malware/ransomware, you will need to implement some sort of full, incremental, differential backup strategy. Suggest following the grandfather-father-son backup strategy. https://en.wikipedia.org/wiki/Backup#Backup_methods

• Hourly/Daily
• Weekly
• Monthly

Most importantly, automate backups.

Show signature for my implementation.

 

[elorimer]

I'm looking forward to the responses you get so I can go to school on them. Here is some of my thinking, with 4 windows PCs and three Chromebooks.

I used to think that a bare metal backup of a Windows PC was a good idea, and a local backup on the PC was a good idea too. Given digital licenses and how easy it is to do a complete fresh reinstall from MCT, I don't think so anymore. I have a list of apps and pesky numeric licenses (curse you, Nuance!) for reinstalling software, so it is really just files to worry about. That leads me to think that moving Documents, Downloads, Video and Music to the NAS is the right way to go, and do the backup from there. Local storage is irrelevant. If you do that, you have to watch out for other backup methods, since a system backup will backup all that too.

Then snapshots for ransomware and accidental deletion. Then replicating to another pool on the NAS for hardware failure. Then replicating to another NAS physically somewhere else for flood and fire.

Last, I am coming to the view that Onedrive and Google Drive are good insurance as well and should be part of the mix too. Backblaze is remarkably cheap if you don't have to download from it.

 

[winnielinnie]

if my network gets ransomware, apparently ransomware now will look for nas servers on the network and encrypt them.

This doesn't affect your snapshots, which are read-only and can only be destroyed from the server itself (not via a network share over Windows). So make sure you have some periodic snapshots taken at regular intervals.

Use a sync software (syncback, bvckup2), and sync files and folders to my nas

Make sure that the sync software writes modifcations "in place" rather than creating temporary files. Otherwise, if something like a 1GB Thunderbird inbox file has even the slightest modification, your snapshots will each consume an additional 1GB (when in fact only several KB have changed).

 

[Arwen]

Agreed for using ZFS snapshots as some ransomware mitigation.

I also use ZFS snapshots on my NAS' external backup disks, so each backup disk has multiple backups on it. Which also allows more ransomware mitigation, (and versions of files changed).

One other note, some people implement VPN software to allow remote management of their Home NAS. The VPN should protect your Home LAN from intrusion.

 

[Geneleco]

Implement a 3-2-1 backup strategy

To protect against accidental file deletion and/or malware/ransomware, you will need to implement some sort of full, incremental, differential backup strategy. Suggest following the grandfather-father-son backup strategy

• Hourly/Daily
• Weekly
• Monthly

Yes based my strategy on the 3-2-1 idea,
That does sound like i good idea i will implement that thank you.

 

[Geneleco]

I'm looking forward to the responses you get so I can go to school on them. Here is some of my thinking, with 4 windows PCs and three Chromebooks.

I used to think that a bare metal backup of a Windows PC was a good idea, and a local backup on the PC was a good idea too. Given digital licenses and how easy it is to do a complete fresh reinstall from MCT, I don't think so anymore. I have a list of apps and pesky numeric licenses (curse you, Nuance!) for reinstalling software, so it is really just files to worry about. That leads me to think that moving Documents, Downloads, Video and Music to the NAS is the right way to go, and do the backup from there. Local storage is irrelevant. If you do that, you have to watch out for other backup methods, since a system backup will backup all that too.

Then snapshots for ransomware and accidental deletion. Then replicating to another pool on the NAS for hardware failure. Then replicating to another NAS physically somewhere else for flood and fire.

Last, I am coming to the view that Onedrive and Google Drive are good insurance as well and should be part of the mix too. Backblaze is remarkably cheap if you don't have to download from it.

Yes i agree it doesnt seem worth it to image windows,
Yes i do have snapshots setup on my nas, although i will tweak them to be more usefull.
Thank you for your input

 

[Geneleco]

This doesn't affect your snapshots, which are read-only and can only be destroyed from the server itself (not via a network share over Windows). So make sure you have some periodic snapshots taken at regular intervals.

Ah i didnt realise snapshots where that powerful, good to know.
Daily snapshots seems often enough i would imagine.

Make sure that the sync software writes modifcations "in place" rather than creating temporary files. Otherwise, if something like a 1GB Thunderbird inbox file has even the slightest modification, your snapshots will each consume an additional 1GB (when in fact only several KB have changed).

I will make sure to check for that, any suggestions for specific software?
Thank you

 

[winnielinnie]

I will make sure to check for that, any suggestions for specific software?

Unfortunately, I only know of rsync (with its "--inplace" option). As it stands, there's no user-friendly GUI for Windows.

Many of these sleek and shiny offerings, some commercial, some free, don't mention anything about "in place" file modification, which I assume is because it doesn't matter to the majority of users and customers. However, for a copy-on-write filesystem that uses snapshots (ZFS, Btrfs) it makes a big difference.

Every documentation I read through makes no mention of such an option, nor is it clear how a modified file is treated. Usually these software programs implement their own "copy-on-write" method by creating a new version of the file (such as file.ext.tmp), and then renaming it to replace the original file. From a traditional filesystem's point of view, this is basically an older version of the file being modified and replaced by the newer version. However, for ZFS these are entirely different records (blocks), and thus snapshots that save the different records will reflect the additional used space. (In this case, each modified file taking up additional space that is needed for every version of the file in its entirey whenever a previous snapshot exists.)

Hopefully someone knows of such backup software, whether free or not, that modifies an updated file "in place".

 

[Heracles]

I travel a lot and am sometimes away for 2 months at a time,

You could use Nextcloud for that, just like I do.

Configure all your devices to sync with Nextcloud on your TrueNAS. Photo, files, folder, Contacts, Calendars, you can sync basically anything and everything. Mobile, Windows, Linux, Mac, every kind of endpoint can connect to it.

Nextcloud itself has versioning and undelete.
On top of that, you can take ZFS snapshots and then replicate these to another server.

My complete backup strategy is in my signature...

Also, know that whatever your backup solution will be, it will not be of any use until you successfully restore data from it. Do not only take your backups : practice yourself on how to restore them as well. Often, you will end up unable to restore for any reason. If you find out during a test restore, no problem and just fix it. If you find out during an actual incident, well.... too bad.

 

[elorimer]

Otherwise, if something like a 1GB Thunderbird inbox file has even the slightest modification, your snapshots will each consume an additional 1GB (when in fact only several KB have changed).

You might have seen that Microsoft just changed its sharepoint retention policies for outlook files for exactly this reason. The continually changing .pst files were swamping available storage when they were directed to that location.

 

[Geneleco]

Unfortunately, I only know of rsync (with its "--inplace" option). As it stands, there's no user-friendly GUI for Windows.

Many of these sleek and shiny offerings, some commercial, some free, don't mention anything about "in place" file modification, which I assume is because it doesn't matter to the majority of users and customers. However, for a copy-on-write filesystem that uses snapshots (ZFS, Btrfs) it makes a big difference.

Every documentation I read through makes no mention of such an option, nor is it clear how a modified file is treated. Usually these software programs implement their own "copy-on-write" method by creating a new version of the file (such as file.ext.tmp), and then renaming it to replace the original file. From a traditional filesystem's point of view, this is basically an older version of the file being modified and replaced by the newer version. However, for ZFS these are entirely different records (blocks), and thus snapshots that save the different records will reflect the additional used space. (In this case, each modified file taking up additional space that is needed for every version of the file in its entirey whenever a previous snapshot exists.)

Hopefully someone knows of such backup software, whether free or not, that modifies an updated file "in place".

I did look at rsync and grsync, they dont seem reasonable to push onto my family.
Yes i do see how that would be a problem.
I read for bvckup2 it uses delta copying, i wonder if this is suitable. More info here : https://bvckup2.com/kb/delta-copying

 

[Geneleco]

A

You could use Nextcloud for that, just like I do.

Configure all your devices to sync with Nextcloud on your TrueNAS. Photo, files, folder, Contacts, Calendars, you can sync basically anything and everything. Mobile, Windows, Linux, Mac, every kind of endpoint can connect to it.

Nextcloud itself has versioning and undelete.
On top of that, you can take ZFS snapshots and then replicate these to another server.

My complete backup strategy is in my signature...

Also, know that whatever your backup solution will be, it will not be of any use until you successfully restore data from it. Do not only take your backups : practice yourself on how to restore them as well. Often, you will end up unable to restore for any reason. If you find out during a test restore, no problem and just fix it. If you find out during an actual incident, well.... too bad.

Ah yes i didnt think about nextcloud, ill look into it more, but it may be a good option, thank you

 

[winnielinnie]

I did look at rsync and grsync, they dont seem reasonable to push onto my family.

Grsync for Windows hasn't been updated in 12 years. It's abandoned. The only feasible rsync options for Windows would be

• rsync + cygwin
• cwRsync (same as above, just "pre-packaged, GUI version costs money)
• acrosync (the only "GUI" option, yet not free, and development has stagnated for the last couple years)

I read for bvckup2 it uses delta copying

Nearly all of them will boast delta transfers, yet it doesn't address whether or not they modify a file "in place" or by rather creating a temp file first. The latter is almost always the case, even for paid software. There doesn't seem to be much emphasis on "efficiency with CoW filesystems!" A "destination is a destination", whether a USB drive, network share, or cloud drive. Who cares if you're using ZFS and snapshots, right?

---

I tried bvckup2, and it yields the same results as every other backup/sync software I tried.

I made a 1GB .zip archive that contains a bunch of files within.

I ran bvckup2 to a share on my TrueNAS server.

Dataset is consuming 1GB total.

I then made a snapshot called "snap001" of this dataset.

Dataset is consuming 1GB total. snap001 is consuming nothing.

I added a small 4KB file inside the .zip archive.

I ran bvckup2 to a share on my TrueNAS server. It did a "delta" transfer of the .zip file.

Dataset is consuming 2GB total. snap001 is consuming 1GB.

When I did the above steps with rsync (--inplace):

Dataset is still consuming 1GB total. snap001 is consuming almost nothing.

---

The "dangers" of using "in place" don't really flesh out in reality:

Adding / removing files from an existing .zip archive, how does ZFS handle it?

Since ZFS is copy-on-write (COW), and snapshots are done at the block level (unlike other alternatives that rely on the file level), what is "supposed" to happen when you update an existing .zip archive by adding more files to it? Assume the following: The .zip archive is using a default...

www.truenas.com

 

[elorimer]

When I did the above steps with rsync (--inplace):

Dataset is still consuming 1GB total. snap001 is consuming almost nothing.

We've gotten somewhat afield of the OP. But I'm curious--how does this deal with ransomware? How do you restore the filesystem to a prior state?

 

[winnielinnie]

We've gotten somewhat afield of the OP. But I'm curious--how does this deal with ransomware? How do you restore the filesystem to a prior state?

The same as always, leveraging your ZFS snapshots (which are read-only and protected, only destroyable from the server itself). Nothing is fundamentally different, except that your snapshots are smaller and more efficient due to the nature of allowing ZFS to handle CoW natively, rather than third-party software utilizing their "own" method of "copy-on-write" (i.e, temp files). Why should each snapshot consume the amount of space of the full size of the original files, when they can instead consume the amount of space of the only differences? Why should the entirety of the file size of my Thunderbird inbox, Outlook inbox, .zip archive, etc, cause each of my snapshots to take up all of that extra space when I simply clicked "Download new messages" and only receive a few new emails to my inbox? Isn't it preferable that the snapshots only consume the amount of space represented by those new emails, rather than the full size of the inbox every time?

Based from what the OP wrote in their initial post:

My plan is to,
-Use a sync software (syncback, bvckup2), and sync files and folders to my nas (custom truenas build), this is to protect me from Accidental file deletion. And i will be able to access the files directly on the nas without having to do a lengthy restore.

The different methods between modifying a file "in place" as opposed to creating a temporary file and renaming it doesn't change the way that the user will protect/restore older versions or accidentally deleted files. The main difference is treating a CoW filesystem (ZFS, Btrfs) in a more efficient manner.

 

[elorimer]

I think I need to study up a bit more on this.

Say I have a 1GB outlook .pst file. I snapshot my file system. I've been negligent in cleaning out my inbox, though, and I sit down and clean out stuff, so my outlook .pst file is now only .5GB. If I snapshot it again, I thought my file system would be using 1.5GB, being the old file plus the new file. Is it still 1GB though, in your view?

 

[winnielinnie]

If I snapshot it again, I thought my file system would be using 1.5GB, being the old file plus the new file.

Depends on how the file was modified/transferred. Conventional methods simply make a backup of each version of the file in its entirety (file-based software and filesystems). ZFS and Btrfs are block-based, and thus if something is modified in place, the snapshots' consumption reflect only the differences, not the whole files.

---

snap001 might have inbox.pst comprised of records 123, 124, 125, 126, 127

snap002 might have inbox.pst comprised of records 123, 125, 127, 128

I highlighted in bold the records shared between the two snapshots that reference the file inbox.pst.

So whether you restore from snap001 or snap002 determines which version of inbox.pst you will be restoring.

---

ArsTechnica has a great article about ZFS that serves as a primer:

ZFS 101—Understanding ZFS storage and performance

Learn to get the most out of your ZFS filesystem in our new series on storage fundamentals.

arstechnica.com

---

UPDATE: "Blocks" and "records" are used interchangeably, but technically with ZFS files are comprised of records, records are comprised of blocks. The block size is determined at pool creation with the ashift value, which by default is 4KB for spinning HDDs in TrueNAS to match the typical 4KB sector size. The larger the maximum record size, the more efficient is ZFS's built-in compression, as it has more "breathing room" to work with in an attempt to reduce the number of blocks that comprise a record.

Imagine you allow 1MB records, and the compression algorithm manages to reduce the number of 4K blocks that comprise a particular record from 256 down to 218. That's 85% the size of what would have been the original uncompressed record.

 

[Geneleco]

Grsync for Windows hasn't been updated in 12 years. It's abandoned. The only feasible rsync options for Windows would be

• rsync + cygwin
• cwRsync (same as above, just "pre-packaged, GUI version costs money)
• acrosync (the only "GUI" option, yet not free, and development has stagnated for the last couple years)

Nearly all of them will boast delta transfers, yet it doesn't address whether or not they modify a file "in place" or by rather creating a temp file first. The latter is almost always the case, even for paid software. There doesn't seem to be much emphasis on "efficiency with CoW filesystems!" A "destination is a destination", whether a USB drive, network share, or cloud drive. Who cares if you're using ZFS and snapshots, right?

---

I tried bvckup2, and it yields the same results as every other backup/sync software I tried.

I made a 1GB .zip archive that contains a bunch of files within.

I ran bvckup2 to a share on my TrueNAS server.

Dataset is consuming 1GB total.

I then made a snapshot called "snap001" of this dataset.

Dataset is consuming 1GB total. snap001 is consuming nothing.

I added a small 4KB file inside the .zip archive.

I ran bvckup2 to a share on my TrueNAS server. It did a "delta" transfer of the .zip file.

Dataset is consuming 2GB total. snap001 is consuming 1GB.

When I did the above steps with rsync (--inplace):

Dataset is still consuming 1GB total. snap001 is consuming almost nothing.

---

The "dangers" of using "in place" don't really flesh out in reality:

Adding / removing files from an existing .zip archive, how does ZFS handle it?

Since ZFS is copy-on-write (COW), and snapshots are done at the block level (unlike other alternatives that rely on the file level), what is "supposed" to happen when you update an existing .zip archive by adding more files to it? Assume the following: The .zip archive is using a default...

www.truenas.com

I seem that very unfortunate, very weird that more software doesnt have the option to do a in place edit.
although the dangers dont appear in your testing, is that the main reason why it is not used? I guess the dangers are there in theory

 

[Geneleco]

Depends on how the file was modified/transferred. Conventional methods simply make a backup of each version of the file in its entirety (file-based software and filesystems). ZFS and Btrfs are block-based, and thus if something is modified in place, the snapshots' consumption reflect only the differences, not the whole files.

---

snap001 might have inbox.pst comprised of records 123, 124, 125, 126, 127

snap002 might have inbox.pst comprised of records 123, 125, 127, 128

I highlighted in bold the records shared between the two snapshots that reference the file inbox.pst.

So whether you restore from snap001 or snap002 determines which version of inbox.pst you will be restoring.

---

ArsTechnica has a great article about ZFS that serves as a primer:

ZFS 101—Understanding ZFS storage and performance

Learn to get the most out of your ZFS filesystem in our new series on storage fundamentals.

arstechnica.com

---

UPDATE: "Blocks" and "records" are used interchangeably, but technically with ZFS files are comprised of records, records are comprised of blocks. The block size is determined at pool creation with the ashift value, which by default is 4KB for spinning HDDs in TrueNAS to match the typical 4KB sector size. The larger the maximum record size, the more efficient is ZFS's built-in compression, as it has more "breathing room" to work with in an attempt to reduce the number of blocks that comprise a record.

Imagine you allow 1MB records, and the compression algorithm manages to reduce the number of 4K blocks that comprise a particular record from 256 down to 218. That's 85% the size of what would have been the original uncompressed record.

Really good infot thank you