Backup tips (and apps)

[sfatula]

Unfortunately, No they are not and that is a very common mistaken.

Which is precisely why I said they are a form of backup and it was not useful in disaster situations. But that's a bad word that didn't mean what I wanted (form), not my intent. Without having to type a lot more, I was trying to say as I was rushed that it does serve one backup purpose in that one can use it to restore versions of files in a pinch very easily. That is one purpose of a backup, but no, it's not a true backup to IT folks, I agree. Though it IS a backup to English students as that's what backup means in the dictionary. So, this is always a problem as using the word without defining your meaning of it can mislead folks. Strictly speaking of the meaning of the word, it is a backup. But I did say not usefuil for disasters which I thought clarified it. Apparently you thought otherwise.

Snapshots can indeed be stored on other machines via replication. As the guy has another machine, I was not clear where he was putting them. One can remove them from the machine they were taken on, so, they can be stored on other machines. Not familiar with QNAP so not sure if that's even possible. I realize he didn't use the word replication.

 

[Heracles]

Strictly speaking of the meaning of the word, it is a backup.

No because by definition a backup must not be the same as the original.

When a data is modified (deletion being a kind of modification), snapshots turns the new state to a new original data and the previous original data remains the same original data as it was.

So snapshots does not add any extra to any orignal. They only prevent an original data to be replaced by another original data. The two original datas will co-exist. But as long as you have active data and passive data (snapshots) you have only originals and not a single copy of anything.

In that way, to turn copy=2 or copy=3 in ZFS would be closer to a backup because it will create an actual copy of the data. Both copies will be handled the same way in the same ZFS structure but at least, they will be actual copies.

But because the 2 copies are so tightly linked as a single one, they too do not deserve to be called backups.

 

[sfatula]

No because by definition a backup must not be the same as the original

When a data is modified (deletion being a kind of modification), snapshots turns the new state to a new original data and the previous original data remains the same original data as it was.

So snapshots does not add any extra to any orignal. They only prevent an original data to be replaced by another original data. The two original datas will co-exist. But as long as you have active data and passive data (snapshots) you have only originals and not a single copy of anything.

In that way, to turn copy=2 or copy=3 in ZFS would be closer to a backup because it will create an actual copy of the data. Both copies will be handled the same way in the same ZFS structure but at least, they will be actual copies.

But because the 2 copies are so tightly linked as a single one, they too do not deserve to be called backups.

Yes, IT definition. Merriam Webster: "a copy of computer data"

We agree, I promise you, I even use the 3 copy rule. You are merely reading what I wrote in a manner you wish to read it in.

But I disagree with your contention that new copy overwrites the old. You never used .zfs folder? You can easily restore any version of any file you have snapshotted to any version of any snapshot you retained.

 

[Davvo]

Snapshots referenced existing data that if compromised makes snapshots useless.
Backups are exact copies of data stored outside the system.

 

[sfatula]

Snapshots referenced existing data that if compromised makes snapshots useless.
Backups are exact copies of data stored outside the system.

Again, what I originally wrote, you guys are trying to make a point when there isn't one. That's exactly why I wrote they are not useful in disaster situations. Would someone not wonder why a "backup" is not useful in disaster situations? I would. Fully aware what us IT folks call a backup. I think you are triggered by the use of the word backup. That's why I clarified it. I get it, I know you think people might read it and see "backup". But if they continue to read, they'll see what I wrote in the same sentence. Perhaps You are suggesting not to use the word backup. It's 100% clear to me but I can understand your point.

Of course, backups might be offsite, but if they are compromised, rot, etc., they make the backup useless. Thus the idea of making more than 1 copy, I actually have 2 offsite copies (and other copies). But local snapshots very useful for restoring a stupidly purged file.

 

[Davvo]

The referenced data can be invalid not only in disaster scenarios, but I get your point.

 

[Heracles]

You never used .zfs folder? You can easily restore any version of any file you have snapshotted to any version of any snapshot you retained.

Did you study how that folder (and ZFS) works ? ZFS is copy-on-write, that is no matter snapshot or not, a new write will produce a new copy. What snapshots do is that they prevent the old copy to be removed once that new write is done. And that is what is referenced under the .zfs folder. Yes it is easy to access but again, what is there is the one and only copy of an original data which itself is different from the other original data available in the rest of the ZFS environment.

The content of the .zfs folder is not a separate or an extra copy of what is available somewhere else in the system.

you guys are trying to make a point when there isn't one.

There is one : by using the word backup for this, you confuse new users and contribute to the confusion about what backups are and what they are not.

The same way the term RAID creates its own kind of confusion.

By insisting that the word Backup is appropriate here, you insist to maintain that confusion and that confusion will translate to data loss for many users.

 

[sfatula]

Did you study how that folder (and ZFS) works ? ZFS is copy-on-write, that is no matter snapshot or not, a new write will produce a new copy. What snapshots do is that they prevent the old copy to be removed once that new write is done. And that is what is referenced under the .zfs folder. Yes it is easy to access but again, what is there is the one and only copy of an original data which itself is different from the other original data available in the rest of the ZFS environment.

The content of the .zfs folder is not a separate or an extra copy of what is available somewhere else in the system.



There is one : by using the word backup for this, you confuse new users and contribute to the confusion about what backups are and what they are not.

The same way the term RAID creates its own kind of confusion.

By insisting that the word Backup is appropriate here, you insist to maintain that confusion and that confusion will translate to data loss for many users.

Yes, again, I am fully aware how the .zfs folder and snapshots works. You are clearly not hearing me. I was merely pointing out it's one common use of backups and useful for those situations.

Ok, so, you are policing my use of the word backup and suggesting that I may be confusing others. I acknowledged that point. And I certainly agree on the use of the word Raid5 when people may (or may not) mean Raidz1. I will endeavor to try and use the word backup in an IT manner not a dictionary manner. But am far from perfect, I might incorrectly use it elsewhere accidentally.

That being said, I think you are a little overly enthusiatic about users losing data due to my non IT use of the word. Nevertheless, I do (as I already said) see your point.

If someone else had written they are backing up their system using snapshots, you would have found me correcting them. Words do matter.

 

[danb35]

Re-using your logic, mirrors or RaidZx would also be "kind of backups" when they too are clearly not.

Yes, they are, by your own definition.

So I will rather avoid confusion and define a backup as a different copy or the original. That way, snapshots, mirrors and RaidZ are clearly no backups.

A mirror contains two (or more, in the case of a >2-way mirror) copies of the original, thus it meets your definition. RAIDZ may not contain a second copy as such (or it might, depending on the relationship between record size and file size), but allows a second copy to be reconstructed. Heck, setting copies=2 on a dataset meets your stated definition of a backup (for data written after that setting was made). A snapshot may or may not meet your definition (depending on what's happened to the data in question since the snapshot was taken).

The problem with such mottos as "RAID is not a backup" is that (1) they depend on an overly-specific definition of a "backup", and (2) they ignore the question of what the backup is supposed to protect against. Some possibilities:

• Inadvertent deletion or modification of a file
• Bad sectors/UREs on a drive
• Drive failure
• Major system failure
• Destruction of the system
• ...and the building or room housing it
• ...and the geographical area where it's located

Snapshots protect against the first; RAID (in any of its forms) doesn't. RAID can protect against the second and third; snapshots don't. A periodic copy of the data onto another pool in the same system can (if properly versioned) protect against the first three (and clearly meets your stated definition of a backup), but probably not the fourth, and certainly the fifth and following scenarios. An appropriately-versioned, periodically-updated copy kept offsite may protect against all of these, depending on how far offsite it's kept.

I think this is a much more helpful way to look at the problem, because I don't ultimately care whether there is "a different copy of the original;" I care whether I can get my data back after $BADTHING happens. And at some point, the cost of protecting against $BADTHING is going to outweigh the risk of $BADTHING.

 

[sfatula]

Well said. So, using snapshots on a zfs mirror or raidz (the combination), mostly protect against 1-3 on your list. But you used the word RAID, lol. He likely meant offsite though.

Each of my "techniques", didn't say the other word, have a different purpose so I agree with breaking down the various things you are protecting against. And snapshots do exactly what I want locally, which is ability to recover a file from some previous version. They also serve to feed an offsite backup copy, where I can also recover files by using the snapshot.

My offsite backup copy via replication is way out of my geographical area, I'm sure Colbert OK is a prime target for nukes you never know. But, the bank vault is likely safe enough where I have another copy. And I have the Kopia copy remotely as well. And I have a few downloaded copies of certain data just for convenience to make recovering quicker. And I have a document on all of this so I know how to proceed quickly in a disaster. And....

 

[DarkCorner]

If I wasn't clear, I'll say it again.

1. On TrueNas I'm already taking more Snapshots.
2. I also have RSync between two TrueNAS (and also several Debian server), so I'm familiar with how to do this.
3. With 1MB of upload I cannot do Full Backups. There are files that are 20GB large and a VM can be 180GB large, in a single file.
4. It's not about backing up between two locations, but between a TrueNas in the office and the Cloud? How do I bring the NAS to the Dropbox headquarters to do the first full? Come on, let's be serious.
5. I need to make a backup between a TrueNAS and a QNAP, 30cm distant and connected on the same switch. I tried all the services.
6. SFTP is used to transfer files, not a backup where we manage incrementals and periodic rotation.
7. As I was saying, I wasn't even able to install the apps, so I not only have a problem with two RSyncs not talking to each other, but also with apps not installing.

Now I'm doing this using FreeFileSync on Windows. Every day I read the shared folders on TrueNAs and write to the shared folders on QNAP. It's not a Backup, but at least I only transfer the differences.

If I could do it directly from TrueNAs at least it wouldn't take up the LAN bandwidth.

 

[sfatula]

1. I need to make a backup between a TrueNAS and a QNAP, 30cm distant and connected on the same switch. I tried all the services.
2. SFTP is used to transfer files, not a backup where we manage incrementals and periodic rotation.
3. As I was saying, I wasn't even able to install the apps, so I not only have a problem with two RSyncs not talking to each other, but also with apps not installing.

If I could do it directly from TrueNAs at least it wouldn't take up the LAN bandwidth.

Then you need to post the steps taken and the errors you are getting so that help can be provided. I would expect a backup to Qnap can be done but who knows where to start without details.

The way I delt with my 1.2 mbps upload speed that I had for a few months (but don't any more) was this. If I had a VM or whatever and there was no way that could possibly be uploaded, I didn't. I backed up the most critical files offsite, and even if the VM was critical I skipped it. So, I tailored my backup set to something where an incremental could be done over night. It worked fine. Sure, my VM (in your case) couldn't be backed up, so, it was backed up via some other method. You don't have to backup everything with every method necessarily.

My nightly Kopia incremental backup runs < 5 minutes. It doesn't include my 99% static media library of course, that has a much different technique.

Regarding looking for backup apps, have you considered the Truecharts catalog given you don't want to use docker images?

 

[PhilD13]

A few methods and mileage varies on the various methods and I am no expert, but recently had to connect TN Scale and QNAP together to copy 25TB of data to the TN Scale server (new main server with QNAP as becomming the new backup (copy of) data server). This is an outline of what I found out and it may help guide you some. QNAP even states that their QTS systems are not compatible with their new Hero (ZFS) systems and transferring files between the two is not going to work well. TN, I found somewhere says it is best done from a different computer using a copy program shuch as RoboCopy (now defunct) or similar if going to/from QNAP.

QNAP does have Rsync and it also has a backup program builtin called "Hybrid Backup Sync" app that can be installed from the apps. You select the Sync tab in the Hybrid Backup Sync app and you can setup a oneway Sync job (uses Rsync) to pull the data from the TN Scale server.

If you want to use an Rsync task from TN Scale side you need to setup the data protection sync task using ssh and the ip address of the QNAP server. Push the data and check only the Archive option. If the job tends to fail (I don't think QNAP's and TN's rsync are fully compatible) try unchecking all options (archive).

You will need to be on version 5+ of QNAP's QTS operating system on the QNAP server as you have to add the public ssh key to the QNAP user and (QTS version 4+ does not have that option). if you are going to do rsync from TN Scale to QNAP or TN will not connect to QNAP. Also recommended to use the same user and password on both machines. If you are on version 4+ of QTS, then either you have to manually add the key using the command line for the user, use QNAP to pull the data from TN, or use an external backup program on another computer to transfer files between the servers.