Verify Cloud Sync Backup?

[hkelly]

Hello, I've searched around the internet and can't seem to find an answer to this.

What is a good way to verify that a cloud sync backup works and the files aren't corrupted?

The easiest solution is to restore into a different dataset, and check the files. That works, but not when the backup gets really big. I don't want to pull down 100s of gigs of data to verify a few files in the backup are the same as they were on my NAS.

Is there a better way to do the backup? I've read about duplicati, but have also seen issues with restoring and it not being very stable.

Thanks!

 

[AndrewShumate]

So really restoring and checking the file is the 'correct' way to test and verify a backup.

But as you mentioned it can take a lot of storage and be time consuming. So another way would be, depending on where you're backing up to, would be to use their API to hash the file uploaded and the local copy.

MD5 or SHA1 should be fine for this purpose as we're not really worried about collision issues for security purposes.

 

[hkelly]

Do you know if it's easy to get the MD5 or SHA1 from Backblaze B2? I can't find anything on their site about it.

I'm wondering if I could make a snapshot in truenas of the dataset, delete one of the files, do a pull/sync and then verify it?

For example, say I have a dataset that I backup with these files

• fileA
• fileB
• fileC

I make a snapshot of the dataset (in case the backup is corrupted :) ), write down the MD5 or SHA1 for fileB, and then delete fileB. After it's deleted, I can flip my push/sync to a pull/sync and run it. That'll restore fileB. I can then check the MD5 or SHA1 for the restored fileB and open the file to make sure it's ok.

I'm backing up music production, so it's big folders that I can easily verify. It's not a photo album or something where this would be extremely tedious.

Trying to avoid pulling down possibly 100gb every month or so to verify a couple sessions .

 

[AndrewShumate]

So I came across a Python script that would use the BackBlaze backup API. There is an API for B2, and B2 takes a hash of the file. I'm sure someone out there has written a script to use the B2 API to pull the SHA1 hashes and has the files in a directory and compare them...