S3 Immutable Storage and TrueNAS SYNC Mode

[planedrop]

Hoping to get some info from someone who has already tried this (and if I can't find anyone that has I will probably give it a shot and report my findings).

Currently have a TrueNAS setup that backs up to an S3 bucket on a nightly basis. So far that portion of things works perfectly, backups are quick and files get removed from S3 as they should. However, this S3 bucket isn't setup in immutable/compliance mode yet and I'm curious if things will still work if I were to turn that on.

Does TrueNAS need to modify files by any means in order to perform the backup? Or is it only reading file names and overwriting them, deleting them, or adding new files? Does overwriting to S3 essentially create a new file and then delete the prior, or would write access to said file be needed?

It's not the end of the world if I can't turn this on but still is a nice feature and hoping to get some info on how TrueNAS behaves with it's S3 setup.

 

[dschuett]

I have just started a cloud sync with the same set up. With my testing, compliance mode works as intended. The TrueNAS sync task still copies new files, but since it is unable to delete or change any files because of the immutability on the S3 bucket, the sync status in TrueNAS will show "Failed". Kind of annoying, but since it does still copy the new files, that is not a big deal to me.

 

[planedrop]

I have just started a cloud sync with the same set up. With my testing, compliance mode works as intended. The TrueNAS sync task still copies new files, but since it is unable to delete or change any files because of the immutability on the S3 bucket, the sync status in TrueNAS will show "Failed". Kind of annoying, but since it does still copy the new files, that is not a big deal to me.

Hmmmm this is interesting because this is not what I'm seeing. I have had SYNC mode tasks both to Wasabi in Compliance mode and Backblaze B2 in immutable mode and it still shows successful like it should (and the files get marked for deletion and are kept according to retention policies). Maybe something else is causing yours to say failed? A locked file or something? Make sure to have snapshot first (or whatever it's called) in the Cloud Sync Task selected otherwise any files that are locked by another app won't sync.

I should have updated this post a while back when I got both those backups working with Success in the tasks section, they were failing though without having the snapshot option selected since often times my VMs would start a backup job to the TrueNAS while the TrueNAS was backing up to a cloud provider.

 

[dschuett]

Hmmmm this is interesting because this is not what I'm seeing. I have had SYNC mode tasks both to Wasabi in Compliance mode and Backblaze B2 in immutable mode and it still shows successful like it should (and the files get marked for deletion and are kept according to retention policies). Maybe something else is causing yours to say failed? A locked file or something? Make sure to have snapshot first (or whatever it's called) in the Cloud Sync Task selected otherwise any files that are locked by another app won't sync.

I should have updated this post a while back when I got both those backups working with Success in the tasks section, they were failing though without having the snapshot option selected since often times my VMs would start a backup job to the TrueNAS while the TrueNAS was backing up to a cloud provider.

I actually started using Wasabi, and I am seeing the same thing I mentioned before even enabling snapshot first. I really wish I knew what was causing it to think it failed.

 

[planedrop]

I actually started using Wasabi, and I am seeing the same thing I mentioned before even enabling snapshot first. I really wish I knew what was causing it to think it failed.

Interesting, mine have been working fine.

Check the cloud sync logs, that should give you an indication of what is going on (requires the use of the CLI): https://www.truenas.com/community/threads/where-are-the-logs-for-cloudsync.76179/

 

[dschuett]

Interesting, mine have been working fine.

Check the cloud sync logs, that should give you an indication of what is going on (requires the use of the CLI): https://www.truenas.com/community/threads/where-are-the-logs-for-cloudsync.76179/

Interesting. I am using encryption. I wonder if that has something to do with it. I have attached the logs.

 

[planedrop]

Interesting. I am using encryption. I wonder if that has something to do with it. I have attached the logs.

I'm also using encryption in my setup, are you encrypting file names or just the file contents?

What are all your settings for the upload (other than stuff to keep private of course like the bucket name)?

Looks like it's having some kind of permissions issue but not sure what would trigger that. IIRC the logs I saw about files being locked were different than Access Denied.

 

[dschuett]

I'm also using encryption in my setup, are you encrypting file names or just the file contents?

What are all your settings for the upload (other than stuff to keep private of course like the bucket name)?

Looks like it's having some kind of permissions issue but not sure what would trigger that. IIRC the logs I saw about files being locked were different than Access Denied.

Attached are all my settings from my TrueNAS and Wasabi.

 

[planedrop]

Attached are all my settings from my TrueNAS and Wasabi.

Thanks for the info here.

So a couple things, first I just looked at my Wasabi and realized I don't actually have compliance mode on, pretty sure I tested it in another bucket though without issues. I can say with 100% certainty that it works with Backblaze and their immutability.

In terms of the settings, one thing I would try is disabling file name encryption, I noticed some quirky behavior with that on and have since turned it off, I'm still encrypting the file contents just not the names.

 

[dschuett]

Thanks for the info here.

So a couple things, first I just looked at my Wasabi and realized I don't actually have compliance mode on, pretty sure I tested it in another bucket though without issues. I can say with 100% certainty that it works with Backblaze and their immutability.

In terms of the settings, one thing I would try is disabling file name encryption, I noticed some quirky behavior with that on and have since turned it off, I'm still encrypting the file contents just not the names.

Just an update, I have confirmed that it does indeed work with BlackBlaze with the same settings. I have reached out to Wasabi, to see if they can determine why this may be happening. I will post back here with their response.

 

[dschuett]

Attached is the response from Wasabi support. This actually makes sense, because I think versioning is on by default with BackBlaze. I even noticed that it makes a copy of the file in BackBlaze if you tried to delete it from the TrueNAS side. I will try turning on versioning on Wasabi to see if it stops TrueNAS from reporting a failed job.

 

[planedrop]

Attached is the response from Wasabi support. This actually makes sense, because I think versioning is on by default with BackBlaze. I even noticed that it makes a copy of the file in BackBlaze if you tried to delete it from the TrueNAS side. I will try turning on versioning on Wasabi to see if it stops TrueNAS from reporting a failed job.

While the response overall makes sense, I don't think this is the best way to do things and how Backblaze does it is much better. It's fine for something to be immutable, but it should still appear as if deleted when the TrueNAS deletes it, this way TrueNAS thinks it deleted it and then it just stays in place based on the compliance/immutability settings. TrueNAS shouldn't even realize that this is a non-editable bucket.

Overall I was actually disappointed with Wasabi, pricing is good until you realize that all data must be kept for 30 days, then it quickly gets out of hand.