Backblaze B2 and Cloud Sync Question

[briancmoses]

I saw that Backblaze posted a blog the other day, describing how to use the FreeNAS Cloud Sync to write data to Backblaze B2 (https://www.backblaze.com/blog/how-to-setup-freenas-cloud-storage/) and I was excited to try it out. I more-or-less mirrored what's described in the blog, except I used my own Backblaze Buckets and local paths from my FreeNAS Machine.

I manually kicked off my Cloud Sync task, but no files were ever transferred. I waited 20-30 minutes and checked on Backblaze and saw that I'd exceeded my Class C Transaction Cap for the day of 2500 free transactions. Out of curiosity, I added my credit card and triggered the Cloud Sync task again. Within seconds, I'd doubled my prior Class C transaction count, so I deleted the task and figured I'd spend some time to try and understand what I might've done incorrectly.

In looking at Backblaze's reporting, it seems the bulk of the Class C Transactions were split between b2_authorize_account and b2_create_bucket. Because of this, I'm concerned there might be a problem with how I set up the Cloud Credentials in FreeNAS, although I'm skeptical of that. In setting up the Cloud Sync task, I didn't have any issues seeing and picking my B2 bucket, I figured if I'd entered my Cloud Credentials incorrectly that I wouldn't even see my bucket listed.

Can any of you provide some guidance on why I'd see the Backblaze daily Class C Transactions immediately hit its cap when executing my Cloud Sync task? Would that explain why I'm not seeing any data transferred to Backblaze B2?

Naturally I'd sit down and write this when I'm not actually in front of my FreeNAS box, I'm running FreeNAS 11.1 but I'm likely behind by an update or two. If you have any questions about my setup, please let me know and I'll update my post with answers to your questions!

Edits:

1. My FreeNAS version is 11.1-RELEASE, I see a few Cloud Sync bugs in the changelogs since, sans any better options I may update and see if that helps at all.
2. I upgraded to FreeNAS-11.1-U6 tonight, but I haven't yet re-enabled my sync task due to time constraints.

 

[Liriel]

I am seeing the same behaviour as described above, with no data actually being transferred after triggering a manual run and an excess of Class C transactions.

I am up to ~112,000 Class C transactions today, equally split between b2_authorize_account and b2_create_bucket.

EDIT: I upgraded to U6 (from U5) and did not pre-create the destination folder in the bucket (I had done this previously), and used the default scheduling options (previously scheduled by own). Not sure which of these made the difference, but everything is working now and sync is proceeding at my max upload speed.

 

[diskdiddler]

Ok it's not just me.
https://redmine.ixsystems.com/issues/42683#change-278285

I set up a job, it worked fine, then I added a file, re-ran the job and it just didn't appear for a LONG time (we're talking 50KB, it should've uploaded in under 10 seconds)

 

[briancmoses]

I'm glad to see others in the same boat! I haven't tried since upgrading to U6, hopefully I'll get some time to try that this weekend.

@diskdiddler -- Am I understanding correctly from your bug that you're running the 11.2-beta2 and that yours magically started working later? Were you seeing the same high Class C transaction rate that @Liriel and I were both seeing?

Edit: Spelling is hard.

 

[diskdiddler]

I am unsure abou the transaction rate. I will need to take a closer look early next week.

 

[briancmoses]

It seems like I'm in @Liriel 's boat too, thankfully. After upgrading to U6 and re-creating my Cloud Sync task, I've been able to toss data both in the root of one bucket, and in sub-folders in a different bucket in different Cloud Sync Tasks.

 

[diskdiddler]

I've recreated my task and it appears to be working

 

[slamilton]

For anyone in the future, the fix is to use `--fast-list` to prompt rclone to use one API call to list all directory contents. See https://github.com/rclone/rclone/issues/1277 for details

That said, there is still an open issue with using encryption and backblaze together that `--fast-list` cannot solve, and as of writing if you're going to use crypt on top of backblaze you'll see an increase in C transactions. See https://github.com/rclone/rclone/issues/1640 for details