Backblaze B2 - Large Volume of Class C Transactions

[mmltd]

Hi: first time here and hope I'm in the right bit of the forum. I've done a few searches on this and there are a few hints of my problem but nothing clear. Happy to RTFM if I can find an M to R!

I've been using the B2 backup since about February this year and all has been well until a couple of months ago. I have suddenly started racking up huge Class C bills with literally millions of Class C transactions per day which has cost my business a lot of money.
Now this is partially my fault (no limit on Class C transactions for reasons I don't understand but that's down to me) and it came to light when I was going through my credit transactions which on this account I do quarterly - after all what could possibly go wrong!

Does anyone else have issues with huge amounts of Class C transactions? My number of files etc have been growing more or less linearly throughout the period (in fact the rate of growth has reduced somewhat) and there have been no other obvious changes. I have about 11TB stored in my FreenNAS bucket at about 6.2M files (my apps generate a lot of crap!). Meanwhile the Class C suddenly jumped by a factor of 315 in one month (three hundred and fifteen times bigger!) and the doubled again the next month.

I have one suspicion, I think (and I can't be 100% about the date) the huge jump in Class C coincides with me updating to 11.1-U6. I also switched on snapshots but am sure that was a long time before this issue started (but was after I started using B2).

For the meantime I have switched the cloud sync off but obviously that only leaves me with local backups which is not good. This seems to have stopped the massive volume of Class C transactions, which suggests FreeNAS is the issue. I've also emailed Backblaze to see if they have any ideas.
The local backups do not use any part of FreeNAS as I use an external app which just backups the files from an SMB mount so unlikely to be any weird interaction there.

Has anyone else had this behaviour in Class C transactions?
Can someone point me in the right direction to start solving this?

Happy to mess around on the command line, just not sure where to start looking! I used to administer an Ultrix 32 system but that was twenty years ago......

Thanks

 

[dlavigne]

Were you able to figure this out?

 

[mmltd]

Nope and clearly no one else has any thoughts.

I've capped the C- transactions for now but left the backup running. I've still got full backup on local and will experiment with a fresh bucket over Christmas. Hopefully my B2 backup will catch up a little on the capped transactions over the holiday while I won't be generating much new data. Rather discomforting, but I can't afford hundreds of dollars a month!

 

[theman00011]

I'm also having this problem so it apparently still exists. I set up a brand new cloud sync task in version 11.2-RELEASE-U1 and set a cap in Backblaze and it constantly runs into the cap, usually minutes after upping it. Other backup software is able to sync without abusing constant b2_list_file_names calls which are what is eating up the class C transactions so something needs to be done. It's been posted before and I believe a ticket was made but it doesn't seem to have been fixed.

There seems to be a PR from 6 months ago adding a supposed fix but I guess it just never made it in? I'm not sure.

https://github.com/freenas/freenas/pull/1502

 

[dlavigne]

Looks like it is slated for 11.3: https://redmine.ixsystems.com/issues/27364 .

 

[theman00011]

I see. I feel like this needs to be pushed into a minor update instead of 11.3 because it basically renders B2 cloud sync useless with the GUI. It just isn't feasible to pay for who knows how many transactions just because the fix isn't out yet. With the PR already made and merged, it should be trivial to add it to the next minor release and test it so that people can use B2 cloud sync through the GUI again.

 

[dlavigne]

Please add a request for that as a comment in that ticket. The dev will need to investigate how much churn that middleware change would introduce to the 11.2 branch.

 

[theman00011]

How can I add a comment to the ticket? I don't see an option anywhere to do so.

 

[dlavigne]

Make sure you are logged into the bug tracker then click Edit on the ticket.

 

[Dreded]

Looks like it is slated for 11.3: https://redmine.ixsystems.com/issues/27364 .

that looks great... but where does rclone currently read its command line options from so I can add --fast-list

I had to do this ages ago and now that I upgraded to FreeNAS 11.2 I can no longer do backups due to this issue.. to give context if I just let it run wild it would cost us ~ $73/month as opposed to the $5/month if it has the --fast-list option(lots and lots of folders with 1 or 2 files its annoying but it is what it is)

EDIT: the file I modified before was /usr/local/lib/python3.6/site-packages/middlewared/plugins/backup.py but it seems to be structured quite different now

 

[Tim1962]

I am on 11.2 and recently started using BB B2 sync, and started getting relatively large numbers (50-100,000) of C Class issues.

I then chunked up the backups into smaller daily sections rather than one almighty weekly job and C Class charges have disappeared and rarely exceed 100-200 daily.

Worth a try? purely n=1 evidence

 

[Dreded]

I am on 11.2 and recently started using BB B2 sync, and started getting relatively large numbers (50-100,000) of C Class issues.

I then chunked up the backups into smaller daily sections rather than one almighty weekly job and C Class charges have disappeared and rarely exceed 100-200 daily.

Worth a try? purely n=1 evidence

I used to do 2 daily.. now I have cut back to 1 daily and not backing up non critical to keep my costs down to $30/month if I understand the issue correctly its that for folders only containing 1 file it will count as a transaction as it does each folder individually... where adding the --fast-list option on a bucket style system will always return 1000 files per transaction even if those files are spread across 1000 folders.. so 1 instead of 1000 class C transactions used.

I am probably going to resort to cancelling the integrated backup and making a cronjob do the backup for me, as my python skills are almost non existent and I cannot figure out where to add additional command line arguments in the backup.py file(I know it can be done and is relatively easy I just cannot follow the logic)

 

[Soloam]

Damm! I had the same problem, 200 usd! Is there a way to cancel this and delete everything and avoid this bill? I can cancel my card! This is ridiculous!

 

[Soloam]

Ok, I talked with them and they are going to make me a exception and a partial refund. This is a major issue and I hope it's solved asap

 

[Hazimil]

I've written a script to manage my Backblaze B2 backup, which you may find useful:

https://www.ixsystems.com/community...p-nas-data-to-backblaze-b2-cloud-storage.123/

Jonathan