TrueNAS 12 Cloud Sync to Office 365 - Sizes differ

[Bondsc]

Hi,

I'm using TrueNAS 12.0 U5.1 and I'm trying to perform a cloud sync to a 1TB OneDrive on my Office 365 Business account. I'm not a business - just a home user.

My specs:
I have 2 x 2TB drives in a RAID 1 shared as "share1" (SMB)
I have 2 x 6TB drives in a RAID 1 shared as "share2" (SMB)
At some point, when I have the money, I'll replace the 2 x 2TB with 2 x 6TB and setup RAID-10.
It's running on a SuperMicro motherboard with a Core i3-9100 @ 3.6Ghz and 16GB ECC RAM.
It is mainly used as a home SMB server plus Plex in a jail and it runs perfectly. It has been running for just over a year and previously I used FreeNAS 9.x.
Tons of free RAM. Services using 5.5GB RAM. ZFS cache is 9.5GB (I'm reading this from the dashboard).

I have some success with cloud sync (some folders are syncing OK) but a number of problems:

Problem 1: I created a cloud credential via the GUI by selecting the "Microsoft OneDrive" provider and signing in to Office 365. In the "Drives list" drop-down only one drive appears "DOCUMENT_LIBRARY - xxxx" and it took a while to work out that this is my Office 365 SharePoint teams site "Documents" folder. My actual OneDrive doesn't appear in this drop-down list and I have no idea how to add it. Anyway, my SharePoint document folder has 1TB storage so I continued with this.... (but I'd like to know how to access my actual OneDrive).

So I now setup a "Cloud Sync Task" also via the TrueNAS web GUI and there are a load of options here but I picked:
Direction: Push (I want to backup from TrueNAS to Office 365 and TrueNAS triggers the sync via this task - a time schedule)
Transfer mode: Sync (If I add a file on TrueNAS I want it copied to Office 365, if I delete the file I want it removed from Office 365. I don't plan to change files at the Office 365 end - i.e. I'm using this as a backup for now).
Remote folder: /TrueNAS sync/XXXX (I created a folder called 'TrueNAS sync' directly in my Microsoft SharePoint teams site "Documents" folder.
Source: A folder on TrueNAS (XXXX)
I left every other setting alone.

So far so good, this works :) Folder XXXX on TrueNAS is synced to /TrueNAS sync/XXXX in Office 365 SharePoint and I've confirmed the files "made it".

But XXXX is just one folder of many on True NAS that I want to sync so I setup some additional Cloud Sync tasks to backup different folders at different times (so I now have about 5 Cloud Sync tasks in the TrueNAS GUI). This backs up about 10% of my NAS - just the folders that really matter and that I want to Sync (I only have 1TB storage in Office 365 vs 8TB in TrueNAS).

Problem 2: Most folders start syncing but eventually report a status of "Failed" (in the Cloud Sync tasks GUI). If I download the logs, I see lots of errors like:
2021/09/28 06:30:03 ERROR : On Top of the World.docx: corrupted on transfer: sizes differ 16409 vs 22112
2021/09/28 06:30:03 INFO : On Top of the World.docx: Removing failed copy
2021/09/28 06:30:03 ERROR : Among us SUS lyrics.docx: corrupted on transfer: sizes differ 23988 vs 32629
2021/09/28 06:30:03 INFO : Among us SUS lyrics.docx: Removing failed copy
2021/09/28 06:30:03 ERROR : Viola/Link to cripple creek viola for CMA tune.docx: corrupted on transfer: sizes differ 12225 vs 18650
2021/09/28 06:30:03 INFO : Viola/Link to cripple creek viola for CMA tune.docx: Removing failed copy
I really have NO IDEA what to do about this.

Problem 3: I start getting emails from Microsoft (some automated thing) telling me that thousands of files are being deleted on my SharePoint site and that they are all going into the recycle folder. Now I presume this is because of the thousands of "Removing failed copy" entries in the above log. Trouble is, they aren't really being deleted - they go into the Office 365 recycle bin and still count towards my Office 365 storage (so I have to manually delete them from the Office 365 end or I run out of storage real quick). When I stop syncing with Office 365, I stop getting the emails from Microsoft.

This error is only happening for about 10% of the files (a guess). LOTS of files have synced OK and look OK at the Office 365 end.
I cannot see anything special about the 10% of files with a problem - they open from TrueNAS just fine and don't appear to be corrupted.

Because 10% of the files aren't synced, every night when my daily backups run, they sync for hours and hours (because every night they are continually trying to sync the remaining 10%).

So this boils down to 3 problems:
a) Why can I only see my Office 365 SharePoint document share from TrueNAS and cannot see my actual Office 365 OneDrives shares?
b) Why do I get these corrupted on transfer errors? This is the main problem.
c) Why do I keep getting these emails from Microsoft (my guess is because of b) removing the failed copy).

A few last notes:
I have a pretty decent broadband connection (for the UK). About 300Mb/s down and 50Mb/s up. Very stable and I reliably get these transfer speeds.
The NAS seems very stable. I have no other problems and no reports of any corruption from the users (5 people in my family).

So can anyone see what I've done wrong? Or have seen this problem before? Should I raise a bug?

Thanks

Simon

 

[jgreco]

Well, hopefully you mean you have a ZFS mirror. If you have actually set up RAID1, such as with a hardware controller, you need to swap that out, see this article.

If you simply misspoke and meant that you had a ZFS mirror, that's fine of course, but we do encourage people to say the correct things, because accuracy on a technical forum helps convey issues. You're encouraged to look at the Terminology and Abbreviations Primer.

I don't have any comments on the SMB stuff, sorry.

 

[Bondsc]

Yes sorry I misspoke / used incorrect terminology.
I have two storage pools "Pool1" and "Pool2".
"Pool1" has a Dataset called "DataSet1" and an IOCage. This is 6TB (2 x WD Red 6TB) and is a mirror. Compression is lz4.
"Pool2" has a Dataset called "DataSet2". This is 2TB (2 x WD Red 2TB) and is a mirror. Compression is lz4.

I also have a boot pool which is on a 256GB SSD.

Some of my Office 365 cloud syncs are from a folder or folders in DataSet1 and some are from DataSet2.

The status for both pools lists errors as 0.

Simon

 

[elorimer]

It's been a while since I did this, but OneDrive personal and OneDrive Business are two different things, and if your drive is not "Business" but instead "Document_Library" you have a problem, I think.

Also, because you are syncing, I'd be worried about those deletions--it means that things on your Business drive that are not on your TrueNAS are being deleted. So I would stop right now and sort out whether you are deleting stuff you mean to keep.

I pull from my OneDrive Business down to my TrueNAS as a backup for my OneDrive Business. It may be that what you want is to push up to OneDrive Business rather than sync.

 

[Bondsc]

Hey,

Thanks for the reply.

I'm 99% certain that the deletions are happening at the Office 365 end because of the failed sync (due to size difference).
So the sync is doing the following:
Copy local file A to remote file A
Oops, something wrong
Delete remote file A
Repeat

So I don't appear to be losing any local data (i.e. data on the TrueNAS)

Yes I'm definately OneDrive Business (I pay monthly for this service) and yet the only drive which appears in TrueNAS is marked "DOCUMENT_LIBRARY - xxxx" which is still part of my Office 365 Business subscription, but part of the SharePoint site, not the OneDrive for Business. So the files (well the ones which copy OK) are making it to my Office 365 Business, just not exactly into the location I'd planned. But I can live with the files being in this alternative place because my goal is to have an additional backup (I'm not too bothered exactly where), but obviously the sync problem is causing a huge problem.

So as you suggested, I've changed the transfer mode to COPY (from Sync) and I've manually started the Cloud Sync (copy) and I'll see what happens (this particular folder is a few GB so it won't take long to copy).

But I don't think SYNC is dangerous (from the perspective of losing files on my OneDrive) because the wording (in live help) says:
"SYNC: Files on the destination are changed to match those on the source. If a file does not exist on the source, it is also deleted from the destination".
This makes no mention of files being deleted on the source, so I'm hopefully safe?
The docs for COPY say:
"COPY: Files from the source are copied to the destination. If files with the same names are present on the destination, they are overwritten".
My experience with this (in the past) has been problematic.
For example if I have file A on the source, I get file A on the destination. Good so far....
If I rename file A on the source to file B on the source then I get file A and file B on the destination (because it only copies, never deletes).
Gets confusing after a while.....

Simon

 

[Bondsc]

Oh and I've re-read what you said:
>> "it means that things on your Business drive that are not on your TrueNAS are being deleted"
Yes, agreed, but I'm syncing to a new folder "/TrueNAS sync/" on my Office 365 Business in the ShareFile document library (which I've previously never used), so there is nothing in that area to lose....
My normal OneDrive for Business (which I DO use) is fine. It still has the same stuff in it and TrueNAS seemingly cannot see it.

But I still have two main issues:
1) Why can TrueNAS only see my ShareFile site and not OneDrive for Business? I can see both quite clearly from the Office 365 portal and my laptops can see the OneDrive for Business just fine as well. I'm using the same account in all cases.
2) Why am I getting these sync errors (file size differences)?

Simon

 

[Bondsc]

Nope failed again. Exactly the same errors in the log. So both COPY and SYNC have the same problem.

In this particular folder there seem to be about 74 files (out of 361) which fail to copy with this same error (sizes differ).
Other folders vary (different percentages of failure but the same kinds of errors).
One folder (with a similar amount of data in it) copies perfectly.

I've also just read through more logs and there are two types of error.

Type 1: File sizes differ:
e.g.
2021/09/28 06:18:23 ERROR : 11 plus/Test papers/CGP GL Bucks tests papers 2018/Synonym finder & vocabulary/Bank Vocab Sheet.doc: corrupted on transfer: sizes differ 37888 vs 50688
2021/09/28 06:18:23 INFO : 11 plus/Test papers/CGP GL Bucks tests papers 2018/Synonym finder & vocabulary/Bank Vocab Sheet.doc: Removing failed copy
2021/09/28 06:18:23 ERROR : 11 plus/Test papers/CGP GL Bucks tests papers 2018/Synonym finder & vocabulary/Aphacards.doc: corrupted on transfer: sizes differ 129024 vs 142336
2021/09/28 06:18:23 INFO : 11 plus/Test papers/CGP GL Bucks tests papers 2018/Synonym finder & vocabulary/Aphacards.doc: Removing failed copy


Type 2: Hash differ:
e.g.
2021/09/28 06:18:19 INFO : Images/Simba/PC010066.JPG: Removing failed copy
2021/09/28 06:18:20 ERROR : Images/Simba/PC010065.JPG: corrupted on transfer: QuickXorHash hash differ "755610faa59a2f3309d65ade9f372b5877471bcb" vs "f07e12b1abd22cdf54405f1fe798b00e84c122ee"
2021/09/28 06:18:20 INFO : Images/Simba/PC010065.JPG: Removing failed copy
2021/09/28 06:18:21 ERROR : Images/Simba/PC010067.JPG: corrupted on transfer: QuickXorHash hash differ "389081083b7025fdbf74e45988421030bf46c8c2" vs "fb3dcac50b677a41a119b0681c2dce64b97ffa1b"
2021/09/28 06:18:21 INFO : Images/Simba/PC010067.JPG: Removing failed copy
2021/09/28 06:18:21 ERROR : Images/Simba/PC010068.JPG: corrupted on transfer: QuickXorHash hash differ "f53d1e8047a93129eaa38b5e07461fd812894728" vs "4a7c95fe98e82a30cc67e5cf9c4e1e9c4daa3586"
2021/09/28 06:18:21 INFO : Images/Simba/PC010068.JPG: Removing failed copy

(presumably in this latter example, the file sizes were the same so a hash comparison was performed to check the contents and that failed). It automatically removes the failed copy so I can't have a look at the broken copy.

Oh and I forgot to mention. If I use my PC to copy the file from the NAS to the ShareFile site (by dragging from Windows Explorer NAS share to the ShareFile website) then the copy works fine... So the problem appears to very much be a TrueNAS problem.

Anyone got any ideas?

For now I've disabled all Sync tasks.

Simon

 

[Bondsc]

One more thing..... It's the same files each time. Not random.
I've had a really close look at the problem files and I can't see anything wrong with them:
It's a mixture of data types (.jpg, .docx, .pptx, .doc).
I can open the source files without a problem, they aren't locked. Some are years old, some were created this year.

 

[Bondsc]

OK. I had some Google Drive storage (not much but I'm about to purchase more) and so I've just configured TrueNAS to sync some of these problematic folders to Google Drive. Worked perfectly first time.

So it looks like the issue is related specifically to TrueNAS performing a cloud Sync to Office 365 Business.

I'd still prefer to Sync to Office 365 Business because I have a lot of Microsoft Cloud Storage that I automatically get as part of my Office 365 subscription (which I use for many things so I'm not about to cancel it). So does anyone have any suggestions? Do I raise a bug report?

Simon

 

[elorimer]

The fact that it worked for GDrive makes me think it is a credential issue rather than a TrueNAS/rsync issue. Look in Cloud Credentials for your One Drive credentials, and click edit to look at them. There is a field for "Drive Type", which should give you a choice of "Personal", "Business" or "Document_Library". It needs to be "Business".

 

[Bondsc]

Hi,

Thanks for your response.

Well most of the files are syncing so I don't really understand how that can be credentials related, but I guess credentials could be the cause of it not seeing all of my Office 365 drives, but I'm using the same Office 365 account that I use from my Windows PC and that can see everything (plus manually copy files from the NAS to the SharePoint Documents folder and / or the OneDrive.

But on the dropdown you are referring to, the options are "Personal", "Business" and "Document_Library" but this option get reset when I pick the Office 365 drive from the "Drives list". i.e. I pick the "DOCUMENT_LIBRARY - xxxx" drive (the only drive on offer) and when I do that, it resets "Drive account type" to "Document_Library".

Anyway, I manually changed the drop-down to "Business" clicked Save and tried re-syncing. Same result.

I also tried using my Office 365 admin account credentials (when setting up the TrueNAS Cloud Crtedentials) instead of my normal user account and same result again.

If this problem is credentials, I don't see how I'm doing it wrong... TrueNAS asks for my Office 365 credentials, I get an Office 365 login popup, I fill it in, done. I get a list of drives in a drop-down and I only see the document library. If I do the same with the same accounts on Office.com then I see the OneDrive for Business and the SharePoint site.

So from my perspective, this looks like a TrueNAS bug. I'm pretty sure TrueNAS is using a completely different API when transferring files to Microsoft as opposed to Google, so the problem could be an issue with the TrueNAS implemention of the OneDrive for Business API for copying files (I don't know which API it uses - I haven't researched that).
I do know that the API for OneDrive for Business is different to the API for OneDrive personal so I'm wondering how people using TrueNAS are synching to OneDrive for Business?

But obviously I'd appreciate suggestions / ideas because I don't know where to go from here. I could purchase more Google cloud storage because this is working with TrueNAS but that is an ongoing monthly cost. I could stop using TrueNAS and use Windows or Linux because they can see my Office 365 Business drive, but I'd rather stick with TrueNAS as it has worked so well for everything else and it really does seem to be a great product.

Thanks for your help so far,

Simon

 

[Bondsc]

>> I do know that the API for OneDrive for Business is different to the API for OneDrive personal so I'm wondering how people using TrueNAS are synching to OneDrive for Business?

I meant to say "how many people"... I'm asking because I'm guessing it might be a small number? i.e. Using TrueNAS Cloud Sync + OneDrive for Business (as opposed to personal).

 

[elorimer]

I think it might be a large number. If you are in the Office365 environment anyway, OneDrive is cheap; it's like GDrive if you are in their environment. But for most folks who have it through their work I'm guessing syncing is disabled, because of property rights issues.

It might be fruitful to work through the Microsoft troubleshooting steps for OneDrive sync.

I remember a while ago having my OneDrive rsync fail on one particular file. I think I moved it out of the sync dataset, synced, then moved it back.

 

[Bondsc]

OK thanks. I'll look through the troubleshooting steps.
Short term I think I'll also purchase some more Google storage - it's cheap enough and I need that extra backup while I'm investigating the problem.
I also backup to external USB drives, but they are "onsite" (i.e. at home) and I want to protect from things like house fires where I'd lose all storage. I was considering a safe to put in the workshop but I need to research whether you can keep drives in locations where it can get cold and damp.

Thanks again,

Simon

 

[Bondsc]

So a quick update....
I've not managed to resolve the sync to Office 365 / OneDrive Business problems - I still can only see the SharePoint documents folder from TrueNAS and I still get the Sync errors.
I've read a bunch of Microsoft guides and not found any problems with my setup or solutions to these problems, but I don't think there is anything wrong at the Office 365 end - I can access everything from the Office 365 portal just fine and I have two PCs using OneDrive connecting to my OneDrive for Business just fine. Same with my Android phone.... The problem appears to only be between TrueNAS and Office 365 / OneDrive for Business.

So I purchased 2TB for my Google Drive and I'm now syncing to that. In total 1.5TB is syncing and it has been working great for a week with no sync errors.
So I've removed all of my Office 365 OneDrive sync tasks and credentials from TrueNAS and I'll just sync to Google for now.

If I ever find the cause (or if a TrueNAS release fixes it), I'll reply with an update.

 

[elorimer]

If I ever find the cause (or if a TrueNAS release fixes it), I'll reply with an update.

I came across this post. I wonder if that would help.

 

[Bondsc]

Yes I think that exactly describes the issue. Thanks for letting me know. It requires me to edit a Python file directly (which I'm fine with) but I bet that will play havok on the next TrueNAS upgrade.
But I'll try it, see if it works, and perhaps put in a feature request for it to be a GUI option (because it seems to be a OneDrive "feature" that TrueNAS will just have to deal with)