Cloud Sync - Azure block list limit with files larger than 200GB

sanstey

Cadet
Joined
Oct 5, 2017
Messages
2
I'm running into a problem syncing files larger than ~200GB (50,000 blocks) to Azure blob storage. I've determined this is due to the default rclone block size being 4MB. I don't see any GUI settings for adjusting this value. According to rclone documentation, the chunk_size or RCLONE_AZUREBLOB_CHUNK_SIZE environment variable can be manually configured. Is there any way to change this in TrueNAS for Cloud Sync tasks? Has anyone else run into this issue and found any workaround?

Sample error from logs:
Code:
2022/09/06 19:06:55 ERROR : [redacted filename]: Failed to copy: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42
===== RESPONSE ERROR (ServiceCode=BlockListTooLong) =====
Description=The block list may not contain more than 50,000 blocks.
RequestId:d43d187b-101e-0004-3956-c26572000000
Time:2022-09-07T01:06:55.4343185Z, Details: 
   Code: BlockListTooLong
   PUT https://[redacted].blob.core.windows.net/[redacted]?comp=blocklist&timeout=31536001
   Authorization: REDACTED
   Content-Length: [6995753]
   Content-Type: [application/xml]
   User-Agent: [rclone/v1.57.0-DEV]
   X-Ms-Blob-Cache-Control: []
   X-Ms-Blob-Content-Disposition: []
   X-Ms-Blob-Content-Encoding: []
   X-Ms-Blob-Content-Language: []
   X-Ms-Blob-Content-Md5: [61x3MSUgbJRvLxuSJLogcQ==]
   X-Ms-Blob-Content-Type: [application/octet-stream]
   X-Ms-Client-Request-Id: [a3887cc2-739e-4e0d-61d0-f6fdddab5412]
   X-Ms-Date: [Wed, 07 Sep 2022 01:06:52 GMT]
   X-Ms-Meta-Mtime: [2022-09-06T11:29:57.121939852-06:00]
   X-Ms-Version: [2020-04-08]
   --------------------------------------------------------------------------------
   RESPONSE Status: 400 The block list may not contain more than 50,000 blocks.
   Content-Length: [240]
   Content-Type: [application/xml]
   Date: [Wed, 07 Sep 2022 01:06:54 GMT]
   Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0]
   X-Ms-Client-Request-Id: [a3887cc2-739e-4e0d-61d0-f6fdddab5412]
   X-Ms-Error-Code: [BlockListTooLong]
   X-Ms-Request-Id: [d43d187b-101e-0004-3956-c26572000000]
   X-Ms-Version: [2020-04-08]
 
Top