sanstey
Cadet
- Joined
- Oct 5, 2017
- Messages
- 2
I'm running into a problem syncing files larger than ~200GB (50,000 blocks) to Azure blob storage. I've determined this is due to the default rclone block size being 4MB. I don't see any GUI settings for adjusting this value. According to rclone documentation, the chunk_size or RCLONE_AZUREBLOB_CHUNK_SIZE environment variable can be manually configured. Is there any way to change this in TrueNAS for Cloud Sync tasks? Has anyone else run into this issue and found any workaround?
Sample error from logs:
Sample error from logs:
Code:
2022/09/06 19:06:55 ERROR : [redacted filename]: Failed to copy: -> github.com/Azure/azure-storage-blob-go/azblob.newStorageError, github.com/Azure/azure-storage-blob-go/azblob/zc_storage_error.go:42 ===== RESPONSE ERROR (ServiceCode=BlockListTooLong) ===== Description=The block list may not contain more than 50,000 blocks. RequestId:d43d187b-101e-0004-3956-c26572000000 Time:2022-09-07T01:06:55.4343185Z, Details: Code: BlockListTooLong PUT https://[redacted].blob.core.windows.net/[redacted]?comp=blocklist&timeout=31536001 Authorization: REDACTED Content-Length: [6995753] Content-Type: [application/xml] User-Agent: [rclone/v1.57.0-DEV] X-Ms-Blob-Cache-Control: [] X-Ms-Blob-Content-Disposition: [] X-Ms-Blob-Content-Encoding: [] X-Ms-Blob-Content-Language: [] X-Ms-Blob-Content-Md5: [61x3MSUgbJRvLxuSJLogcQ==] X-Ms-Blob-Content-Type: [application/octet-stream] X-Ms-Client-Request-Id: [a3887cc2-739e-4e0d-61d0-f6fdddab5412] X-Ms-Date: [Wed, 07 Sep 2022 01:06:52 GMT] X-Ms-Meta-Mtime: [2022-09-06T11:29:57.121939852-06:00] X-Ms-Version: [2020-04-08] -------------------------------------------------------------------------------- RESPONSE Status: 400 The block list may not contain more than 50,000 blocks. Content-Length: [240] Content-Type: [application/xml] Date: [Wed, 07 Sep 2022 01:06:54 GMT] Server: [Windows-Azure-Blob/1.0 Microsoft-HTTPAPI/2.0] X-Ms-Client-Request-Id: [a3887cc2-739e-4e0d-61d0-f6fdddab5412] X-Ms-Error-Code: [BlockListTooLong] X-Ms-Request-Id: [d43d187b-101e-0004-3956-c26572000000] X-Ms-Version: [2020-04-08]