Gorf
Dabbler
- Joined
- Jan 5, 2017
- Messages
- 20
I've been trying to use Cloud Sync to move replicated data up to S3. The process works fine more or less. So setting it up and running it the first time seems fine. However it's when the process repeats on schedule that I have problems. The job basically re-uploads 100% of all the data. I'm not sure why. Maybe i'm misunderstanding the Cloud Sync Tasks process.
I tested syncing using the s3cmd tool from the command line and I discovered that it too was having the same issue. I can make it stop by telling it to ignore MD5 and only do file-size comparisons. Which, for what I am doing, is fine.
Is that maybe what is going on under the Cloud Sync? I really want an incremental copy of this data. It's over 10TB and I don't want to move that much data every week as the process from this location is slow.
Alternatively, it would be straightforward enough to just create a small VM of Linux and run the aws sync tools. But then I have to figure out how to best provide it access to the whole of the underlying pool/datasets. hmm NFS?
I tested syncing using the s3cmd tool from the command line and I discovered that it too was having the same issue. I can make it stop by telling it to ignore MD5 and only do file-size comparisons. Which, for what I am doing, is fine.
Is that maybe what is going on under the Cloud Sync? I really want an incremental copy of this data. It's over 10TB and I don't want to move that much data every week as the process from this location is slow.
Alternatively, it would be straightforward enough to just create a small VM of Linux and run the aws sync tools. But then I have to figure out how to best provide it access to the whole of the underlying pool/datasets. hmm NFS?
