Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

rclone vs. symlinks, or it's not a backup until you test it...

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

Bruce Howells

Neophyte
Joined
Jan 13, 2017
Messages
5
So, I have a repository of rear (relax and recover) images that are symlink-heavy, and Cloud Sync (ie, rclone)'ed it out to a couple remotes. All seemed fine, until I needed to pull the files back.

It looks like the default rclone settings generated for a Cloud Sync task include neither -l (which would sync a copy of the referred file, not preserve the symlink) or -L (which preserves the symlink by saving it on upload and recreating it on download.) As a result, symlinks are silently broken.

This is likely an unpleasant surprise to those expecting the lightly-documented Cloud Sync feature to, well, sync.

It probably makes sense to surface this in the UI somewhere, but in the meantime, forcing -L gives me the behavior I need... and might actually be the right thing to file a bug report suggesting.

Any guidance on how to apply a local change so my generated rclone.conf includes -L?
And should I suggest that -L (preserve symlinks, recreate them on restore) be the default behavior (vs. losing symlinks on copy)?
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
7,516
So, I have a repository of rear (relax and recover) images that are symlink-heavy, and Cloud Sync (ie, rclone)'ed it out to a couple remotes. All seemed fine, until I needed to pull the files back.

It looks like the default rclone settings generated for a Cloud Sync task include neither -l (which would sync a copy of the referred file, not preserve the symlink) or -L (which preserves the symlink by saving it on upload and recreating it on download.) As a result, symlinks are silently broken.

This is likely an unpleasant surprise to those expecting the lightly-documented Cloud Sync feature to, well, sync.

It probably makes sense to surface this in the UI somewhere, but in the meantime, forcing -L gives me the behavior I need... and might actually be the right thing to file a bug report suggesting.

Any guidance on how to apply a local change so my generated rclone.conf includes -L?
And should I suggest that -L (preserve symlinks, recreate them on restore) be the default behavior (vs. losing symlinks on copy)?
This is an 11.3 feature.
1580038359802.png


Do note that "Follow Symlinks" generally a bad idea. Suppose you have a Linux client with WINE installed on it that rsyncs its home directory to your NAS. WINE may set up a symlink for DOS device "z:" to "/". Your rclone dataset now contains a symlink loop, and rclone will continually upload data until the process is stopped.
 
Last edited:

Bruce Howells

Neophyte
Joined
Jan 13, 2017
Messages
5
Thanks for the edit.

I took a look at the 11.3 documentation and the answer didn't jump out at me - since rclone is "interesting" in how it handles symlinks, does the Follow Symlinks option turn on -l or -L?
 

plsuh

Neophyte
Joined
Apr 26, 2019
Messages
11
I've been chasing this down among other things so that I can do a backup to B2, and on FreeNAS the checkbox turns on -L (--copy-links). For backups, -l (--links) would be preferable.
 

anodos

Sambassador
iXsystems
Joined
Mar 6, 2014
Messages
7,516
Some care need to be taken with symlink following. For instance, a file-based backup of a Linux home directory may contain symlinks to `/` (seen this on many occasions. This will create a symlink loop for rclone (extremely bad situation IIRC, since it may continually upload files under new paths until killed). I believe I added a small CLI utility in 11.3 to scan for symlinks pointing back to our boot device scanlnk -a quick -p /path/to/share. It's pretty fast.
 

Neutron

Newbie
Joined
Jul 7, 2021
Messages
1
Did you ever find a solution on how to add the -L Option?
The "Follow Symlink" Option exists in "TrueNAS-12.0-U3.1", but Symlinks get followed if it's selected or not (at least on my SFTP Server).
 
Top