2 way sync (FreeNAS <-> Synology) with rsync over ssh/sftp

[Sander Jansen]

Hey,
Me and a friend want to sync some folders. We both have SSH already active. I'm using a freenas box and my friend is using synology.
For ease of things, i want my FreeNAS to run the sync. (as FreeNAS is able to freely program rsync and synology cant)

As i said, we both have SSH already active and the plan was to have rsync run over ssh. as that is probably the easiest option to get our stuff in sync again.
Previously i used allway sync (windows) to do this but that's not really a option anymore.

I have tried this in the past and logging in seemed to be a problem as i remember ... something about synology not liking to allow ssh to other people then root or liking people to login with keys?... so synology uses sftp, witch in my eyes is the same as ssh. Also how can i use rsync to go both ways? not just one way. basically what it needs to go is:

1) upload nieuw files from FreeNAS to synology
2) download nieuw files from synology to FreeNAS
3) remove files that have been removed on the synology side
4) remove files that have been removed on the FreeNAS side
5) leave files alone that exists even though dates mismatch!

Point 5 is perhaps the most important! as we both have a lot of files from the previous sync, so a lot is already in both sides but with different dates as from copying.

I need to make a rsync job for about 4 folders including around 10 TB total. The plan is that it runs once a day or something... its not super important data as it mostly movies, games, programs and operating system iso's, i dont know about rsync but the programs folder includes over 1 million files (FTD memorial forum... yeah... old shit man :) ) so things need to be able to handle a lot of stuff.

So now the question... how can i go about this the easiest way? I know synology had rsync as well but when we tried to work with modules, it only worked one way.(back then)
The thing is, that on the synology side, there has to be as less manual tweaking as possible. If possible even no tweaking, just sftp on that side.

So is there anyone that has experience with setting something like this up?

Also maybe a small note: Do i have to have the user on my local side that is needed to login to synology? my local account is sander and my ssh account on synology is Sander but with upper case and a different password. how is this going to work?

EDIT:
For now im trying again with the ssh key way... i made a user that is exacly the same as my account on the synology box. I made a key using: ssh-keygen and after enabling a autentication manager service thing, i got my key. now for uploading it to synology, i get premission denied. i did:

Code:

ssh-copy-id -p (portnumber) (path to key-file) Sander@(remote address of synology) 

then it askes for my password and just says: premission dennied, try again.

On the other hand, normal login to ssh also wont let me login... and i know the password is correct as i can login with filezilla just fine.

 

[NetSoerfer]

I've been fiddling with a similar setup myself lately, trying to get off-site backups using rsync to work between my FreeNAS and a Synology DiskStation.

For the Synology side of the configuration, I took a lot of information from the following post in the Synology Community Forum: The complete idiots guide to setting up Rsync with SSH keys

Some things that come to mind when reading your questions:

Who's client, who's server?
You can execute rsync on either machine, and sync in both directions that way. The machine executing rsync is the client, the other machine is the server.
NAT: If the server is in a home network behind a router, it must be reachable through port forwarding.

DiskStation as server:

• uses rsyncd with modules (at least I haven't got it to work without rsyncd/modules)
• modules are automatically set up for all Shared Folders
• the user that rsync will log in as must have rsync permission (through user or group permissions > Applications > rsync)
• rsync must be activated in Control Panel > File Services > rsync
• SSH must be activated and reachable from the internet in Control Panel > Terminal & SNMP > Terminal (see below: rsync via SSH)

DiskStation as client:

• set up Control Panel > Task Scheduler to run rsync regularly (you can setup Task Scheduler to send you an e-mail on successful task execution)
• alternatively set up a script to

SSH login and Users

• On the client, the user that executes rsync must have access to an unecrypted SSH private key (which should be created in ~/.ssh/).
• On the server, the user that rsync will log in as must accept the client's SSH key: add the clients public key to the server's ~/.ssh/authorized_keys file
• SSH expects you to set proper permissions for ~/.ssh - the directory must be 700 (rwx --- ---), the private key must be 600 (rw- --- ---)!
• The users on client and server don't necessarily have to match, since you can specify the sever user to log in with.
• The DiskStation must be configured to Enable user home service in Control Panel > Users > Advanced to allow SSH authentication (whether it's used as client or as server)
• If you want the use the DiskStation as server, you may have to edit /etc/passwd to allow login via SSH: The user's login shell must be changed from /sbin/nologin to /bin/sh

rsync via SSH
Since rsync itself is unecrypted, you will want to encrypt your data while it is transferred over the internet. rsync itself allows you to do this by sending the data through an SSH connection.

• The server's sshd must be reachable from the internet
• Try to manually establish an SSH connection before establishing one through rsync - you will likely run into some issues that are a lot easier to troubleshoot if you can narrow them down to either rsync or SSH: ssh -P PORT -i /absolute/pathto/.ssh/privatekey USER@SERVER
• Once that works, try to run rsync but make sure you use test directories with meaningless data first: rsync -e "ssh -P PORT -i /absolute/pathto/.ssh/privatekey" /path/to/source USER@SERVER:/path/to/destination

Please note that the rsync example above does not use modules (that's actually just about as far as I've got by now). If you're going to use modules, you'll have to adapt accordingly.

You can run rsync in both directions from the same machine:

• rsync -e "ssh -P PORT -i /absolute/pathto/.ssh/privatekey" /localpath/to/source USER@SERVER:/remotepath/to/destination
• rsync -e "ssh -P PORT -i /absolute/pathto/.ssh/privatekey" USER@SERVER:/remotepath/to/source /localpath/to/destination

Read the rsync man pages to find out which rsync options will best suit your requirements

One more word of advice: rsync can quite easily delete entire directory structures if misconfigured. Have a backup of the data you're going to synchronize, and test with meaningless data or copies of your live data until you're certain it will do what you want it to: Test adding data, modifying data, deleting data, moving data around. Test file permissions after synchronization if those are important. This is the most annoying part (because you think it works and you want to get it done), but it also is the most important part.

Hope this gets you started in the right direction.

 

[NetSoerfer]

With the amount of data you're looking at, it may make sense to copy the data from the source directory to an external hard drive, carry the external hard drive to your friends house, and copy the data from the external hard drive to the target directory.

rsync works fine if the target directory contains the correct data already.

 

[Sander Jansen]

That is the case, Both directory's have most of the data already... and with upload speeds of about 30 Mbits it doesn't take to long to update it again. im going to try later today to set it up