What's the fastest, safest, and most reliable way to transfer from Synology to TrueNAS?

[SwisherSweet]

Hi folks,

I have about 25TB of data I need to transfer from my Synology NAS to my new TrueNAS server. They are now both on 10Gbe connections locally. I have tested using Synology Hyper Backup with the RSync Single Copy option to backup a test share from my Synology Server to my TrueNAS server, but it takes a relatively long time and it does this strange thing where for multiple days it will sit at 100% complete but not actually be done. There's little to no traffic being exchanged during this time. I contacted Synology about this, and they said I just need to wait it out. After about a week at 100%, it does finish. I'm not entirely sure what's going on during this long wait, but I don't feel confident in this process.

I could use SMB and Synology does have a nice way to mount a remote SMB share and copy files using their File Station tool. This has proven fast and efficient and I don't need a 3rd machine to perform the transfer. I will probably just do this, unless I will somehow miss transferring critical file metadata/attributes, or have issues with character sets or file name lengths, folder depths, or other issues.

Does anyone have any suggestions on the fastest, safest, and most reliable way to transfer from Synology to TrueNAS?

 

[jgreco]

I'd look at using manual rsync on both ends, where you can ask for verbose output that may hint at WTF it is doing.

rsync is incredibly good at its job but is basically a swiss army knife worth of "can be difficult to use" in certain cases.

Among the exceedingly useful things that rsync can do is a dry run mode where it tells you what it *would* do, which also makes it pretty easy to tell if your job that mysteriously ended with some odd error message actually completed successfully but it was just all PO'd about user ownerships, permissions, etc., etc.

 

[SwisherSweet]

I'd look at using manual rsync on both ends, where you can ask for verbose output that may hint at WTF it is doing.

Thank you. I have used Rsync locally before, but never between two servers.

I was able to SSH into my Synology NAS and verified it does have Rsync. I found this article which shows the process simply, although my understanding is the switches / options can make things complex:

Code:

rsync -avh /home/test-rsync/ root@5.252.161.46:/home/receive-rsync/

Assuming I initiate the command from my Synology NAS and my data is located at /Volume1/Archive, and my TrueNAS is at 192.168.0.210, then would my command look like this?

Code:

rsync -avh /Volume1/Archive/ root@192.168.0.210:/mnt/tank/synology-backup/

I also assume that I would need to setup an Rsync module on the TrueNAS server pointing to /mnt/tank/synology-backup/.

Am I correct so far?

According to the article I mentioned previously, the author asserts the "a" switch will "copy files recursively", whereas according to this answer, the "a" switch "applies archive settings to mirror the source files exactly, including symbolic links and permissions". I'm not sure which to believe, but both are options I want.

Since we mostly use Macs around here and all our data is copied from Macs, would I also need the "E" switch, which "copies extended attributes and resource forks (OS X only)"?

Thanks again.

 

[winnielinnie]

I also assume that I would need to setup an Rsync module on the TrueNAS server pointing to /mnt/tank/synology-backup/.

Not necessary. Rysnc modules are only for when rsync is used as a daemon. Rsync works perfectly fine over SSH, without any extra service or configuration. (I like using the modules, for my own personal reasons, though.)

Assuming I initiate the command from my Synology NAS and my data is located at /Volume1/Archive, and my TrueNAS is at 192.168.0.210, then would my command look like this?

It's missing the"-n" flag, which was mentioned earlier. This will let you do a dry run before committing to anything.

According to the article I mentioned previously, the author asserts the "a" switch will "copy files recursively", whereas according to this answer, the "a" switch "applies archive settings to mirror the source files exactly, including symbolic links and permissions". I'm not sure which to believe, but both are options I want.

It's the same thing. "Archive mode" with "-a" is an rsync preset that incorporates several parameters into a single flag. (Out of convenience.) Otherwise, you'd have to use several flags in the same command.

Since we mostly use Macs around here and all our data is copied from Macs, would I also need the "E" switch, which "copies extended attributes and resource forks (OS X only)"?

If the files were saved to your Synology without their "resource forks", then it won't make a difference. They were already stripped. Depends how your SMB shares were configured. (Under TrueNAS this option is a checkbox named "Enable Alternate Data Streams".)

EDIT: Are you sure about the "-E" flag? I'm seeing "-E: preserve executability" when I look it up.

 

[jgreco]

Rsync works perfectly fine over SSH, without any extra service or configuration.

The difference here is that when you use rsync over ssh, it typically causes encryption and probably compression. Using the daemon mode allows you to run raw data over the network without encryption or compression, which add a substantial amount of CPU effort (and therefore time) to the process. Even with rsync in daemon mode, you can set the job up by hand without using the TrueNAS GUI.

 

[winnielinnie]

The difference here is that when you use rsync over ssh, it typically causes encryption and probably compression. Using the daemon mode allows you to run raw data over the network without encryption or compression, which add a substantial amount of CPU effort (and therefore time) to the process.

Which is why I personally use daemon mode myself. (* And why I was worried that they might remove the "Rsync Modules" from the GUI if it doesn't have enough adoption.)

(I like using the modules, for my own personal reasons, though.)

But for the sake of this "one-time-use of Synology to TrueNAS", might as well keep it simple and just use rsync over SSH.


Don't turn this thread into a mishmash!
(Reference post no longer exists. Now I just look like a jerk out of context.)

 

[SwisherSweet]

Thanks @jgreco and @winnielinnie for this very helpful information. Before reading your comments I did enable the Rsync Module and tested it out and it worked, so I assumed it was needed. I guess it wasn't after-all.

 

[winnielinnie]

The "single" colon : in your destination denotes the default (over SSH). To specify the name of a module, you'd instead use "double" colons ::. (Only the module name in the latter; not the absolute path.)

 

[jgreco]

Thanks @jgreco and @winnielinnie for this very helpful information. Before reading your comments I did enable the Rsync Module and tested it out and it worked, so I assumed it was needed. I guess it wasn't after-all.

There's always another way to accomplish any task. It's basically impossible to know them all. It's never a bad idea to ask if you're unclear.

Don't turn this thread into a mishmash!
(Reference post no longer exists. Now I just look like a jerk out of context.)

There's a "mismatch" joke in there somewhere, isn't there? But yeah. I probably shouldn't have corrected him when he incorrected me. With apologies to Steve Hofstetter.

 

[winnielinnie]

The difference here is that when you use rsync over ssh, it typically causes encryption and probably compression. Using the daemon mode allows you to run raw data over the network without encryption or compression, which add a substantial amount of CPU effort (and therefore time) to the process. Even with rsync in daemon mode, you can set the job up by hand without using the TrueNAS GUI.

Just a reminder of the "trend" I'm seeing with the development of TrueNAS; it's on the table that the "Rsync Service" (i.e, rsyncd + modules) will be deprecated in future versions of TrueNAS's web GUI.


Look at the release notes for SCALE 22.12.3:

Rsyncd Server

Determine if enabling an rsync server is necessary on the system. Rsync tasks that have Rsync Mode set to SSH or are externally configured with SSH don’t need the rsync service enabled. The rsyncd application is available when TrueNAS must be used to create insecure non-SSH rsync tasks.

This doesn't mean you can't use rsync in daemon mode (and all the advantages it offers, such as the ones you named), but to remove this from the GUI seems odd. It's a mature, complete feature, and such maintenance should be minimal. Why even consider removing it at all?

This also ties into the earlier ticket I mentioned: https://ixsystems.atlassian.net/browse/NAS-111949

We need to track rsync mod in usage stats.

We will need it to decide if we will remove that feature or not.

Having to resort to creating your "Rsync Modules" without the GUI, once again, veers away from using TrueNAS as an appliance.

 

[danb35]

it's on the table that the "Rsync Service" (i.e, rsyncd + modules) will be deprecated in future versions of TrueNAS's web GUI.

...as S3 already is, in 22.12.3.

 

[Arwen]

It would appear that the RSync daemon & module from the GUI will be replaced by an App in SCALE. In theory this gives more security, because the daemon will be isolated and the path / datasets it can act on, will be conscious choices.

If iXsystems supplies a fully functional RSync App, and supports it to some degree, (unlike Core Plugins), then I guess this could be progress.

But for me, that's annoying. I more or less quit using NFS at home for backups because RSync daemon was easier. The other files sent to my NAS were generally copied via SCP. And any management of the files, like reorganization, was generally done via SSH and local SHELL on the NAS.


Of course, this move might be due a little to their Enterprise users not needed RSync as much as us free users.

 

[winnielinnie]

It would appear that the RSync daemon & module from the GUI will be replaced by an App in SCALE. In theory this gives more security, because the daemon will be isolated and the path / datasets it can act on, will be conscious choices.

I can't wait until they gut the cluttered "Replication Tasks" in the GUI, so that they can offer it as an App for better security and modularity.

Same with "Cloud Sync Tasks".

Do the majority of users really leverage the "NFS Shares and Service" feature? It's mostly SMB, right? Let's "App-ify" NFS!

Apps for eeeeeeeeeeeeeeeeeverything!

"There's an app for that!"

Oh wait, that slogan is already trademarked...

 

[winnielinnie]

In theory this gives more security, because the daemon will be isolated and the path / datasets it can act on, will be conscious choices.

My goodness, I just realized how dangerous the NFS service is! It has access to all paths and datasets on your server. They should remove it in favor of an "NFS App", where the paths can be isolated, and only consciously selected by the user.

 

[jgreco]

so that they can offer it in an App for better security and modularity.

Not Free/TrueNAS related, but I once investigated the possibility of placing NFS servers in a FreeBSD jail. There's some stuff that prevents this from happening, alas.

 

[Arwen]

Yes, in some instances NFS has to run on the base OS. For example, Solaris 10/11 need NFS service run from the Global Zone. It won't work in a Non-Global Zone due to kernel module access. Something similar might apply to Linux containers. Unless the container has kernel module access, NFS service might be limited to Linux base OS.


That said, in the past their WERE implementations of NFS service in user land. But, because of the overhead, it ran noticeably slower than the kernel module version.


For those that don't know, Solaris 10 introduced light weight virtualization called Zones. The parent OS would be the Global Zone that manages the children, called Non-Global Zones. At one point Solaris x64 supported Linux as a Non-Global Zone, translating the Linux kernel calls into Solaris kernel calls or library calls.

 

[danb35]

Look at the release notes for SCALE 22.12.3:

You're focusing on rsync, and I'd mentioned s3 because I have that service active, but they're actually throwing out a lot of functionality:

New applications for deprecated SCALE services Dynamic DNS, TFTP, OpenVPN, WebDAV, rsync daemon, and S3 MinIO.

I have to admit, I've never understood why OpenVPN (client or server) was ever a service in TrueNAS (and their configuration for it was entirely byzantine), but why remove these? I've heard that one benefit, at least in the case of MinIO, is that the app can be kept up-to-date without needing to update TrueNAS itself, and I can see that, though time will tell how well iX keeps their apps up to date. I can see moving DDNS and OpenVPN to apps; they never belonged in the base OS anyway IMO. But TFTP? WebDAV? rsync? These all seem entirely consistent with a NAS, and they're all mature software. This really seems like a dumb move on iX' part.

 

[jgreco]

I have to admit, I've never understood why OpenVPN (client or server) was ever a service in TrueNAS (and their configuration for it was entirely byzantine),

Because it's something that'd be useful in SOHO contexts, but the problem is that it was pretty clear that the implementation was poorly designed. Speaking as someone who's done multiuser/multipurpose/multirole OpenVPN installations, it isn't particularly straightforward, because it needs to be integrated into the network and routing, something that TrueNAS has historically been terrible at. If you use TUN style connections, which are desirable if you want to be able to integrate client devices that do not support ethernet (such as Apple iDevices), you need to figure out how to fix the user's routing environment, which may not even be possible. TAP is much more flexible, not requiring routing changes, but then you lose the Apple iDevices etc.

But TFTP? WebDAV? rsync? These all seem entirely consistent with a NAS, and they're all mature software. This really seems like a dumb move on iX' part.

They're killing those? Weird.

 

[danb35]

They're killing those? Weird.

The SCALE release notes say they're all deprecated as built-in services and to be replaced with apps. Presumably the built-in services will be removed in a future release (and knowing iX, it will be a hotfix to a point release). What next? FTP? iSCSI? nut?

And I get why people would want an OpenVPN server, just as I get why people would want DDNS support--but in both cases, as a built-in service to the NAS doesn't make sense to me (even assuming they were implemented sensibly, which OpenVPN--client and server--definitely isn't on TrueNAS). Both really belong on the edge device, but failing that, implementing them as an app or in a VM would make sense. But not TFTP and rsync.

 

[winnielinnie]

But not TFTP and rsync.

Agreed. Mainly for the reasons you stated: a NAS appliance's primary role is to provide storage of data and fundamental services to facilitate transfer and storage.

And there are very good reasons to treat Rsync Daemon (Modules) as a first-class feature.

In a local network you gain the benefit of less CPU overhead, convenient short-hand "module" names, and faster transfers for high bandwidth setups; all without needing to worry about security concerns or SSH keys. (It's your local network. If "Rsync Modules" are a security concern, then by the same reasoning NFS should be treated as a "security concern".)