Freenas 9.2.1.5 slow rsync and double remote host?

[bluecolor]

hi.
There are a few problems about rsync.

1. 30/40mb/s DeltaCopy windows rsync server can write data. normal file copy speed of 110MB / s. How do I make rsync faster?
2. rsync remote host How do I send two servers at the same time? (sample: 192.169.0.1,192.168.0.6 wrote. did not work: (
3. the rsync is scheduled to occar every 1 minutes. a shorter example: 20 seconds adjusted?

freenas 9.2.1.5
Supermicro X9DA7
64GB Ram
2 x120gb ssd
10x 3 tb sata
10000 rpm system disk

 

[cyberjock]

Well, if you don't want some harsh truths.. don't read the rest of my post.

1. Uh.. you don't. Rsync is single threaded. It was never meant to be lightning fast on transfers. It was meant to validate the source and destination files match while minimizing network traffic. There was never any emphasis on performance. Minimizing network traffic and ensure there was no corruption on the destination was it.
2. You don't. You have to do each rsync to each server independently.
3. Holy freakin crap... Your rsync schedule should never be more frequent than the time it takes to run. If you actually have more than 10TB of data rsync is going to take days to run.

You *may* want to consider zfs replication. But, doing it every minute is completely idiotic and basically impossible. Every hour, maybe. Every 30 minutes, possibly.

 

[bluecolor]

:(
ok. no problem.
1. Do you have a plug-speed sync I can do? How do I sync or fast?
2. 2 FreeNAS. Can I failover for CIFS or iSCSI?
3. the same subnet when I gave two different nic 2 ip address does not work. sample:
1.nic: 192.168.10.10/24
2.nic: 192.168.10.11/24 (can not) 192.168.10.11/32 (not working)

 

[cyberjock]

Your English isn't the greatest, but I'll try to answer your questions as best as I can.

1. ZFS Replication is very fast. It'll transfer data as fast as you can pull it off the pool, or your network, whichever is slower.
2. There is no failing over in that regard. TrueNAS does have some failover but I know almost nothing about it. You'll have to contact iXsystems to discuss that.
3. Only 1 NIC on a particular machine should ever be on a given subnet. So that would be why.

 

[Chris Moore]

I have 5 NICs in my main system (one built into the board and four on an Intel NIC) and I have four of those connected but not Link Aggregation. I have passed ports through to Jails so the jail is using a separate physical connection with a different IP address, instead of sharing the one that my CIFS share is on.
I had originally got the 4 port card thinking I would be able to use it for a LACP setup, but it didn't provide me the results I had hoped for. I wanted to speed up the replication between my two FreeNAS systems. Since it didn't work, I found another use for the extra ports. It may be over-kill, but it does work.

Sorry, I just noticed this was a very old thread. I found it because I was looking for a way to speed up Rsync.
I guess that is a lost cause?

 

[Ericloewe]

I was looking for a way to speed up Rsync.

rsync is generally CPU-bound. Calculating hashes for every file is not a trivial task.

 

[Joe Goldthwaite]

I've also been trying to speed up rsync. Right now it's taking about a day per terabyte to copy from my old FreeNAS to a new one.
I found an article about how to use parallel processes to do it. In my case I've got 8 cpu cores so running parallel tasks should help me fill up the network faster. The thread is here if you want to check it.

 

[pirateghost]

I've also been trying to speed up rsync. Right now it's taking about a day per terabyte to copy from my old FreeNAS to a new one.
I found an article about how to use parallel processes to do it. In my case I've got 8 cpu cores so running parallel tasks should help me fill up the network faster. The thread is here if you want to check it.

Why would you be using rsync to go from one FreeNAS to another? Use replication or ZFS send/receive

 

[Chris Moore]

@pirateghost The reason to NOT use ZFS send is that it copies all the features of your existing pool.
For example, if you had enabled deduplication, the only way to get rid of it is to just copy the files.
If you do a ZFS send, those settings are copied over.
The thing to look at is the SCP command. You can run multiple, threads with it and saturate your network.

 

[titan_rw]

@pirateghost The reason to NOT use ZFS send is that it copies all the features of your existing pool.
For example, if you had enabled deduplication, the only way to get rid of it is to just copy the files.
If you do a ZFS send, those settings are copied over.
The thing to look at is the SCP command. You can run multiple, threads with it and saturate your network.

As far as I know, this is incorrect.

The settings of the destination dataset can be different from the source dataset when using zfs replication.

For example, if the source dataset has dedupe and compression enabled, and you replicate that to a dataset with dedupe and compression disabled, then you lose dedupe and compression on the replicated copy.

Compression / deduplication happen at a lower level than zfs send operates on. So if you replicate from a compressed deduplicated dataset to another compressed deduplicated dataset, the data is uncompressed, and 'duplicated' (un-deduplicated?) by zfs send. Then when zfs receive writes the data to the destination dataset it is recompressed and deduplicated back to disk.

 

[Ericloewe]

As far as I know, this is incorrect.

The settings of the destination dataset can be different from the source dataset when using zfs replication.

For example, if the source dataset has dedupe and compression enabled, and you replicate that to a dataset with dedupe and compression disabled, then you lose dedupe and compression on the replicated copy.

Compression / deduplication happen at a lower level than zfs send operates on. So if you replicate from a compressed deduplicated dataset to another compressed deduplicated dataset, the data is uncompressed, and 'duplicated' (un-deduplicated?) by zfs send. Then when zfs receive writes the data to the destination dataset it is recompressed and deduplicated back to disk.

I tend to think you're right.

One of the current WIP features for openZFS are compressed send/receive streams. These would operate by avoiding the decompression/recompression stages at both ends and instead transmitting the compressed blocks directly. It makes sense that dedup would operate similarly.

 

[Chris Moore]

As far as I know, this is incorrect.

The settings of the destination dataset can be different from the source dataset when using zfs replication.
snip...

Perhaps we are talking about two different things? Or just using different terminology? There is a method of taking a snapshot of your ZFS dataset on one system and sending that to another system where you receive it as a dataset that is mounted in the same condition as the source. The pool that you import the dataset into can have any settings at all and they do not affect the settings of the dataset that is being received. It is not brought in as individual files, but as a complete dataset with all the settings of the original dataset. I have done this. It is not theoretical. The method I used did maintain compression and deduplication across the send. The pool that the dataset was brought into did not have deduplication enabled, but the dataset inside the pool did have deduplication set.

When the dataset I was sending was originally created, it was supposed to be a backup and deduplication was enabled in an effort to reduce the space consumed by the backup. It did not end up saving a measurable amount of space but it did increase processor overhead. When the backup was later "restored", the deduplication settings came with it even though we didn't want it.

The additional processor overhead of deduplication was not an issue for the quad core Xeon based servers I have, but it caused the dual core Pentium based server to have problems. The worst thing about the whole deduplication fiasco was that it didn't save a measurable amount of space on disk.

 

[Chris Moore]

Back to the original purpose of the thread. Something that goes faster than rsync.

I have one for you. If you are copying data from one system to another via the network, you can use the command scp. It must be called from the command line. I get there by ssh from a Linux system. You could use screen or tmux to make sure the process isn't dropped, what ever. Just know that if the connection is lost, the process will terminate, so you want to make sure that doesn't happen, because it can take hours for a big copy to finish. The command stands for secure copy and it connects to the other system by ssh. It will use all available resources (processor, network, etc.) to complete the copy as fast as possible. One of the systems I recently used this with had 80% CPU load while the process was running and this was the only thing it was doing. That system needs a better CPU. Still, it had finished copying 2 TB by the time I checked on it six hours later.

You can do a google search and find many examples of how to use it, but be sure to give the full 'absolute path' for both source and destination. Here is an example:

Code:

scp -r /mnt/pool/dataset/* root@192.168.10.11:/mnt/pool/dataset

This example is for running scp from the source of the data. You can use it to pull data from another computer but the formatting of the command is different.
The "-r" is to set recursive so it will do any directories it encounters. The first path is for the source directory on the system running the command. The 'root@' and IP address is the user name and address of the destination system. The : tells the program "put the data here" and then you give the destination path. The destination path and the source path do not need to be the same. The "dataset" (or what ever name you use) folder must exist on the destination server.
I have used this many times and it will go very quickly, as compared to rsync.