Slow to create and delete datasets while rsync is writing to disk

[truenas-nav]

Hello,

I'm currently running an rsync copy from another NAS to TrueNAS. During this time the CPU is consistently running at 75% or higher.

TrueNAS server specs:

-Intel Xeon CPU E3-1220 V2 @ 3.1GHz
-Memory: 32GB
-Pool: 8 x 16TB in z2
-10gbps NIC (copying at 1gbps as the source NAS is only 1gbps)

Whenever I create a dataset I see the spinning circle for about 10 minutes. It eventually completes...

Same happens when I delete a dataset.

What can I do or check to improve the time to complete system admin tasks?

Thanks!

 

[artlessknave]

a 1220 v2 isn't that beefy to begin with, and rsync uses a lot of CPU cycles and disk; it has to traverse and read the ENTIRE filesystem on both sides to compare them.
you would likely have a better experience if you can just zfs replication, otherwise you just have to wait until rsync finishes.

 

[truenas-nav]

a 1220 v2 isn't that beefy to begin with, and rsync uses a lot of CPU cycles and disk; it has to traverse and read the ENTIRE filesystem on both sides to compare them.
you would likely have a better experience if you can just zfs replication, otherwise you just have to wait until rsync finishes.

Thanks for confirming and providing your advice. I don't expect to be using rsync continuously going forward, just to initially copy data over, so I guess that's fine.

Thanks!

 

[truenas-nav]

a 1220 v2 isn't that beefy to begin with, and rsync uses a lot of CPU cycles and disk; it has to traverse and read the ENTIRE filesystem on both sides to compare them.
you would likely have a better experience if you can just zfs replication, otherwise you just have to wait until rsync finishes.

One question: What hardware would you recommend purchasing these days to have a problem free NAS? LFF disks 12 bays. Supermicro?

 

[winnielinnie]

During this time the CPU is consistently running at 75% or higher.

Likely because you're using rsync over SSH, and the default cipher used does not leverage hardware acceleration.

You can either (1) use rsync via daemon/module (no encryption, in the plain), or (2) switch to an AES cipher with SSH. (The other client/NAS must also support it.)

Forum thread for reference:

RSYNC over 10gb link

Hi, I have a NAS to NAS 10gb link, currently trying to Backup these VM replicas from the one NAS to a different one using RSYNC. While iperf returns 7.77 Gbits/sec, RSYNC fluctuates only at around 47.42MB/s. Is there any resolution to this, asking as have seen several threads about RSYNC and...

www.truenas.com

 

[artlessknave]

One question: What hardware would you recommend purchasing these days to have a problem free NAS? LFF disks 12 bays. Supermicro?

used supermicro is one of the top choices. as you have only listed your CPU, its difficult to say much more. X9SCM mobo seems fairly likely, which would fit in basically any supermicro 2u 12 bay chassis.

if you are looking for non rack mount that becomes more difficult. you haven't specified though.
there are at least 2 resources dedicated to hardware advice.

 

[truenas-nav]

Likely because you're using rsync over SSH, and the default cipher used does not leverage hardware acceleration.

You can either (1) use rsync via daemon/module (no encryption, in the plain), or (2) switch to an AES cipher with SSH.

Forum thread for reference:

RSYNC over 10gb link

Hi, I have a NAS to NAS 10gb link, currently trying to Backup these VM replicas from the one NAS to a different one using RSYNC. While iperf returns 7.77 Gbits/sec, RSYNC fluctuates only at around 47.42MB/s. Is there any resolution to this, asking as have seen several threads about RSYNC and...

www.truenas.com

Thanks for your recommendation and advice.

I'm currently using rsync via: Service >>> Rsync >>> Rsync Module

My disk pool is configured with encryption, is that the reason?

How can I change the SSH ciphers? Where is the file stored to modify the default?

Thanks!

 

[truenas-nav]

used supermicro is one of the top choices. as you have only listed your CPU, its difficult to say much more. X9SCM mobo seems fairly likely, which would fit in basically any supermicro 2u 12 bay chassis.

if you are looking for non rack mount that becomes more difficult. you haven't specified though.
there are at least 2 resources dedicated to hardware advice.

Thanks, yes I left off a little info.

Business use, rack mount.

What would be a good off the shelf second hand option? Would a HP DL380 or Dell R720xd do the trick?

 

[artlessknave]

Would a HP DL380 or Dell R720xd do the trick?

yes (hpdl380 g8+), though they will be more power than your 1220v2. they will have more juice though. you wont have any support on them them, HP is notoriously hard to get bios/firmware updates for, and they tend to do things like run ALL fans at MAX speed if any fan dies.
I would probalby go with a dell, unelss you have the HP hardware already floating around.

Business use, rack mount.

you might want to consider more business oriented options, such as official ix truenas enterprise systems.

 

[winnielinnie]

I'm currently using rsync via: Service >>> Rsync >>> Rsync Module

Over 75% CPU usage with rsync over a plain, unencrypted connection? That doesn't sound right.

So from the other NAS, you're using the format: rsync -options /source/path truenas.server::module

Because if you're not using the double-colons (::), then it means you're using rsync over SSH. You need to use server::module in order to invoke rsync in module mode.

How can I change the SSH ciphers? Where is the file stored to modify the default?

Under Services -> SSH -> Auxiliary Parameters

But for now, don't modify anything, since you want to use rsync in module mode anyways.

My disk pool is configured with encryption, is that the reason?

Sure it uses CPU, but ZFS encryption uses an AES (hardware-accelerated) block cipher by default.

 

[artlessknave]

Over 75% CPU usage with rsync over a plain,

it's a 1220 v2. it's a decent CPU but this operation as described would use a good chunk of it.