Pin-pointing perf bottleneck

[anRossi]

I re-purposed a spare computer to be a low-traffic backup FreeNAS box.
I've read the hardware suggestions and I understand that I have a non-recommended setup and I don't expect great performance as a result.
I'm just trying to pinpoint where the bottleneck is, out of intellectual curiosity.

I have:
CPU: AMD Athlon(tm) 64 X2 Dual Core Processor 3800+ (2000.05-MHz K8-class CPU)
real memory = 8589934592 (8192 MB)
avail memory = 8228229120 (7847 MB)
atapci1: <nVidia nForce MCP55 SATA300 controller> (no AHCI, no SATA3)
nfe0: <NVIDIA nForce MCP55 Networking Adapter>
e1000phy0: <Marvell 88E1116 Gigabit PHY> (Does the PHY affect perf?)
4x WDC WD10EFRX-68PJCN0 drives in a RAIDZ2.

Results of dd tests:
write 16GB:
17179869184 bytes transferred in 91.180303 secs (188416452 bytes/sec)
read 16GB:
17179869184 bytes transferred in 55.818385 secs (307781553 bytes/sec)

So that's not the problem. That's very nice perf.

Results of SMB tests:
read ~8GB:
70 MB/s
write ~8GB:
~40MB/s (towards the end of the copy, it jumped about +10MB/s)

That's pretty good. I would expect this kind of perf.

My backup rsync job is run over ssh and goes about 25MB/s. Since this is what the box will be doing most of the time, this is the one I care about.
This is kind of on the slow side. (a 200GB backup takes a long time)

The dd test uses close to 100% CPU, so I would consider that the upper bound on perf.
The SMB test don't use much CPU (probably about 25%).
I think the rsync task itself doesn't use much CPU (~10% CPU) but the ssh daemon uses at least 50% CPU.
I assume encryption to be the bottleneck (and AES-NI looks really attractive right now). When I used this computer as my main system, I recall a truecrypt benchmark showed AES at about 50 MB/s, so this is surprising to me.
Does RAIDZ2 add enough processor overhead that it slows encryption significantly?

Are my suspicions correct that the encryption performance of this processor is the bottleneck of rsync-over-ssh?

 

[eraser]

Can you configure your SSH session to use a different encryption cipher (try 'none' or 'blowfish')? Also confirm that SSH Compression is not enabled.

 

[cyberjock]

Your dd numbers are about what I'd expect for your setup.

Your CIFS write numbers are a little low, but nothing I'd be too terribly worried about considering you have 8GB of RAM and your CPU isn't fast.

Your rsync numbers are about what I'd expect too. rsync is slow. It's very CPU intensive, uses 2 threads but is basically single-threaded for the CPU intensive stuff. Even on my Xeon system I was getting only about 35MB/sec. It's a well known fact that rsync is slow. So either accept the speed you have or consider something like a ZFS snapshot/replication. Those go lightning fast!

 

[anRossi]

Eraser, with compression off and even using a crap cipher like arcfour128, the best I get (using sftp) is 30MB/s. I need to face facts that my beloved dualcore AMD is *old*. :(

cyberjock, I had no idea rsync was so slow. I figured it was a pretty fast protocol, since so many places still use it and its delta algorithms, blah blah.

Your suggestion regarding snapshots is the topic of another thread I've been meaning to start. Stay tuned. ;)

 

[cyberjock]

Rsync the protocol isn't slow. But the way i does its delta algorithms and validates that the data written on the destination machine is in sync i a heavy burden for processors that don't have very high single core performance. Even iXsystems does rsync from 2 local very fast pools and the often top out below 100MB/sec. rsync is meant to minimize data that is sent across the network and minimize the likelyhood of corruption during transfer. It never promised high performance. ;)