Where is my bottleneck? Unexpectedly slow performance

Hoeser

Dabbler
Joined
Sep 23, 2016
Messages
23
Hi Guys,

Long time user, first time poster. I ran FreeNAS for years as a fiber channel SAN for a couple of ESXi hosts but I have recently just built a new FreeNAS box for that purpose, plus a little more. I've switched to 10GB NIC's instead of the FC setup, as the FC setup wasn't really supported and was annoying to maintain.

Anyway, I'm getting what I feel is less than expected performance from this box and I'm really not sure why it's so slow. I must be missing something simple (I hope).

Hardware:

2x Intel E5-2670 v2 (2.5GHz x 10cores)
Supermicro X9DRi-LN4+
64GB ECC PC8500
3x LSI 9211-8i in IT mode
Intel X540-T2
FreeNAS 11.2U1 (latest at the time of posting)
Supermicro 24 drive chassis

Currently running 3 zpools -

Chops - Two vdevs of raidz2 6x12TB Western Digital HC520 (12 drives total)
Vern - Single vdev raidz1 4x4TB HGST 7200rpm (4 drives total)
SSD - Single vdev raidz1 5x500GB Samsung Evo 860 SATA (5 drives total)

For the sake testing, I've set all pools to sync=disabled. This is all happening directly on the FreeNAS box, though the performance is roughly the same from a VM mounting an NFS share from the box over the 10GB storage network. The file is a standard h264 mkv on a ramdisk.

Results:

root@neutron[/tmpfs]# rsync -av --info=progress2 testfile.mkv /mnt/Chops/test
sending incremental file list
testfile.mkv
2,844,230,150 100% 196.17MB/s 0:00:13 (xfr#1, to-chk=0/1)

sent 2,844,924,632 bytes received 35 bytes 210,735,160.52 bytes/sec
total size is 2,844,230,150 speedup is 1.00
root@neutron[/tmpfs]# rsync -av --info=progress2 testfile.mkv /mnt/Vern/test
sending incremental file list
testfile.mkv
2,844,230,150 100% 200.06MB/s 0:00:13 (xfr#1, to-chk=0/1)

sent 2,844,924,632 bytes received 35 bytes 210,735,160.52 bytes/sec
total size is 2,844,230,150 speedup is 1.00
root@neutron[/tmpfs]# rsync -av --info=progress2 testfile.mkv /mnt/SSD/test
sending incremental file list
testfile.mkv
2,844,230,150 100% 205.85MB/s 0:00:13 (xfr#1, to-chk=0/1)

sent 2,844,924,632 bytes received 35 bytes 210,735,160.52 bytes/sec
total size is 2,844,230,150 speedup is 1.00
root@neutron[/tmpfs]#


It seems I can only write out about 200MB/sec with rsync to any of the 3 pools. I really expected far better performance for sequential writes, especially on the Chops and SSD pools.

I hope someone can give me some insight on what I'm doing wrong.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Since the performance is the same for all pools that means something in the rsync is the bottleneck. Why are you testing local performance with rsync? You can start doing local perf tests by disabling compression on your datasets then use dd if=/dev/zero of=/path/to/file bs=1M count=10000 for write tests and dd of=/dev/null if=<test file> bs=1M. That is how to start testing. For best performance you will want to use mirrors vdevs. Also last time i checked those intel 10gig cards max out around 5Gbps, things might have changed though.
 
Joined
May 10, 2017
Messages
838
Like mentioned rsync isn't a good option for performance testing, for a single file copy you can use for example pv:

pv testfile.mkv > /mnt/SSD/test/testfile.mkv
 

Hoeser

Dabbler
Joined
Sep 23, 2016
Messages
23
I'm using rsync locally because currently my primary use case right now is rsync'ing over the network - I just did it locally to eliminate the network as a potential bottleneck. Performance was roughly the same.
 

Hoeser

Dabbler
Joined
Sep 23, 2016
Messages
23
pv is showing about 1.0GiB / sec for all 3 devices.

1012MiB/sec for chops, 1.17GiB/sec for Vern, 1.14GiB/s for SSD.

I'm not sure why rsync is sucking so badly ? I was expecting something around 400MB/sec.
 
Joined
May 10, 2017
Messages
838
I'm using rsync locally because currently my primary use case right now is rsync'ing over the network

I'm afraid you'll find rsync is even slower over ssh, I can't get it above 150MB/s for a single thread, see here for some more detail:
https://forums.freenas.org/index.ph...e-iperf-good-data-copy-slow.72239/post-502575


pv is showing about 1.0GiB / sec for all 3 devices.
Likely optimistic since flie is small and probably fits mostly in the RAM cache.
 

Hoeser

Dabbler
Joined
Sep 23, 2016
Messages
23
I'm afraid you'll find rsync is even slower over ssh, I can't get it above 150MB/s for a single thread

rsync over async NFS is what I'm currently doing. What you claim seems to be reality because I'm doing about 156MB/sec.

Likely optimistic since flie is small and probably fits mostly in the RAM cache.

That's what I figured.

I guess this is all just rsync being terrible.

Time results:

cp testfile.mkv /mnt/Chops/test 0.00s user 2.45s system 99% cpu 2.464 total
rsync -Wav --info=progress2 * /mnt/Chops/test 13.44s user 6.86s system 155% cpu 13.018 total
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Here are your options for ssh ciphers, you can try different encryption types to speed up your rsync over ssh. Usually it's the encryption that slows things down on top of the cpu usage from rsync. But networking usually isn't the bottleneck for rsync+ssh. If you can you should use zfs send/recv, it will be significantly faster. You don't have to use zfs on the receive side but you probably should.
Code:
# ssh -Q cipher
3des-cbc
blowfish-cbc
cast128-cbc
arcfour
arcfour128
arcfour256
aes128-cbc
aes192-cbc
aes256-cbc
rijndael-cbc@lysator.liu.se
aes128-ctr
aes192-ctr
aes256-ctr
aes128-gcm@openssh.com
aes256-gcm@openssh.com
chacha20-poly1305@openssh.com


You should also know that putting a ssd pool on the same system as a hdd pool can be a problem. There are certain pool tuning settings you can use for ssd pools that will improve your performance. The problem is that pool tuning stuff is system level so it applies to all pools on the system. This will not be good for your hdd pools.
 

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
I'm not sure why rsync is sucking so badly ? I was expecting something around 400MB/sec.
rsync is designed to be slow so it doesn't prevent the server from simultaneously serving other data. It is intended to share, not maximize.
 

droeders

Contributor
Joined
Mar 21, 2016
Messages
179
rsync is designed to be slow so it doesn't prevent the server from simultaneously serving other data. It is intended to share, not maximize.

Please provide a reference to back this up, as that sounds like nonsense to me. In my experience, rsync will use all available bandwidth unless you specify a limit with the '--bwlimit' option.

There may be something in rsync that's the bottleneck, but not because it's "designed to be slow".
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Please provide a reference to back this up, as that sounds like nonsense to me. In my experience, rsync will use all available bandwidth unless you specify a limit with the '--bwlimit' option.

There may be something in rsync that's the bottleneck, but not because it's "designed to be slow".
Yeah it's definitely not designed to be slow.
 

Hoeser

Dabbler
Joined
Sep 23, 2016
Messages
23
This is still perplexing to me.

My environment has changed a bit as I continue to build this out, I'm inclined to make a new post, but I hate double posting. Right now I'm struggling with what seem like awful read speeds on both my (nearly identical) FreeNAS systems.

Here is the current scenario:

FreeNAS Host A:
2x Intel E5-2670 v2 (2.5GHz x 10cores)
Supermicro X9DRi-LN4+
128GB ECC PC8500
3x LSI 9211-8i in IT mode
Intel X540-T2
FreeNAS 11.2U1
Supermicro 24 drive chassis
3 pools:
HDD_Main: 10x3TB raidz2 + 10x3TB raidz2 stripe w/ Intel Optane 900P Slog [sync always]
HDD_Alt: 4x4TB raidz1 (not in scope for this thread)
SSD: 5x500GB raidz1 (not in scope for this thread)

FreeNAS Host B:
2x Intel E5-2670 v2 (2.5GHz x 10cores)
Supermicro X9DRi-LN4+
64GB ECC 1333MHz
3x LSI 9211-8i in IT mode
Intel X540-T2
FreeNAS 11.2U1
Supermicro 24 drive chassis
1 pool:
HDD_Main: 6x12TB raidz2 + 6x12TB raidz2 stripe w/ Intel Optane 900P Slog [sync always]

The latest issue

(On Host B)I can dd read the drives themselves at roughly 260MB/sec with bs=1M for 10GB. No issues there, seems about right. gstat will show the device at close too 100% disk busy. Reading out a large file (media server) with dd, using 1M record size - not more than 200MB/sec. This seems terrible for a 6+6 raidz2 stripe ? gstat reflects about 20% usage on average across all the member disks.

This poor performance seems to reflect in rsync copying data from Host B to Host A, via NFS (sync always) to HDD_Main, ~60MB/sec. FreeNAS to FreeNAS 10GB networked. Testing rsync using ssh returns the same result.

Creating a 20GB zero file with dd bs=1M on HostA:
20971520000 bytes transferred in 28.206470 secs (743500352 bytes/sec) (seems about right)
Reading a large media file with dd bs=1M on HostA:
9500588718 bytes transferred in 27.822061 secs (341476814 bytes/sec) (seems poor)

Creating a 20GB zero file with dd bs=1M on HostB:
20971520000 bytes transferred in 31.582340 secs (664026789 bytes/sec) (seems about right)
Reading a large media file with dd bs=1M on HostB:
9500588718 bytes transferred in 46.100726 secs (206083276 bytes/sec) (seems awful)

Writes appear to be fine / as expected for the disk layouts on both systems, however these read performances seem terrible, on both hosts. They're identical build for the most part, so likely a configuration problem shared between the two?

I've been combing threads on this for days and I've yet to turn up anything that has made a lick of difference.
 
Top