10GbE Read/Write Performance and Replication Performance Advice

[Chris Tobey]

Hi Everyone,

I am currently getting ~500 MB/s reads and writes over 10GbE with my system and 175 MB/s replication from this server to a similar (but older) server over 10GbE.

Server Configuration:
Chassis: Supermicro CSE-846BE16-R920B
Motherboard: Supermicro X9DRD-7LN4F
CPUs: 2 x Xeon E5-2603v2
RAM: 12 x 16GB ECC DDR3-1866
SLOG: Intel Optane P4800X.
Interface: Currently 1 of 2 10GbE Chelsio NIC ports, but expandable to 4x10GbE should I be able to take advantage of it.
ZFS Pool:

root@freenas:~ # zpool status
pool: SG1
state: ONLINE
scan: scrub repaired 0 in 4 days 21:39:45 with 0 errors on Thu Sep 20 21:39:59 2018
config:

NAME STATE READ WRITE CKSUM
pool1 ONLINE 0 0 0
raidz2-0 ONLINE 0 0 0
gptid/55ef2498-ac64-11e5-b6ed-0025905800c4 ONLINE 0 0 0
gptid/56ce660c-ac64-11e5-b6ed-0025905800c4 ONLINE 0 0 0
gptid/57c2220a-ac64-11e5-b6ed-0025905800c4 ONLINE 0 0 0
gptid/58a788e2-ac64-11e5-b6ed-0025905800c4 ONLINE 0 0 0
gptid/59a42bc4-ac64-11e5-b6ed-0025905800c4 ONLINE 0 0 0
gptid/5a9d3eee-ac64-11e5-b6ed-0025905800c4 ONLINE 0 0 0
raidz2-2 ONLINE 0 0 0
gptid/f47a731e-ffea-11e5-ab2b-0cc47a1e4734 ONLINE 0 0 0
gptid/f58f6ee3-ffea-11e5-ab2b-0cc47a1e4734 ONLINE 0 0 0
gptid/f699307f-ffea-11e5-ab2b-0cc47a1e4734 ONLINE 0 0 0
gptid/f79e727b-ffea-11e5-ab2b-0cc47a1e4734 ONLINE 0 0 0
gptid/f8aab7e9-ffea-11e5-ab2b-0cc47a1e4734 ONLINE 0 0 0
gptid/f9c2c4df-ffea-11e5-ab2b-0cc47a1e4734 ONLINE 0 0 0
logs
gptid/0d56f7b3-c8f7-11e8-a885-0cc47a1e4734 ONLINE 0 0 0

errors: No known data errors


Replication is done with Replication Stream Compression = Off and Encryption Cipher = Disabled

My question is: what tests should I run to find the max performance my system is capable of, and what changes/tuning should I do to reach that performance.

 

[garm]

With only two vdevs 500 MBps sounds about right

 

[Chris Tobey]

Would I be better off adding more vdevs now even though I don't need the storage?

 

[Allan Jude]

Try benchmarking just the pure read performance:
zfs send -veL pool/dataset@snapshot | dd of=/dev/null bs=1m

And see how fast it is just reading, and throwing it away. Then, you can look for bottlenecks elsewhere if that number is much higher than what you are seeing over the network

 

[Chris Tobey]

I did a test on the 10GbE network between the two FreeNAS 11.1 servers:

Code:

root@freenas:~ # iperf -c freenas2 -P 1 -i 1 -t 50
------------------------------------------------------------
Client connecting to freenas2, TCP port 5001
TCP window size: 32.8 KByte (default)
------------------------------------------------------------
[  3] local 10.16.240.35 port 64330 connected with 10.16.204.16 port 5001
[ ID] Interval	   Transfer	 Bandwidth
[  3]  0.0- 1.0 sec  1.09 GBytes  9.38 Gbits/sec
[  3]  1.0- 2.0 sec  1.09 GBytes  9.37 Gbits/sec
[  3]  2.0- 3.0 sec  1.09 GBytes  9.38 Gbits/sec
[  3]  3.0- 4.0 sec  1.09 GBytes  9.36 Gbits/sec
[  3]  4.0- 5.0 sec  1.09 GBytes  9.37 Gbits/sec
[  3]  5.0- 6.0 sec  1.09 GBytes  9.36 Gbits/sec
[  3]  6.0- 7.0 sec  1.09 GBytes  9.37 Gbits/sec

I appear to have no issues with my network being able to transfer at near 10GbE speeds. I did not think there would be as it's a pure fibre network with enterprise managed switches.

 

[Chris Tobey]

Results of pure read performance test:
1st Read (should not be in ARC):

Code:

root@freenas:~ # zfs send -veL pool/test@auto-20181008.2000-4w | dd of=/dev/null bs=1m
full send of pool/test@auto-20181008.2000-4w estimated size is 45.7G
total estimated size is 45.7G
...
0+1121420 records in
46835+1 records out
49110548232 bytes transferred in 100.836895 secs (487029557 bytes/sec)

2nd Read (should be in ARC)

Code:

root@freenas:~ # zfs send -veL pool/test@auto-20181008.2000-4w | dd of=/dev/null bs=1m
full send of pool/test@auto-20181008.2000-4w estimated size is 45.7G
total estimated size is 45.7G
...
0+1121423 records in
46835+1 records out
49110548232 bytes transferred in 29.353090 secs (1673096364 bytes/sec)

My interpretation of these results is that my disk read speed is ~487 MB/s and my RAM read speed is ~1,673 MB/s.

 

[Allan Jude]

Those values make sense to me.

If you are using SSH for the transport for replication, that might explain why you are seeing 175MB/s instead of 400+MB/s

While doing the not-in-ARC version of that test, run 'gstat -p' in another ssh session. Watch how busy the disks are. It really depends how fragmented your data is, and what type of disks you have, what you might expect the speed to be like.


My system, which is similar to yours (2x RAID-Z2s of 6 disks each):

zfs send -veL media/scratch@sendtest | dd of=/dev/null bs=1m
full send of media/scratch@sendtest estimated size is 67.5G
total estimated size is 67.5G
...
load: 1.02 cmd: zfs 33117 [running] 32.99r 0.00u 20.14s 70% 4380k
0+418252 records in
15400+0 records out
16148070400 bytes transferred in 32.987567 secs (489519895 bytes/sec)
...
0+1723323 records in
69144+1 records out
72502943688 bytes transferred in 116.314186 secs (623337070 bytes/sec)

It got faster as the newer data is spread onto the 2nd vdev.


Trying with fresh data:

zfs create -o compress=off -o primarycache=metadata media/synthetic
#dd if=/dev/zero of=/media/synthetic/zerofile bs=1m count=10k conv=sync
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 9.118756 secs (1177509116 bytes/sec)

#dd if=zerofile of=/dev/null bs=1m
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 16.670399 secs (644100864 bytes/sec)


#zfs set primarycache=all media/synthetic
Prime ARC
#dd if=zerofile of=/dev/null bs=1m
10240+0 records in
10240+0 records out
10737418240 bytes transferred in 3.825981 secs (2806448703 bytes/sec)