Transfer Rates in replication

[nriedman]

I have 2 freenas units I am using to replicate 1 (primary) to 2 (offsite backup) and am only getting <1G throughput on a 10G link. In all I need to replicate multiple pools totalling ~150TB for backup replication so transfer performance is vital in order to keep up with the daily deltas. So first...some specs:

Unit 1: (45 drives storinator)
Freenas version - 11.3-RC1
CPU - E52620 v4 dual
Memory - 256GB
HBAs - 4 x LSI 9305 12Gb/s
Pool consists of 6 8TB Seagate ironwolf 7.2K in a RAIDz2
10GB intel nic

Unit 2:
Freenas version - 9.2.10
CPU - Intel® Xeon® Processor E5-2620 v4
Memory - 262GB
HBAs - Rocket HBA 750s
Pool consists of 10 4TB Hitachi 5.4K in a RAIDz2
10GB intel nic

Results:
iperf from 1 to 2 ~3.5GB using default iperf settings

zfs send pool/snap | ssh 'ip' zfs recv -F Pool/dataset yields 110MiB/s

During the transfer the cpu shows ~2% on unit 1 and ~3% on unit 2. I have been digging into diskinfo, gstat, iostat, and numerous zpool commands to view performance. I did find a drive taking 100% busy time seen in gstat while the rest of the pool was nearly idle. I then removed that drive and replaced it in unit 2 and throughput went from 50MB to 100MB, but am unable to get past that. I have tested with other pools on both sides as I have plenty of drives to work with, but all yield the same result.

Network throughput:

I have been searching the forums over the past week but haven't yet understood where to continue digging. I have verified that it is using the 10G nics instead of the 1G management ones by looking at the reporting pages. Any thoughts.....

 

[Chris Moore]

Because you say it is over a 10Gb network, I am guessing it is local. If the network is trusted, you can do away with SSH for the send and receive to eliminate that overhead.

ZFS send and receive with netcat instead of ssh (for trusted networks)
https://www.ixsystems.com/community/threads/replication-stream-compression.73563/post-509985

Other than that, if you only have one vdev (as you described) the bandwidth of the system is limited by the quantity of disks. One vdev yields roughly the performance of a single constituent disk in the pool.
More vdevs in the pool increases bandwidth. Segregating the drives into multiple pools in the same chassis hobbles your potential performance.
Ideally, all the drives in a system would be in a single pool, with many vdevs, and the storage is then carved up logically.

You might want to reviw some of these materials:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Overview of ZFS Pools in FreeNAS from the iXsystems blog:
https://www.ixsystems.com/blog/zfs-pools-in-freenas/

Terminology and Abbreviations Primer
https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

 

[Chris Moore]

have been searching the forums over the past week but haven't yet understood where to continue digging

Ask. We enjoy helping.

 

[nriedman]

Thanks Chris, I just didn't want to duplicate other threads if I could avoid it.

Correct it is a local LAN so ssh is not required. I could have sworn nc was producing the same results while testing this week albeit I also I had a failing drive. So with that replaced I am now seeing > 3G throughput with just the netcat commands. I have more testing to figure out why iperf is only yielding 3G instead of the 10G which I can achieve with other devices.

I do understand adding more vdevs to the pool would increase performance as more spindles are in the stripe mix for read and write, however, I was concerned with having so many vdevs that if 3 drives went out in the same vdev (assuming raidz2) that the entire pool would be destroyed. With nearly 60 drives in the unit we have had many failures over the years. I assume the write/read performance is not the limiting factor in this equation but I haven't done much testing to be sure of it. A quick dd test on a 6 drive single vdev raidz2 yielded:

dd if=/dev/zero of=testfile bs=10M count=50000
50000+0 records in
50000+0 records out
524288000000 bytes transferred in 204.733069 secs (2560836907 bytes/sec) 2.5GB/s

dd if=testfile of=/dev/zero bs=10M count=50000
50000+0 records in
50000+0 records out
524288000000 bytes transferred in 96.294739 secs (5444617301 bytes/sec) 5.4GB/s

The replication task as configured via the gui is using ssh as seen in the htop list and I am unaware of how to force it to use netcat only. Am I missing something there or does it just need to be configured via command line.

 

[Chris Moore]

replication task as configured via the gui is using ssh as seen in the htop

I think there are detailed instructions at the link:

ZFS send and receive with netcat instead of ssh (for trusted networks)
https://www.ixsystems.com/community/threads/replication-stream-compression.73563/post-509985

but I have not looked in a while. To use zfs send and receive using nc instead of ssh, you probably need to use a command line to do it. I have my replication tasks configured as regular scheduled tasks so I have a bit more manual control over it.

Where I was working up to the end of September, we had several of the 60 drive units from various vendors. The drive failure rate wasn't too bad after the first few went, the ones that were probably defective from manufacturing flaws. What kind of drives are you using and how old are they? I was only getting around a 5% failure rate in the first six months and even lower after that. The department I was working for only had around 280 drives spinning, but some of those were around five years old and I was working on getting a new system so I could move the data off the old drives.

 

[nriedman]

Thanks for the tip on the scheduled tasks. That will provide the level of granularity I am looking for.

For the drives we purchased 4TB Hitachi 5.4K around 4 years ago. We generally buy in bulk from 20 to 30 drives at a time and as such when one fails after so many years we have experienced several others failing around the same time frame (within a few months). This bit us years ago when just running larger pools and using raidz1 we lost the pool. Luckily we utilize multiple layers of backups, but it took a while to get the dust settled. Our latest drive purchase is 8TB Seagate ironwolf 7.2K as recommended from the staff at 45 drives.

The new failure rate I would say is less than 3%, but since we have various ages of drives in the same unit (purchasing once a year or every two years) drive failure because of lifespan is a concern. We have replaced approximately a dozen drives in the last few months. In total we have around 150 capacity drives under zfs from multiple units. Another 75-100 in other vendor chassis's such as emc and hp, but that is altogether different.

Now I can see when running 'zfs send volume@auto-20191223.1645-1w | pv | nc -w 20 ip of destination' I am averaging ~500MiB/s transfer. Then running gstat on the source I can see the drives are nearing >90% busy which indicates to me that you were correct in bringing up the single vdev per pool. I will test with another unit and post the results of adding more vdevs (thus more spindles) and how that affects the transfer performance.

Also for those reading, I had some network issues that were causing slower speeds that were resolved from the original post. A failing sfp was throwing errors and improper network configuration on the router that had packets passing through the cpu a few times for processing and was slowing large transfers (over 1GB sustained utilization). Once that was resolved iperf passes > 9Gbps between the two units.

With the network issues out of the way, then it was using straight netcat without the encryption (ssh) that really sped things up and revealed that my next limitation is the amount of drives in the pool.