ZFS replication speed about half of SMB transfer speeds

[Mannekino]

I'm doing some testing with my new and old FreeNAS servers and ZFS replication seems slow. First off here are the specifications:

My new FreeNAS server (freenas)

• Motherboard: Supermicro X9SCM-F
• CPU: Intel Xeon E3-1260L
• Memory: 32GB ECC (4x 8GB Kingstong ValueRAM KVR1333D3E9SK2/16G)
• HBA: Dell PERC H200 (flashed in IT mode)
• Storage SDD: 2x Samsung PM863 240GB connected through the two onboard SATA3 (6Gb/s) ports
• Storage HDD: 4x Western Digital Red 4TB (connected through the HBA)
• Storage USB: 2x SanDisk Ultra Fit 32GB

My old FreeNAS server (freenas-test)

• System: HP Proliant Microserver N40L
• CPU: AMD Turion II Neo N40L
• Memory: 8GB ECC (2x 4GB)
• Storage HDD: 1x Seagate 3TB (ST3000DM001)
• Storage USB: 1x generic USB drive

I've created a new dataset on my freenas server and put a single file on it of roughly 6 GB. This is a dataset on the pool with the normal spinning disks. When I copy this file from my new freenas server to my own PC I'm getting just over 100 MB/sec read speed (my PC has an SSD).

When I then copy that file to my freenas-test server from my PC I see similar write performance:

This leads me to conclude that my new freenas server should be capable of sending at 100+ MB/sec and my old freenas-test server should be able to receive at a similar speed. After this I created a snapshot of the newly created dataset and used ZFS replication to copy it to my freenas-test server and I'm getting roughly 51 MB/sec as my transfer speed.

Code:

# zfs send data/testing@manual-20190210 | pv | ssh freenas-test zfs receive -F data/testing
5.57GiB 0:01:51 [51.1MiB/s]

Is this the kind of performance I should be expecting with ZFS replication or am I doing something wrong?

 

[l@e]

Can you check cpu utilization on the both of them?

 

[Mannekino]

During the ZFS replication operation? Just check with top of do you want me to check with something else?

 

[l@e]

Yes with top.

 

[Mannekino]

I did some searching and found that I have access to htop on FreeNAS. I did the replication again and here are the results. I took these screenshots halfway through the replication proces (so about 3 GB of the 6GB).

From my new FreeNAS server sending the data

From my old FreeNAS server receiving the data

CPU usage on the receiving server between 70-80% with spikes to 90+% average transfer speed again the same around 51 MB/sec.[/ICODE]

 

[l@e]

Ssh is killing the old nas cpu. So reaching the same 100+ MBps that you get from smb will be difficult.
Still not sure if it just it or anything on zfs replication proces uses sync r/w or not.
2 weeks ago did a replication test with my 2 boxes and it maxed the network, so didnt check cpu. They are both xeon E3.

 

[Mannekino]

Hmm, well it seems plausible that the CPU could be bottle-necking. I have another system I could try and see what kind of performance I will get with that server. My old FreeNAS server has a AMD Turion II Neo N40L processor which is not very fast.

 

[Chris Moore]

Ssh is killing the old nas cpu.

Weak CPU. Turn off encryption. It is all the calculation that is destroying the speed.

 

[Mannekino]

Hmm, I assume you mean turning it off for the SSH connection? I've done some searching on the forums and Google but not getting anywhere. Is it as simple as using a specific flag?

 

[Chris Moore]

I assume you mean turning it off for the SSH connection?

yes. just don't use ssh.
Look:
https://forums.freebsd.org/threads/...wo-servers-on-the-same-lan-without-ssh.67418/

 

[Mannekino]

Ah, I understand now, thanks for pointing me in the right direction. I found this also:

http://www.polyomica.com/improving-transfer-speeds-for-zfs-sendreceive-in-a-local-network/

I will do some more research and testing and if I manage to get it working I will post my solution. Could be next weekend, work again tomorrow.

 

[Mannekino]

I got some private feedback from @KrisBee and did some more reading. I managed to get optimum performance with nc.

Command on receiving server:

Code:

nc -l 8000 | pv | zfs receive -vF data/testing

Command on sending server:

Code:

zfs send -R data/testing@manual-20190210 | pv | nc 192.168.178.10 8000

Result:

Code:

received 5.57GB stream in 67 seconds (85.1MB/sec)

However I wanted to try mbuffer also but I can't connect and I don't know why, hopefully someone can help me troubleshoot.

Command on receiving server:

Code:

mbuffer -s 128k -m 1G -I 8000 | zfs receive -vF data/testing

Command on sending server:

Code:

zfs send -R data/testing@manual-20190210 | mbuffer -s 128k -m 1G -O 192.168.178.10:8000

Result:

Code:

mbuffer: warning: error connecting to 192.168.178.10:8000: Connection refused
mbuffer: error: unable to connect to 192.168.178.10:8000
mbuffer: warning: allocating more than half of available memory
mbuffer: error: unable to open output 192.168.178.10:8000: No such file or directory
mbuffer: fatal: no output left - nothing to do
warning: cannot send 'data/testing@manual-20190210': Broken pipe

I've confirmed mbuffer is listening on port 8000 with netstat -an

Code:

Active Internet connections (including servers)
Proto Recv-Q Send-Q Local Address          Foreign Address        (state)
tcp6       0      0 *.8000                 *.*                    LISTEN

Any idea what's going wrong with the mbuffer option?

 

[datashadow]

Hello @Mannekino.

I just encountered, and solved, the same problem. Look closely at the nestat -an output. You're listening on an IPV6 port as indicated by the tcp6 in the Proto column.

The solution for me was to specify -4 on the receiving side. So try:

Code:

mbuffer -4 -s 128k -m 1G -I 8000 | zfs receive -vF data/testing

Best of luck!

David

 

[Mannekino]

@datashadow ah, thanks a lot, I will give this a try. Makes sense, I didn't notice that.