Replication Using SSH and SSH+NETCAT

[Peter_M]

Hi all,

I have two HP Micro N54L servers running 16Gb of memory and an onboard gigabit network card. I have set up multiple datasets and setup snapshots and replication between the two using SSH without stream compression. All good really pleased.

However watching the network performance during replication it is pegged at about 95 Megabits/sec The CPU is not struggling it fluctuates about 15-20%. If I use SMB transfer and just copy a file to a dataset I can get ~300 Megabits/s still not in Gigabit territory but nearly 3 times the Replication speed.

I was thinking of switching to a non-SSH transfer to see if the SSH thing was the issue (thoughts?) and the SSH+NETCAT seemed like an option. However that option opens a whole bunch of additional configuration dialogue boxes.

So my questions are:

1 - Is SSH likely to be the reason for my poor LAN transfer speeds. I don’t need a permanent secure transport as my replication target server is still in my house just a different part. I need the transfer speed to increase and especially as I’m doing a lot of data take on. I could switch to SSH later.

2 - If that is likely to be the case, has anyone tried the SSH+NNETCAT option how stable is it, did you see a noticeable increase in speed?

3 - Can someone walk me through the SSH+NETCAT configuration please and go slow on the port selection part as I’m not familiar with port selection, is just “pick a number any number” or is there more to it? Also is there a need to do any config to the Target machine? Primary is 192.168.1.2 - Target is 192.168.1.4

From a FRING check of open ports the only ports open on both servers are 22, 80, 139, 443, 445, 6000 and they all seem to have a use predefined.

Many thanks in advance

 

[Adrian]

I let all the Netcat port selection options to default.

 

[Peter_M]

Adrian, Thank you.

I'm not familiar NETCAT at all or it's settings

There are Local and Remote settings which include

Netcat Active Side
Netcat Active Side Listen Address
Netcat Active Side Min Port
Netcat Active Side Max Port
Netcat Active Side Connect Address

Do I need to set up both, what do they mean?

Many thanks

Peter

 

[Adrian]

I set Active Side to LOCAL, and default everything else. I suspect you might need the others if you have a firewall in between the machines.

 

[Peter_M]

Hi Adrian, I tried this I don't think it worked. I'll stick with the SSH for now.

 

[Pitfrr]

I'm using the netcat option to send a snapshot from my main server to a backup server on the LAN.
The difference in speed is quite noticeable (on Gigabit network). If I recall correctly, with replication over the GUI, I had around 300Mb/s. When using netcat, it goes up to +900Mb/s (and CPU around 30%).

I'm using following command in a terminal:

Source:
zfs send [pool]/[Dataset]@[snapshot_name] | nc -l [port number]

Target:
nc [source IP address] [port number] | zfs recv [Pool]/[Target dataset]

Example:
zfs send MainPool/Dataset@manual_snapshot | nc -l 3333
nc 192.168.1.2 3333 | zfs recv Backup/Data

Remarks:

• The port number is any number as long as it's the same for both
• You can get the snapshot list with zfs list -t snapshot
  • or for a specific dataset: zfs list -r -t snapshot -o name,creation pool/dataset
• If the target dataset does not exist it will be created
• I use screen so the session is not killed when the terminal is closed
• I start with the source command
• There is no progress indicator (I tried to play with dpv but didn't get it to work)
  • If anyone has a proposal, I'd be happy
• There is no indication when the transfer is finished either (I look at the CPU and network activity)
  • Ending it with Ctrl+C
  • If anyone knows how to do it differently (i.e. automatically), I'd be happy

 

[Peter_M]

Hi Pitfrr, thanks for the info I'll give it a try.

I also found I have a choke point in the network somewhere. Connecting the servers through a single switch with cat6 cables brings a pronounced positive change in speed.

As for you transfer end issue I've just invested in a FreeBSD book if it says how to respond to the end of a transfer I'll let you know.

Thanks again.

Peter

 

[Dunuin]

I also tried to use netcat but I only get this error as soon as I try to select a destination dataset:

[EFAULT] AttributeError("'NoneType' object has no attribute 'lower'")
Error: Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zettarepl.py", line 402, in list_datasets
async with self._get_zettarepl_shell(transport, ssh_credentials) as shell:
File "/usr/local/lib/python3.7/contextlib.py", line 170, in __aenter__
return await self.gen.__anext__()
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zettarepl.py", line 632, in _get_zettarepl_shell
transport_definition = await self._define_transport(transport, ssh_credentials)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zettarepl.py", line 655, in _define_transport
transport_definition["active-side"] = netcat_active_side.lower()
AttributeError: 'NoneType' object has no attribute 'lower'

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 130, in call_method
io_thread=False)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1084, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/schema.py", line 961, in nf
return await f(*args, **kwargs)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/replication.py", line 703, in list_datasets
return await self.middleware.call("zettarepl.list_datasets", transport, ssh_credentials)
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1141, in call
app=app, pipes=pipes, job_on_progress_cb=job_on_progress_cb, io_thread=True,
File "/usr/local/lib/python3.7/site-packages/middlewared/main.py", line 1084, in _call
return await methodobj(*args)
File "/usr/local/lib/python3.7/site-packages/middlewared/plugins/zettarepl.py", line 405, in list_datasets
raise CallError(repr(e))
middlewared.service_exception.CallError: [EFAULT] AttributeError("'NoneType' object has no attribute 'lower'")

Using just SSH works fine and a destination dataset tree will show up.

What are the right options for this fields?
Netcat Active Side
Netcat Active Side Listen Address
Netcat Active Side Min Port
Netcat Active Side Max Port
Netcat Active Side Connect Address

I used "Netcat Active Side = Local" so my Homeserver is the active side.
I used "Netcat Active Side Listen Address = 192.168.43.10" because that is the IP of my Homeserver in the same Network as my Backupserver.
I tried leaving "Min Port" and "Max Port" blank or using some random chosen ports (9000 tot 10000).
I'm not sure what to insert into "Netcat Active Side Connect Address". Hostname of myHomeserver is "BM-Homeserver" with IP 192.168.43.10/24 and 192.168.42.9/24. Hostname of my Backupserver is "BM-Backupserver" with IP 192.168.43.20/24.

There is no firewall between my Homeserver and my Backupserver but I didn't manually opened any ports in FreeNAS. Do I need to open them somehow? I found nowhere a menu to do this.

Edit:
Both servers are version "FreeNAS-11.3-U3.2".

 

[leecz]

I have encounter the same error when trying to select a destination dataset using SSH+C as Dunuin described.
Both servers are version "FreeNAS-11.3-U3.2" too.

 

[Richard Durso]

Still have this problem on "FreeNAS-11.3-U4.1".

 

[socra]

I also tried to use netcat but I only get this error as soon as I try to select a destination dataset:



Using just SSH works fine and a destination dataset tree will show up.

What are the right options for this fields?
Netcat Active Side
Netcat Active Side Listen Address
Netcat Active Side Min Port
Netcat Active Side Max Port
Netcat Active Side Connect Address

I used "Netcat Active Side = Local" so my Homeserver is the active side.
I used "Netcat Active Side Listen Address = 192.168.43.10" because that is the IP of my Homeserver in the same Network as my Backupserver.
I tried leaving "Min Port" and "Max Port" blank or using some random chosen ports (9000 tot 10000).
I'm not sure what to insert into "Netcat Active Side Connect Address". Hostname of myHomeserver is "BM-Homeserver" with IP 192.168.43.10/24 and 192.168.42.9/24. Hostname of my Backupserver is "BM-Backupserver" with IP 192.168.43.20/24.

There is no firewall between my Homeserver and my Backupserver but I didn't manually opened any ports in FreeNAS. Do I need to open them somehow? I found nowhere a menu to do this.

Edit:
Both servers are version "FreeNAS-11.3-U3.2".

Same error here with 11.3U5

 

[JaimieV]

You don't need to fill anything in those fields, as per Adrian's screenshot back in May.

(There's a bug in 11.3u5 which means you can't edit an existing replication, btw)

I can't repeat the error you're getting, on a new replication I just select "no encryption" and otherwise don't need to mess with it.

 

[sophware]

I get the same error, every time, when creating the task using "ADVANCED REPLICATION CREATION". This is the "no attribute 'lower'" error, when selecting a destination dataset. If you're seeing a "no encryption" option, you're not in the advanced creation route. I need this 1) because I need to choose the "from scratch" option and 2) I can't edit after b/c of the 11.3u5 bug you mentioned.

I almost found a work-around by selecting the destination dataset while the transport was still SSH. After making the selection, I was able to change to SSH-NETCAT, choose remaining options, and save. The job ran with an error, though. Maybe that error was something that happens because of something specific to my situation, and the work-around will go better for others.

 

[sophware]

Update: the work-around works.

 

[KevonLaity]

Hi... Utilizing HPN-empowered OpenSSH and the None code, we're ready to immerse (980 Mbps) a gigabit interface between two ZFS has for a ZFS send/recv activity. With HPN and ordinary encryption, we get around 400-ish Mbps. Also, with non-HPN (otherwise known as "would be expected) and ordinary encryption, we experienced difficulty getting to 300 Mbps. Between similar two workers.

 

[maestrx]

zfs send MainPool/Dataset@manual_snapshot | nc -l 3333
nc 192.168.1.2 3333 | zfs recv Backup/Data

I was using the same till I found mbuffer that is bundled in the BSD. Works the same like nc, includes buffering and live traffic statistics on top.
mbuffer -s 128k -m 512M -4 -I 1337 | zfs receive -Fdv primarypool
zfs send -Rv primarypool@sync_20200528 | mbuffer -s 128k -m 512M -O a.b.c.d:1337

 

[runevn]

Update: the work-around works.

Thanks, that worked for me as well.

 

[Dunuin]

Still the same Problem two years later with TrueNAS 12.0U8 :/

Edit:
And yes, looks like the workaround is still working (setting up the replication task as SSH and only at the end. right before hitting the save button, switching from SSH to SSH+NETCAT).

 

[JustinOtherBobo]

Wish I had scrolled to the bottom first :)

12.0-U8.1 same-o-same-o, use work-around of setting up as SSH first.

Since editing a replication job now works, I configured the replication job as SSH and saved it.
Then I edited it by changing the transport to SSH+NETCAT and setting Netcat active side to LOCAL. The remaining NETCAT options were left as blank defaults.