Replication error "out of memory"

Pitfrr

Wizard
Joined
Feb 10, 2014
Messages
1,523
Hello,

I try to replicate a snapshot from a remote system to a main system.
FreeNAS 9.10.2 running in a VM under ESXi, 64GB or RAM
TrueNAS 12.0-U1, supermicro X9SCM, Xeon 1220L-V2, 32GB of RAM


  • Replication:
All the replication tasks are on the remote system and several tasks already exist to pull snapshots from the main to the remote system. Those tasks are working fine.
Now I added a task to push from the remote to the main system and this one is causing problem with the error "out of memory" with the following log:

Code:
[2021/03/20 08:51:09] INFO     [Thread-401] [zettarepl.paramiko.replication_task__task_7] Connected (version 2.0, client OpenSSH_7.4-hpn14v5)
[2021/03/20 08:51:10] INFO     [Thread-401] [zettarepl.paramiko.replication_task__task_7] Authentication (publickey) successful!
[2021/03/20 08:51:10] INFO     [replication_task__task_7] [zettarepl.replication.run] For replication task 'task_7': doing push from 'Tank/Data' to 'Tink/Data' of snapshot='auto-2021-03-18_00-00' incremental_base=None receive_resume_token=None encryption=False
[2021/03/20 08:51:11] ERROR    [replication_task__task_7] [zettarepl.replication.run] For task 'task_7' unhandled replication error ExecException(141, 'internal error: out of memory\n')
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/run.py", line 154, in run_replication_tasks
    run_replication_task_part(replication_task, source_dataset, src_context, dst_context, observer)
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/run.py", line 237, in run_replication_task_part
    run_replication_steps(step_templates, observer)
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/run.py", line 514, in run_replication_steps
    replicate_snapshots(step_template, incremental_base, snapshots, encryption, observer)
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/run.py", line 565, in replicate_snapshots
    run_replication_step(step, observer)
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/run.py", line 641, in run_replication_step
    ReplicationProcessRunner(process, monitor).run()
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/process_runner.py", line 33, in run
    raise self.process_exception
  File "/usr/local/lib/python3.8/site-packages/zettarepl/replication/process_runner.py", line 37, in _wait_process
    self.replication_process.wait()
  File "/usr/local/lib/python3.8/site-packages/zettarepl/transport/ssh.py", line 146, in wait
    self.async_exec.wait()
  File "/usr/local/lib/python3.8/site-packages/zettarepl/transport/async_exec_tee.py", line 100, in wait
    raise ExecException(exit_event.returncode, self.output)
zettarepl.transport.interface.ExecException: internal error: out of memory


The last line says zettarepl.transport.interface, does it has to do with the transport option in the task's parameters?
The only options available here are SSH, SSH+NETCAT and LOCAL. I'm using SSH because LOCAL will not be working since the replication is sent to the main server.
In the tool tip (for help) LEGACY is mentioned but not available in the drop down list.
The SSH connection is working since I use it for the other tasks (and I can browse the volume on the main server).

Could it also be an incompatibility between FreeNAS 9.10.2 and TrueNAS 12.0?
--> but then pulling works and not pushing? I wouldn't have expected that.

If you have any thoughts, I'd be happy to hear about.
I haven't tried manual replication (zfs send/receive), I'll keep that as last option if I can't get the automated replication to work.
 

Isaias

Dabbler
Joined
Nov 6, 2014
Messages
11
Hello!

I have exactly the same problem. I have a server with truenas 12.0-U7 and the replicas fail on the other one I have with FreeNAS-9.10.2-U6. They only fail in direction from Truenas to Freenas. The other way round they still work.

Let's see if someone can give us a solution. I attach the debug error logs.
 

Attachments

  • 389.rar
    2.8 KB · Views: 123

Ericloewe

Server Wrangler
Moderator
Joined
Feb 15, 2014
Messages
20,175
9.10 is pretty old and the replication stuff was recently rewritten substantially. You should probably update.
 

Pitfrr

Wizard
Joined
Feb 10, 2014
Messages
1,523
I don't remember how I solved it.... it's been a while and I didn't document it! ;-(

But here is the configuration I'm using and it works:
Direction: PULL Transport: SSH Number of retries for failed replications: 5 Stream compression: disabled (I think this one was causing the problem... I vaguely remember something...)
Limits: none Allow blocks larger than 128kB: checked Allow compressed write records: unchecked

In the source section, only Inlcude dataset property is checked
The rest is unchecked.

In the destination section:
Destination dataset read only property: ignore Encryption and Synchornize...: unchecked Sanpshot retention policy: Same as source



Don't know if this will help though... but maybe. :smile:
Good luck.
 

Isaias

Dabbler
Joined
Nov 6, 2014
Messages
11
Thanks you pitfrr.

I thinkg that the problem is de PUSH direction. PULL direction work well. I had tried in other truenas and work well.

Regards.
 
Last edited:

Pitfrr

Wizard
Joined
Feb 10, 2014
Messages
1,523
For me the PULL from TrueNAS to FreeNAS made more sense since the TrueNAS is the backup server and the FreeNAS is the main server.
The backup server might be down sometime so pulling from it was more convenient.
 
Top