Replication task fails on second run

[t3chn0g33k]

The Secondary TrueNAS server is a fresh build with only three datasets (data, storage, virtual). Replication is a PUSH from the Primary to the Secondary using SSH as the Transport. There are some child dataset exclusions. The first replication runs with no issues however, subsequent runs fail with the error below. I can delete the Dataset and the replication will run with success the first time again but fail on the second run.

CRITICAL
Replication "virtual replication" failed: can't compare offset-naive and offset-aware datetimes..
2022-01-09 23:01:46 (America/Chicago)

One TrueNAS Scale server is running on a nightly version at: TrueNAS-22.02-MASTER-20220102-172927 and is the "Primary" and the other "Secondary" server is running TrueNAS Scale at version: TrueNAS-SCALE-22.02-RC.2

Primary Server Specs:
- Dell PowerEdge R720xd w/Intel(R) Xeon(R) CPU E5-2630L v2 @ 2.40GHz | 12 cores (24 threads)
- 96GB ECC Memory
- 3 Pools
- data w/8 drives (RAIDZ2) total 16 TiB
- storage w/2 drives (MIRROR) total 6 TiB
- virtual w/2 drives (MIRROR) total 256 GiB

Secondary Server Specs:
- This one is a virtual machine running inside Proxmox on an HP DL380p Gen8
- Processors: 8 (4 sockets, 2 cores) | Memory: 16 GiB | Disk: 1x10 TiB
- 1 pool with 3/datasets
- data w/ReadOnly set to true
- storage w/ReadOnly set to true
- virtual w/ReadOnly set to true

The primary has several Apps running and a virtual machine however, the Secondary is a vanilla build with only replication in mind.

All 3 Replication tasks kick off behind their respective Periodic Snapshot task and are configured as shown in the screenshots.

 

[t3chn0g33k]

I've tried changing both system to use `Etc/UTC` as the time zone but that does not make a difference. It still allows me to replicate once successfully and then fails with the same type of message "...failed: can't compare offset-naive and offset-aware datetimes".

 

[t3chn0g33k]

From what I can tell from the google-gods this error may be specific to underlying python code but haven't been able to uncover a potential fix/workaround as of yet.

Any thoughts?

 

[t3chn0g33k]

I am more than happy to add more information, I just need to know what will to provide. Is replication between two TrueNAS systems not a popular approach?

 

[CompuLen]

I am also experiencing this very issue while replicating TrueNAS Scale to TrueNAS Core

 

[TheSwert]

I am also experiencing this issue replicating from TrueNAS Scale to TrueNAS Core. The issue only materialized after upgrading to the 22.02 Release version yesterday. I did not have the issue when I was still on RC2.

The TrueNAS Core instance I'm replicating to is on 12.0 U8.

 

[and1049]

I'm also having this issue but with both systems on the latest Scale release. TrueNAS Scale (22.02 release) to TrueNAS Scale (22.02 release).
I did not test this on previous versions of Scale, both systems are new installs.

Thanks!

 

[leepz]

Also experiencing the same error message between my recently upgraded primary and off-site TrueNAS SCALE deployments.
Both servers were updated from RC2 to 22.02, and the working Replication task was untouched. No issues at all prior with RC1 or RC2.

"Replication...failed: can't compare offset-naive and offset-aware datetimes"

I've been able to regain replication functionality however by downgrading the TrueNAS SCALE release from 22.02 back to RC2 on my primary server, leaving the target system still on 22.02. So it looks to be a problem only on the system with the source replication job configured.

Hopefully a fix soon as I would expect replication functionality is a core feature of a production ready release :)

 

[ClassicGOD]

Since I'm having the same issue with Release version I reported it on iXsystem Jira. If anyone is interested: https://jira.ixsystems.com/browse/NAS-114986

 

[francisaugusto]

Same issue here.

 

[Analius]

I have the same issue, though I am replicating within the same system.
No custom snapshot names like @ClassicGOD explains in his issue though.

 

[geoffmyers]

I'm experiencing the same issue. I just commented on the Jira issue (NAS-114986, https://jira.ixsystems.com/browse/NAS-114986). Here is the error message:

Code:

Error: Traceback (most recent call last):
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 423, in run
await self.future
File "/usr/lib/python3/dist-packages/middlewared/job.py", line 459, in __run_body
rv = await self.method(*([self] + args))
File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1261, in nf
return await func(*args, **kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/replication.py", line 454, in run
await self.middleware.call("zettarepl.run_replication_task", id, really_run, job)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1318, in call
return await self._call(
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1286, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1186, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 380, in run_replication_task
self._run_replication_task_job(f"task_{id}", job)
File "/usr/lib/python3/dist-packages/middlewared/plugins/zettarepl.py", line 440, in _run_replication_task_job
raise CallError(make_sentence(message.error))
middlewared.service_exception.CallError: [EFAULT] can't compare offset-naive and offset-aware datetimes.

 

[Number1Onion]

I'm continuing to see the same issue after installing SCALE-22.02.1 (Angelfish). Based on jira, it appears to be fixed?

https://jira.ixsystems.com/browse/NAS-115299?jql=project%20%3D%20NAS%20AND%20fixVersion%20%3D%20%22SCALE-22.02.1%20(Angelfish)%22

 

[arkan]

Hi, I'm having the same problem before and after I've updated my system today 27/03/2022.

On scheduled time it does not do the replication, and if I click to run manually the replication from the "Run Now" button, it will also fail.

 

[Micin]

Hello ppl. im also experiencing the same issue, BUT i found an interesting thing in my settings. So try to look if you have same settings, changing this resolved my issue ;)

Snapshot retention on SOURCE is set for 2 weeks. in replication tasks i wanted backups for twice that in the past, so i set retention for 4 weeks. 1st replication worked, but next failed. when i changed the retention policy on backup task to "same as source" and ran the task again, it worked.

So please try this if you have diferent settings on your Snapshot retention and replication task "retention policy" and let others know, if i was just lucky or the issue is resolved by this :)

 

[arkan]

@Micin Thank you, I've done the same as you mentioned, and it worked.

 

[Micin]

well, it looks like replication does not like diferent snapshots (date/time perspective meaning, im not sure if setting up periodic snapshots on "destionation / backup server" will break replication again. more testing needed on this matter.