TrueNAS HA and sync configs

[sekroots]

Hello folks,

I am looking for a setup to provide High Avaiability (HA) using TrueNAS.

I have 2 instances (environment) Proof of Concept:
- TrueNasA and TrueNasB, each one has 6 HDDs to store data;
- iSCSI Multipath/MPIO;

Actually I am using TrueNasA as active, with periodic snapshots and rsync to TrueNasB as standby. I built a Windows Cluster with 2 Nodes stored at TrueNasA pool.
I saved configs from A, shutdown A and restored backup from A at node B and it works fine.

Questions:
1) How can I sync necessary configs from TrueNasA to TrueNasB?
2) How can I provide HA at this setup?
3) Is there a way to keep TrueNasA´s pool online updated at TrueNasB´s pool?
4) any suggestions to my POC?


Thanks

Ricardo

 

[Patrick M. Hausen]

Buy TrueNAS Enterprise. HA included.

 

[sekroots]

Hello Patrick, thanks for your reply. I sent a message to them, waiting for reply.

No more other options?

 

[Patrick M. Hausen]

Other than programming your own from scratch none that I was aware of.

You can do a poor man's HA by replication tasks that run e.g. hourly. This is what we do with two hypervisor systems. Half of the VMs run on each of the nodes and zvols containing the virtual disks are replicated to the other node. In case one fails we would manually recreate the VMs and start them on the remaining node.

 

[sekroots]

Ok, lets forget about HA for a moment.

Can anyone help me how to sync configs from a TrueNas node to another TrueNas node?

 

[Patrick M. Hausen]

There is an API for everything the web UI does. Literally. The UI uses that API. In your UI click on on the cogwheel in the upper right corner, then on "API Keys", then on "DOCS" to get at the documentation.

The config.save and config.upload methods probably are what you are after. Some assembly (programming) required

 

[sekroots]

 

[Patrick M. Hausen]

Yes. These are the API points. What is the point of your last post?

 

[sekroots]

I just posted it to help next users when searching about this post.

BTW, reading those API points docs didn´t help me yet. I am still looking for how to sync configs between TrueNAS.

 

[Patrick M. Hausen]

Save config on NAS A, upload config to NAS B. Anything better can be readily bought from iXsystems. And if you don't want to do that you need to program your own synchronisation using the provided API. Implement what iXsystems have done yourself or pay them for their solution. There is no free sync, because that is what they live from to give us the free TrueNAS.

 

[sekroots]

Right. I am already doing this setps.
Thanks anyway

 

[sekroots]

How about this situation:

- NasA has periodic snapshots and sync to NasB; OK
- Backup configs from NasA and NasB; OK
- For some reason, NasA goes down, after my NOC team detects it, they will restore NasA configs to NasB(it will reboot as NasA); OK
- my VMs/cluster will work again and same NEW DATA at current NasA(but its original NasB disks); OK
- after some time, IT team will restore original NasA ( it will boot with no network to avoid IP duplication); OK
- we will perfom a snapshot from NEW DATA at NasA(but its original NasB disks); OK
- how can I restore this new snapshot? Not Ok!


Can anyone tell me how can I manually download this new snapshot to restore it when the original NasA is back online?

Thanks

 

[Patrick M. Hausen]

You bring up NAS B with the latest snapshot. Can be as frequent as every couple of minutes. NAS B is now NAS A and running in production. So it will receive new data.

You repair former NAS A and bring it into operation as new NAS B reversing the roles. Start replication anew with the restored NAS as the target/standby.

You cannot add data from snapshots if the target already has newer data written. So you need to switch roles every time there is a major disruption.

 

[sekroots]

I had some problems during tests:

Former NasB ( actually working as NasA as main storage) and former NasA ( actually as NasB) I tried to rsync data from A to B:

2022/03/23 11:51:17] INFO [Thread-22] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.4-hpn14v15)
[2022/03/23 11:51:17] INFO [Thread-22] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) successful!
[2022/03/23 11:51:18] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'tank/Windows' to 'tank/Windows' of snapshot='auto-2022-03-23_10-05' incremental_base='auto-2022-03-23_05-05' receive_resume_token=None encryption=False
[2022/03/23 11:51:19] WARNING [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 1 recoverable replication error RecoverableReplicationError('cannot receive incremental stream: most recent snapshot of tank/Windows does not\nmatch incremental source\nBroken pipe.')
[2022/03/23 11:51:19] INFO [replication_task__task_1] [zettarepl.replication.run] After recoverable error sleeping for 1 seconds
[2022/03/23 11:51:21] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'tank/Windows' to 'tank/Windows' of snapshot='auto-2022-03-23_10-05' incremental_base='auto-2022-03-23_05-05' receive_resume_token=None encryption=False
[2022/03/23 11:51:22] WARNING [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 2 recoverable replication error RecoverableReplicationError('cannot receive incremental stream: most recent snapshot of tank/Windows does not\nmatch incremental source\nBroken pipe.')
[2022/03/23 11:51:22] INFO [replication_task__task_1] [zettarepl.replication.run] After recoverable error sleeping for 2 seconds
[2022/03/23 11:51:26] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'tank/Windows' to 'tank/Windows' of snapshot='auto-2022-03-23_10-05' incremental_base='auto-2022-03-23_05-05' receive_resume_token=None encryption=False
[2022/03/23 11:51:27] WARNING [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 3 recoverable replication error RecoverableReplicationError('cannot receive incremental stream: most recent snapshot of tank/Windows does not\nmatch incremental source\nBroken pipe.')
[2022/03/23 11:51:27] INFO [replication_task__task_1] [zettarepl.replication.run] After recoverable error sleeping for 4 seconds
[2022/03/23 11:51:32] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'tank/Windows' to 'tank/Windows' of snapshot='auto-2022-03-23_10-05' incremental_base='auto-2022-03-23_05-05' receive_resume_token=None encryption=False
[2022/03/23 11:51:33] WARNING [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 4 recoverable replication error RecoverableReplicationError('cannot receive incremental stream: most recent snapshot of tank/Windows does not\nmatch incremental source\nBroken pipe.')
[2022/03/23 11:51:33] INFO [replication_task__task_1] [zettarepl.replication.run] After recoverable error sleeping for 8 seconds
[2022/03/23 11:51:42] INFO [replication_task__task_1] [zettarepl.replication.run] For replication task 'task_1': doing push from 'tank/Windows' to 'tank/Windows' of snapshot='auto-2022-03-23_10-05' incremental_base='auto-2022-03-23_05-05' receive_resume_token=None encryption=False
[2022/03/23 11:51:43] WARNING [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' at attempt 5 recoverable replication error RecoverableReplicationError('cannot receive incremental stream: most recent snapshot of tank/Windows does not\nmatch incremental source\nBroken pipe.')
[2022/03/23 11:51:43] ERROR [replication_task__task_1] [zettarepl.replication.run] Failed replication task 'task_1' after 5 retries

Workaround:
I deleted all zvols from NasB and run rsync task again. It worked.

Questions:
- Is it possible add a question during TrueNAS boot to choose which role it will assume: A then load stored configA.db or B then load configB.db ?

 

[Patrick M. Hausen]

I deleted all zvols from NasB and run rsync task again. It worked.

The log you provide does not look like rsync. Are you sure you are using rsync and not ZFS replication?

Is it possible add a question during TrueNAS boot to choose which role it will assume: A then load stored configA.db or B then load configB.db ?

Of course - checkout the source code, build your own version of TrueNAS with that function added.
Not with TrueNAS as it is.

 

[sekroots]

Yes, I got a mistake, its was a replication task.

 

[mikeerm]

There is an API for everything the web UI does. Literally. The UI uses that API. In your UI click on on the cogwheel in the upper right corner, then on "API Keys", then on "DOCS" to get at the documentation.

The config.save and config.upload methods probably are what you are after. Some assembly (programming) required

Could you give me a clue how to call this with the cli command from the shell?

Code:

cli -c 'system config save configsave="{ secretseed: true, pool_keys: true, root_authorized_keys: true }"'

results in

Code:

Error: ValueError("Pipe 'output' is not open")

Any help highly appreciated. I tried many, many things already...