Reinstall

[stajo]

Hi. After a current out my backup server wont boot anymore. It finds the boot and storage disks but it seems the boot sector is corrupt. I wish to revive it the easiest way possible to its former state before the current out. The backups on this server comes from replication tasks from my TrueNas Core Music and Movies servers, so the data doesnt change much. I have a backup from a few weeks ago on dropbox and I have flashed a fresh TrueNas 13 Core on a USB stick. Can anyone point me to a guide on how to restore it to its former state the easiest way? It is a pretty vanilla setup done some years ago and upgraded when there has come new versions. It is (was) connected to my local Active Directory, that is the only tweak I can think of.

 

[Jailer]

Boot from your Fresh install and upload a copy of your saved config file. You do have a copy of your current config file saved somewhere correct?

 

[stajo]

Boot from your Fresh install and upload a copy of your saved config file. You do have a copy of your current config file saved somewhere correct?

I do. So when in the process can I choose the upload of the config file?

 

[LarsR]

After the Installation when you First Login to the GUI you can import the old config file

 

[stajo]

So I did the fresh installation and imported the config file. All seems to have gone smooth, it looks the same as before. However my replications from music and movies servers fail. I get "Host key for server 'IP for the reinstalled server' does not match". How do I solve that easiest?

 

[chuck32]

I'm using scale not core. I'm not sure to what the message refers. I've never had problems although I reinstalled the destination machine multiple times.

Are you using ssh key authentication? If yes you probably need to add the public to your newly installed machine again. (Or setup a new ssh key pair), although I doubt it's that since I'd expect an error message that would read something like "access denied (public key)".

Or it refers to the fingerprint, then you need to find the known_hosts file on your source machine (probably under /root/.shh/, the one you didn't reinstall and delete all entries there that refer to the IP. Make a backup of that file first. Hopefully you should be prompted to confirm the host key on the next attempt to initiate the connection.

 

[stajo]

Yes I'm using ssh key authentication, that is the normal way when setting up replication. The message after ""Host key for server 'IP for the reinstalled server' does not match" follows as: "got 'long key string', expected 'other long key string". So it expected a specific ssh key and got back another. This message is on the source machines. How do I practically give the newly installed target machine the original ssh key?

 

[chuck32]

It it's referring to the authentication key you need to repeat the steps you did during initial configuration. You should be able to retrieve the public key from your source machine in the ssh section. There you should see a button download public key.
With that error message you can also confirm if that's the key that is asked for.

Edit: see here:

Configuring SSH Connections

Provides information on how to configure Secure Socket Shell (SSH) connections on your TrueNAS.

www.truenas.com

Maybe you can also just refer to the last section where it says discover remote key, if it's related to the known hosts.

 

[stajo]

Ok. I was looking for a way to transfer the ssh keys from host machines to the newly installed target machine first, before deleting and redoing all the replication tasks. I can under System -> SSH Connections see the keys used for replications. Those keys are not on the new target machine. The keys on the new target machine are keys that I have used before when using one time replications when reinstalling host machines on new hardware. But on the source machines under System -> SSH Connections I can choose Edit and under Edit: Discover remote host key. When I do that the ssh-rsa key changes. Maybe it imports it from the target machine? I have not pressed save after that yet, but can that be the correct method needed?

 

[chuck32]

Let me explain what I think:

1) Using SSH Keys to authenticate means that you need a keypair, a private key and a public key.
2) When establishing SSH connections the host is identified via a fingerprint.

It is still unclear to me which one exactly is the root cause of your problems.

I can under System -> SSH Connections see the keys used for replications. Those keys are not on the new target machine. The keys on the new target machine are keys that I have used before when using one time replications when reinstalling host machines on new hardware.

Okay, so first of all you want to revert back to using your original keypair, that was used in the replication. I assume these are the ones present on your source machine.

Under shh keypairs on your source machine, download the public key.

, expected 'other long key string". So it expected a specific ssh key and got back another.

Compare the public key to the expected key. If it matches, this should solve your issue.

On the remote machine, go to SSH keypairs and paste the public key.

If this does not solve the issue. Proceed with step tackling step 2.

Edit: Discover remote host key. When I do that the ssh-rsa key changes. Maybe it imports it from the target machine? I have not pressed save after that yet, but can that be the correct method needed?

Now, after you changed the public key on the remote system, see what key gets pulled when using the discover remote host key. I don't know if this pulls the public key or the fingerprint. If this pulls the fingerprint, save and hopefully you're done.
On my scale system the key displayed here is similiar but diffirent from my public key, so I assume this is the associated fingerprint. It should be safe to discover and save. Copy the key you see before and save it somewhere just in case.

If all of the above failed, you may try this, but probably it's better to report back here before to confirm:

Go to /root/.shh. Make a backup of known_hosts, i.e. sudo cp /root/.shh/known_hosts /root/.shh/known_hosts_backup. I assume this is the correct location. Next edit /root/.shh/known_hosts and remove all lines referring to the IP of your remote system and save. Try to initiate the ssh session.

How many replication tasks do you have? I wonder if it wouldn't be easier to just generate a new ssh connection and change the replication tasks.

 

[stajo]

Hi, thanks. Yesterday I tried to click "Discover remote host key" button under System -> SSH Connections on the source machines and Save, but that does not seem to have helped. The scheduled replications has failed with following log:

[2024/03/24 00:00:02] INFO [Thread-144] [zettarepl.paramiko.replication_task__task_1] Connected (version 2.0, client OpenSSH_8.8-hpn14v15)
[2024/03/24 00:00:02] INFO [Thread-144] [zettarepl.paramiko.replication_task__task_1] Authentication (publickey) failed.
[2024/03/24 00:00:02] ERROR [replication_task__task_1] [zettarepl.replication.run] For task 'task_1' non-recoverable replication error ReplicationError('Authentication failed.')

I have 8 replications on the movies server but only one on the music server so I tried to delete the current music replication and create a new including generating a new keypair. It failed with the following error message:

Error: Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/keychain.py", line 563, in remote_ssh_semiautomatic_setup
client = Client(os.path.join(re.sub("^http", "ws", data["url"]), "websocket"))
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 286, in __init__
self._ws.connect()
File "/usr/local/lib/python3.9/site-packages/middlewared/client/client.py", line 124, in connect
rv = super(WSClient, self).connect()
File "/usr/local/lib/python3.9/site-packages/ws4py/client/__init__.py", line 216, in connect
self.sock.connect(self.bind_addr)
File "/usr/local/lib/python3.9/ssl.py", line 1343, in connect
self._real_connect(addr, False)
File "/usr/local/lib/python3.9/ssl.py", line 1330, in _real_connect
super().connect(addr)
ConnectionRefusedError: [Errno 61] Connection refused

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 139, in call_method
result = await self.middleware._call(message['method'], serviceobj, methodobj, params, app=self)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1251, in _call
return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
File "/usr/local/lib/python3.9/site-packages/middlewared/main.py", line 1156, in run_in_executor
return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
File "/usr/local/lib/python3.9/concurrent/futures/thread.py", line 58, in run
result = self.fn(*self.args, **self.kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/schema.py", line 985, in nf
return f(*args, **kwargs)
File "/usr/local/lib/python3.9/site-packages/middlewared/plugins/keychain.py", line 565, in remote_ssh_semiautomatic_setup
raise CallError(f"Unable to connect to remote system: {e}")
middlewared.service_exception.CallError: [EFAULT] Unable to connect to remote system: [Errno 61] Connection refused


I can ping the target machine from the source machine just fine

 

[stajo]

I shall try to leave and reconnect to my AD domain with the reinstalled machine

EDIT. Left and successfully rejoined AD with reinstalled target machine, tried to build new replication and got same error message as above. Unable to connect to remote system: [Errno 61] Connection refused

 

[chuck32]

[zettarepl.paramiko.replication_task__task_1] Authentication (publickey) failed.

The destination machine needs to be supplied with the public key, that corresponds to your private key of the source machine, as I described above.

Try and follow my outline up until the point where I recommend reporting back, so adding the pub key and then discovering.

 

[stajo]

Ok, I downloaded the Public Key on one of the source machine. Then I cut'n'pasted from yesterday nights error message "got-expected" and none of the 3 maches. But in between that I had done the Discover remote key procedure which has changed the key, so maybe not surprising. More surprising is that I have no error message from tonights replication under alerts, so I have nothing to cut'n'paste from. But it has not been running.

Is there somewhere else I can find the expected key to compare with?

 

[chuck32]

I think you made it way harder on yourself since you didn't copy over the public key in the beginning as I suggested.

Okay so let's see, on both your remote and source machine, show screenshots of the ssh connections you setup (redact most of your keys please). I need to see what's the status quo.

and create a new including generating a new keypair. It failed with the following error message:

I don't know that error, using the semiautomatic method and starting with a fresh connection should work when you can reach the machine. The SSH Service is running on both machines?

 

[chuck32]

More surprising is that I have no error message from tonights replication under alerts,

So you have a working ssh connection now for some machines/connections?

 

[stajo]

So you have a working ssh connection now for some machines/connections?

No, or I have no working replications at least, they have not gone tonight either. Just no fresh error messages under Alerts, they are from yesterday nights failed replications. Last nights failed replications did not produce anything under Alerts. They produced error messages under Replication Tasks tho, the "[Errno 61] Connection refused" logs above.

SSH was running on the reinstalled target machine and on the Music machine that I just tried and failed deleting and rebuilding the replications for, but it was not running on the Movies machine for some reason. Started that again now.

Attaching a screenshot of the Music source machines SSH connection. But the reinstalled target machine does not have this under SSH Connections.

 

[stajo]

Maybe I should just delete all Replications, all SSH Connections and all SSH Keypairs, reboot everything and try start fresh?

 

[chuck32]

But the reinstalled target machine does not have this under SSH Connections.

Setup according to documentation, the target machine needs the corresponding public key.

 

[stajo]

It says in the SSH documentation you posted above that "You must also configure the remote system to allow root access with SSH". Should I tick the first box (under Services -> SSH?