Upgraded to FreeNAS 11.3-U4.1 and SSH service failed to start

[ApolloDS]

I upgraded two FreeNAS systems from 11.3-U4 to U4.1 and since then I get the error "SSH service failed to start" when starting the sshd manually.
The same effect is on two upgraded systems from U4 to U4.1.
In the /var/log/messages I see

Code:

root: /usr/local/etc/rc.d/openssh: WARNING: failed precmd routine for openssh

How to troubleshoot this?
Is there any startup-debugging possible?

 

[Fredda]

When I see it correctly you openssh does two precmd calls before it actually starts the demon, you can run them manually and check if they both run successfully or if one of them is the problem
service openssh keygen service openssh configtest
Also check in the GUI the SSH setting, goto Services->SSH, any unusual settings there?

If that does not help, next steps would be to check if openssh starts if called directly and to check the sshd_config files.

 

[ApolloDS]

Hi @Fredda ,
Thank you for these hints.
Well, I deleted the Host Keys in /usr/local/etc/ssh/ssh_host* and then started the ssh with

Code:

service openssh start

The host keys were created and the sshd is running again.

But when I stop and start again in the gui I will get the exact same error again.

How can I figure out what the "failed precmd routine" for sshd means?

 

[Fredda]

Does a service openssh restart work after you've successfully started the openssh?
The precommands for openssh are basically the 2 commands mentioned above, so if one of them fails the openssh start will fail.

 

[ApolloDS]

OK one step further... ssh host keys are not right created but why??
I could do a test with debug like this:

Code:

# /usr/local/sbin/sshd -t -ddd
debug2: load_server_config: filename /usr/local/etc/ssh/sshd_config
debug2: load_server_config: done config len = 477
debug2: parse_server_config: config /usr/local/etc/ssh/sshd_config len 477
debug3: /usr/local/etc/ssh/sshd_config:1 setting Subsystem sftp    /usr/libexec/sftp-server -l DEBUG3 -f AUTH
debug3: /usr/local/etc/ssh/sshd_config:2 setting Protocol 2
debug2: /usr/local/etc/ssh/sshd_config line 2: Deprecated option Protocol
debug3: /usr/local/etc/ssh/sshd_config:3 setting UseDNS no
debug3: /usr/local/etc/ssh/sshd_config:4 setting ChallengeResponseAuthentication no
debug3: /usr/local/etc/ssh/sshd_config:5 setting ClientAliveCountMax 3
debug3: /usr/local/etc/ssh/sshd_config:6 setting ClientAliveInterval 15
debug3: /usr/local/etc/ssh/sshd_config:7 setting NoneEnabled yes
debug3: /usr/local/etc/ssh/sshd_config:8 setting VersionAddendum none
debug3: /usr/local/etc/ssh/sshd_config:9 setting Ciphers +aes128-cbc
debug3: /usr/local/etc/ssh/sshd_config:10 setting Port 22
debug3: /usr/local/etc/ssh/sshd_config:11 setting ListenAddress 127.0.0.1
debug3: /usr/local/etc/ssh/sshd_config:12 setting ListenAddress fe80::21f:d0ff:fe5a:538a
debug3: /usr/local/etc/ssh/sshd_config:15 setting PermitRootLogin yes
debug3: /usr/local/etc/ssh/sshd_config:16 setting AllowTcpForwarding yes
debug3: /usr/local/etc/ssh/sshd_config:17 setting Compression delayed
debug3: /usr/local/etc/ssh/sshd_config:18 setting PasswordAuthentication yes
debug3: /usr/local/etc/ssh/sshd_config:19 setting PubkeyAuthentication yes
debug1: HPN Buffer Size: 131072
debug1: sshd version OpenSSH_8.0, OpenSSL 1.0.2s-freebsd  28 May 2019
debug1: Unable to load host key "/usr/local/etc/ssh/ssh_host_rsa_key": invalid format
debug1: Unable to load host key: /usr/local/etc/ssh/ssh_host_rsa_key
debug1: Unable to load host key "/usr/local/etc/ssh/ssh_host_ecdsa_key": invalid format
debug1: Unable to load host key: /usr/local/etc/ssh/ssh_host_ecdsa_key
debug1: Unable to load host key "/usr/local/etc/ssh/ssh_host_ed25519_key": invalid format
debug1: Unable to load host key: /usr/local/etc/ssh/ssh_host_ed25519_key
sshd: no hostkeys available -- exiting.

 

[Fredda]

How do the keys look like? Take a look at them, maybe it is obvious.

 

[ApolloDS]

I don't know why but the ssh_host_rsa_key.pub looks strange:

Code:

1024 65537 149509460879029281115828594705655509563312477965276962629579961175666760812110929116035710591856124953101528070140439848818896348530238437097909193830924521531806466626055756429225424130926760631388547792825528642590696195652256601029210078802348418477012615862505598784321534229174100324686790757061788762883 root@filer

Should it not start with "ssh-rsa..."?

 

[ApolloDS]

Maybe this is the problem...
Somehow the upgrade got the keys not created/recreated and they don't match the 2048 4096 schema.
See here: https://www.ixsystems.com/community/threads/ssh-host-key-generation.62728/

So what is the "magic* how the keys are generated and synced to the database?

 

[Fredda]

There is something wrong with the keygeneration, it sould look like this: ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAABgQ...
I think you should file a bug at the FreeNAS bugtracker https://jira.ixsystems.com/