SSH works only temporialrly

[Zythyr]

Problem:
My SSH works only temporarily upon starting the freenas server. It stops working few minutes after first access using passwordless SSH login. Upon a fresh restart of the freenas server, I am perfectly able to access another freenas server using SSH login using Shell. After I exit the Shell and retry (few minutes after) to do SSH login, I get an error indicating the remote host identification has changed (see below). After deleting the known_hosts file and reattempting to connect using SSH, the Shell gives timed out error (see below). After getting this error, if I restart both the freenas servers, the issue goes away only for the first attempt. This issue happens on both of my freenas servers.

Below is error. Please note I have change the fingerprint to XX for security reasons.

[root@SVR2 ~]# su user1
[user1@SVR2] /root% ssh user1@192.168.1.113
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
@ WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! @
@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@@
IT IS POSSIBLE THAT SOMEONE IS DOING SOMETHING NASTY!
Someone could be eavesdropping on you right now (man-in-the-middle attack)!
It is also possible that a host key has just been changed.
The fingerprint for the RSA key sent by the remote host is
XX:XX:80:36:1e:3c:91:6d:c3:08:2c:8d:XX:XX:XX:XX.
Please contact your system administrator.
Add correct host key in /mnt/VOL2/home/user1/.ssh/known_hosts to get rid of
this message.
Offending RSA key in /mnt/VOL2/home/user1/.ssh/known_hosts:1
RSA host key for 192.168.1.113 has changed and you have requested strict checkin
g.
Host key verification failed.
[user1@SVR2] /root%
[user1@SVR2] /root% cd ~/.ssh
[user1@SVR2] ~/.ssh% rm known_hosts
[user1@SVR2] ~/.ssh% ssh user1@192.168.1.113
The authenticity of host '192.168.1.113 (192.168.1.113)' can't be established.
ECDSA key fingerprint is XX:XX:0e:3c:75:83:c5:b7:d8:ba:7c:f4:b1:XX:XX:XX.
Are you sure you want to continue connecting (yes/no)? yes
Warning: Permanently added '192.168.1.113' (ECDSA) to the list of known hosts.
Connection to 192.168.1.113 timed out while waiting to read
[user1@SVR2] ~/.ssh%

What I am trying to do?
I want to use SSH + rsync to transfer files from freenas server 1 onto freenas server 2.

Using the rsync over ssh documentation guide, I managed to get passwordless SSH to work and also transfer the host key. However, the issue because SSH only works temporarily, I am unable to use rysnc over ssh.

When I do a fresh restart of both freenas servers, I manually run the rsync task. I can see Network/Disk activity indicating rysnc is transferring files, but after few minutes it suddenly both rsyn and ssh services stops. When I go to Shell to manually log into the other freenas server using SSH, I am unable to and I get the error I indicated above.

My config
SVR1 Config:
Hotename: SVR1.local, Gigabit Ethernet 192.168.1.113
Build: FreeNAS-9.2.1.9-RELEASE-x86 (2bbba09)
Platform: Intel(R) Pentium(R) 4 CPU 3.40GHz
Memory: 6052MB
Hard drive: Seagate 3TB

SVR2 Config
Hotename: SVR2.local, Gigabit Ethernet 192.168.1.112
Build: FreeNAS-9.3-STABLE-201501212031
Platform: Intel(R) Core(TM)2 CPU 6600 @ 2.40GHz
Memory: 6052MB
Hard drive: Seagate 3TB

 

[dlavigne]

Something funky is going on if you get host key errors after a reboot as the host keys only change when the OS is reinstalled.

If you're getting timeout errors, check that there are entries for each system in the other system's hosts file. Also, is this a LAN or a WAN link?

 

[cyberjock]

You do realize both of those boxes have only 3/4 of the minimum RAM called for in the FreeNAS manual, right? So I'd start by meeting the minimum requirements and see how things go.

Also keep in mind that just based on the CPUs I have no doubt you aren't using recommended hardware, so unexpected random behavior is not unheard of.

 

[pirateghost]

Running 6gb ram on x86 build huh?

 

[SweetAndLow]

Kind of sounds like you have ip conflicts on the network. Did you configure static ips outside the dhcp range.

 

[Zythyr]

Kind of sounds like you have ip conflicts on the network. Did you configure static ips outside the dhcp range.

@SweetAndLow Thanks a lot for pointing this out! All I had to do was reset the router and it fixed all my issues! I don't know why I didn't think of this before. Really appreciate it.

On a side note, I have an off-topic question about rysnc via ssh.

I was backing up my server 1 onto server 2 using rsync over ssh. Looking at the disk usage in the Reporting tab, I noticed that server 2 spent almost 2-3 hrs reading AND writing data. When I looked at the disk space used under "View Volumes", the size wasn't changing. The disk space used stayed at around 523GB. After few hours it seems that the actual writing process started (only writing, no reading) and the disk space used was increasing until entire 1.5TB transfer from server 1 was complete.. Note: Part of the data (~523GB) already existed on server 2 due to previous attempts for using rsync over ssh.

Why was2-3hrs spent on Reading AND Writing data during which actual space used didn't change? Was rsync re-transferring all the data that already existed? Now that rysnc has compelted all the transfer, if I run the rsync task again, will it re=transfer all the data instead of only transfer changed files?