Log reporting "error: kex_exchange_identification: Connection closed by remote host"

[Newfoundland.Republic]

Hi folks - Maybe I never noticed this before, but every hour (on the half-hour, my timezone is GMT -2.5) I see the following in my logs:
Feb 16 09:30:13 freenas sshd[76681]: error: kex_exchange_identification: Connection closed by remote host

I do have an rsync backup task from my FreeNAS box to my old Synology NAS but those backup tasks are working fine...

Does anyone have any idea what this is?

Thanks
Mike

 

[jeidy]

I have the same problem:
"error: kex_exchange_identification: Connection closed by remote host "

 

[Newfoundland.Republic]

Circling back - This seems like the FreeNAS server is trying to reach out to a remote server. Could this be FreeNAS trying to check for updates? I don't think so as one would think that the FreeNAS server would be reaching out to the repositories using https...

 

[Rand0mUser]

Hello,

I noticed same error message today:

I checked using tcpdump as explained here: Limiting closed port rst response...
Using this command:

Code:

tcpdump -n -v 'tcp[tcpflags] & (tcp-rst) != 0'

And I confirmed what I already guessed, this is caused by Endpoint security port scan of my UDM-Pro which is running at regular intervals:

Since the scan feature will try to connect to many ports, it will create those errors.

So I would say that you may have some device with a port scanner running on your network. If this is a security feature like it is in this case, everything is fine. If you're not aware of any legitimate port scanner on your network, you should check which IP is trying to connect to your freenas because you may have a hacked device somewhere.

 

[Newfoundland.Republic]

Hmmmm.... I shutdown my VM with TrueCommand and the messages stopped. I'll turn it back on and see if that is the culprit...

Edit: Nope, not TrueCommand. The messages stopped but I want to find out why...

 

[Newfoundland.Republic]

Okay. I think I figured this out.

• Find out what process was initiating the ssh connection: lsof -n | grep ssh | grep DEL
• Output from lsof gives the following: lsof: no pwd entry for UID 975
• UID 975 does not exist
• Find out what process is running as UID 975: ps -auxww | grep 975
• Output from ps is:

975 4678 0.0 0.0 6384 2028 - IsJ Sat10 0:00.00 daemon: /usr/local/openjdk8/bin/java[4679] (daemon) 975 4679 0.0 2.3 3019012 761564 - IJ Sat10 6:33.59 /usr/local/openjdk8/bin/java -Djava.awt.headless=true -Xmx1024M com.ubnt.ace.Launcher start 975 4759 0.0 0.6 1001736 207980 - IJ Sat10 5:16.28 bin/mongod --dbpath /usr/local/share/java/unifi/data/db --port 27117 --unixSocketPrefix /usr/local/share/java/unifi/run --logappend --logpath /usr/local/share/java/unifi/logs/mongod.log --bind_ip 127.0.0.1

That is my UniFi Controler plugin. Seems like the controller is trying to reach out to, something...

I'll update as I findout more.

 

[MikeyG]

@Newfoundland.Republic I seem to be experiencing this as a result of the netdata plugin. Were you ever able to find a root cause or solution?

 

[Newfoundland.Republic]

Seems like it is from the UniFi controller plugin. I'm waiting for the next update to the UniFi software to see if it continues. If it does, I'll be trying a Ubuntu VM (in Bhyve, not VMware) to see if it continues. I think it may be either (a) the way the UniFi plugin is designed or (b) how the FreeNAS plugin architected.

I'll be looking into it once I upgrade pfSense to 2.5 :)

 

[MikeyG]

I take it back - I removed the plugin completely and still get this error.

 

[MikeyG]

Figured it out. This is caused by an hourly active discovery that is done from my pfsense box running ntopng. I recently created a virtual IP for another subnet, after which this error started appearing. I'm not sure if this has to do with multiple discovery requests coming from different subnets, but at least I have somewhat of a cause now.

 

[Newfoundland.Republic]

I stopped the UniFi Controller plugin and replaced it with a Ubuntu VM with the UniFi Controller installed on my ESXi host. After doing that, the key exchange error disappeared.

This still does not really answer the question: Is it the UniFi Controller service the issue or is it the way the plugin itself is constructed? I have three Emby plugins (one for each of three subnets) and they do not produce the same log results...

 

[marcevan]

I just turned off SSH on my TrueNAS to see if that stops it, as I use the GUI while local and never SSH into this as not an SA and despise those who keep saying "TrueNAS is not right for you then". If it were headless, I would agree, but so much is done by GUI so they need to stop being that condescending.