After Upgrading to 11.1-U5 from 9.10 communication on port 8400 stopped

[JohnFLi]

I run commvault to backup items on my freenas box.
Been working mostly fine for a few years. By mostly, i mean its fine except I have a couple of steps I have to do after a reboot to get teh commvault service to run.
Other than that, it's been working fine.

Until TODAY
This morning, I upgraded to 11.1-U5 from 9.10-U6. Upgrade went fine and dandy, except I can no longer get the commvault service to talk with the commvault server.
The services are running just fine on freenas:

Code:

root@G1PPFreeNas01:/etc # commvault status
--------------------------------- Instance001 ---------------------------------
[ General ]
 Version = 11(BUILD80) SP11 (1114128)
 CommServe = G1VPCMS01.firemountain.local
 Home Directory = /opt/commvault/Base
 Log Directory = /opt/log/commvault/Log_Files
 Core Directory = /opt/commvault
 Temp Directory = /opt/commvault/Base/Temp
 Platform Type = 4
 Cvd Port Number = 8400
[ Package ]
 1002/CVGxBase 11(BUILD80) = File System Core
 1101/CVGxIDA  = File System
[ Physical Machine/Cluster Groups ]
 Display Name = 10.100.1.52
  - Client Hostname = 10.100.1.52
  - Job Results Directory = /opt/commvault2/iDataAgent/jobResults
root@G1PPFreeNas01:/etc #

When trying to get teh server to talk to freenas, it fails saying communication is down.
The commvault client runs on ports 8400, 8401 and 8403

I have run sockstat -4 -l and as you can see below, it does appear to be listening.

Code:

root@G1PPFreeNas01:/etc # sockstat -4 -l
USER	 COMMAND	PID   FD PROTO  LOCAL ADDRESS		 FOREIGN ADDRESS
root	 ClMgrS	 68271 6  tcp4   *:44737			   *:*
root	 sshd	   61838 4  tcp4   10.100.1.181:22	   *:*
root	 syslogd	60147 6  udp4   10.100.1.181:514	  *:*
nobody   mdnsd	  53623 4  udp4   *:58954			   *:*
nobody   mdnsd	  53623 6  udp4   *:5353				*:*
nobody   mdnsd	  53623 7  udp4   *:5353				*:*
nobody   mdnsd	  53623 8  udp4   *:5353				*:*
root	 smbd	   53570 34 tcp4   *:445				 *:*
root	 smbd	   53570 35 tcp4   *:139				 *:*
root	 nmbd	   53565 17 udp4   *:137				 *:*
root	 nmbd	   53565 18 udp4   *:138				 *:*
root	 nmbd	   53565 19 udp4   10.100.132.113:137	*:*
root	 nmbd	   53565 20 udp4   10.100.132.255:137	*:*
root	 nmbd	   53565 21 udp4   10.100.132.113:138	*:*
root	 nmbd	   53565 22 udp4   10.100.132.255:138	*:*
root	 nmbd	   53565 23 udp4   10.100.1.52:137	   *:*
root	 nmbd	   53565 24 udp4   10.100.1.255:137	  *:*
root	 nmbd	   53565 25 udp4   10.100.1.52:138	   *:*
root	 nmbd	   53565 26 udp4   10.100.1.255:138	  *:*
root	 nmbd	   53565 27 udp4   192.168.168.9:137	 *:*
root	 nmbd	   53565 28 udp4   192.168.168.255:137   *:*
root	 nmbd	   53565 29 udp4   192.168.168.9:138	 *:*
root	 nmbd	   53565 30 udp4   192.168.168.255:138   *:*
root	 nmbd	   53565 32 udp4   10.100.1.181:137	  *:*
root	 nmbd	   53565 33 udp4   10.100.1.255:137	  *:*
root	 nmbd	   53565 34 udp4   10.100.1.181:138	  *:*
root	 nmbd	   53565 35 udp4   10.100.1.255:138	  *:*
root	 syslog-ng  53423 18 udp4   *:1031				*:*
root	 hptsvr-htt 8767  3  tcp4   *:7402				*:*
root	 uwsgi	  7597  3  tcp4   127.0.0.1:9042		*:*
root	 consul	 7433  4  tcp4   127.0.0.1:8300		*:*
root	 consul	 7433  8  tcp4   127.0.0.1:8302		*:*
root	 consul	 7433  9  udp4   127.0.0.1:8302		*:*
root	 consul	 7433  11 tcp4   127.0.0.1:8301		*:*
root	 consul	 7433  12 udp4   127.0.0.1:8301		*:*
root	 consul	 7433  13 tcp4   127.0.0.1:8600		*:*
root	 consul	 7433  14 udp4   127.0.0.1:8600		*:*
root	 consul	 7433  15 tcp4   127.0.0.1:8500		*:*
root	 consul-ale 7426  7  tcp4   127.0.0.1:8542		*:*
root	 python3.6  7410  7  udp4   *:32272			   *:*
root	 uwsgi	  7407  3  tcp4   127.0.0.1:9042		*:*
www	  nginx	  7341  6  tcp4   *:80				  *:*
root	 nginx	  7340  6  tcp4   *:80				  *:*
root	 rsync	  7144  5  tcp4   *:873				 *:*
root	 snmpd	  5517  7  udp4   *:161				 *:*
root	 snmpd	  5517  11 tcp4   *:199				 *:*
root	 sshd	   5493  5  tcp4   *:22				  *:*
nobody   proftpd	3283  0  tcp46  *:21				  *:*
root	 ntpd	   3138  21 udp4   *:123				 *:*
root	 ntpd	   3138  24 udp4   127.0.0.1:123		 *:*
root	 ntpd	   3138  25 udp4   192.168.168.9:123	 *:*
root	 ntpd	   3138  26 udp4   10.100.1.52:123	   *:*
root	 ntpd	   3138  28 udp4   10.100.132.113:123	*:*
root	 ntpd	   3138  29 udp4   10.100.1.181:123	  *:*
root	 rpc.statd  3065  6  udp4   127.0.0.1:638		 *:*
root	 rpc.statd  3065  7  udp4   10.100.1.52:638	   *:*
root	 rpc.statd  3065  8  tcp4   127.0.0.1:638		 *:*
root	 rpc.statd  3065  9  tcp4   10.100.1.52:638	   *:*
root	 nfsd	   3061  5  tcp4   10.100.1.52:2049	  *:*
root	 mountd	 3055  7  udp4   127.0.0.1:738		 *:*
root	 mountd	 3055  8  udp4   10.100.1.52:738	   *:*
root	 mountd	 3055  9  tcp4   127.0.0.1:738		 *:*
root	 mountd	 3055  10 tcp4   10.100.1.52:738	   *:*
root	 nfsuserd   3043  3  udp4   *:710				 *:*
root	 nfsuserd   3042  3  udp4   *:710				 *:*
root	 nfsuserd   3041  3  udp4   *:710				 *:*
root	 nfsuserd   3039  3  udp4   *:710				 *:*
root	 nfsuserd   3038  3  udp4   *:710				 *:*
root	 ctld	   2809  6  tcp4   192.168.168.9:3260	*:*
root	 rpcbind	2472  9  udp4   *:111				 *:*
root	 rpcbind	2472  10 udp4   *:808				 *:*
root	 rpcbind	2472  11 tcp4   *:111				 *:*
root	 python3.6  272   17 tcp4   *:6000				*:*
?		?		  ?	 ?  udp4   10.100.1.52:2049	  *:*
?		?		  ?	 ?  udp4   *:*				   *:*
?		?		  ?	 ?  udp4   *:765				 *:*
?		?		  ?	 ?  tcp4   *:601				 *:*
root@G1PPFreeNas01:/etc #

Any ideas?

 

[dlavigne]

Anything in /var/log/messages when it fails?

 

[JohnFLi]

Nothing I can see since the state changed to up after teh last time I rebooted.