WTF: ?Bigger issue causes plugins & mysql to stop working?

[28061]

Morning,

At an unknown point in the last few days, some weird stuff happened in my server rack.

All I know for sure is that within a 36hour period my BT home hub reset itself to default settings, my Neet airstream receiver totally stopped working and several features of FreeNas also stopped working. MY UPS logs record no unusual activity. At some point I also upgraded to 9.10.1, but the following errors occur whether or not I'm using 9.10.1, or if I rolled back to 9.10.

RE FreeNas, none of my plugins work and the MySql database running in a jail doesn't work either. More specifically, I have SabNabd, SickRage & CouchPotato installed into a jail called sabnzbd_1. The jail starts and stops fine but when I try to start a plugin, I get the following:

Code:

sabnzbd_1 su: in openpam_check_desc_owner_perms(): /etc/pam.d/su: insecure ownership or permissions
sabnzbd_1 su: system error

Additionally, I also have MySql installed into a jail called KodiLibrary. Again the jail appears to start and stop ok, and responds to ping. But all of my Kodi clients are reporting failures to connect to Mysql. When I use the shell, open the jail and run service mysql-server start, I get

Code:

su: pam_start: system error
/usr/local/etc/rc.d/mysql-server: WARNING: failed to start mysql

What the hell happened!! I don't know where to start... Everything was working fine a few days ago. Thankfully, my CIFS shares continue to work fine. I've definately not changed any permissions, and errors persist regardless of the version I run... Any help would be hugely appreciated!

 

[styno]

Am I correct that you
- are running 1 jail, and installed all of the plugins manually in that jail, not via the FreeNas plugin system?
- are running a 2nd jail, also manually installed kodi & mysql
- did the upgrade to 9.10.1 and afterwards your setup was no longer starting properly?

Can you post the output of "ls -al /etc/pam.d/" from within both jails?

 

[28061]

Thank you very much for your prompt reply.

Am I correct that you
- are running 1 jail, and installed all of the plugins manually in that jail, not via the FreeNas plugin system?

Yes, I created the jail, downloaded the plugin files from here, and then uploaded the files into the jail using the FreeNas GUI.

- are running a 2nd jail, also manually installed kodi & mysql

Not installed Kodi - just MySql. My Kodi clients are 1xPC & a couple of Raspberry Pi's.

- did the upgrade to 9.10.1 and afterwards your setup was no longer starting properly?

To be honest I'm not sure whether I upgraded before or after the problems started - it's possible they only started afterwards, but it's also possible they started before the upgrade and I only noticed afterwards.

Output of "ls -al /etc/pam.d/" for sabnzbd_1 jail:

Code:

root@sabnzbd_1:/ # ls -al /etc/pam.d/                                          
total 178                                                                      
drwxrwxrwx   2 media  media    19 Jul 10  2014 .                              
drwxrwxrwx  21 media  media   104 Aug  8 01:40 ..                              
-rwxrwxrwx   1 media  media  2907 Jul 10  2014 README                          
-rwxrwxrwx   1 media  media   318 Jul 10  2014 atrun                          
-rwxrwxrwx   1 media  media   195 Jul 10  2014 cron                            
-rwxrwxrwx   2 media  media   527 Jul 10  2014 ftp                            
-rwxrwxrwx   2 media  media   527 Jul 10  2014 ftpd                            
-rwxrwxrwx   1 media  media   361 Jul 10  2014 imap                            
-rwxrwxrwx   1 media  media   473 Jul 10  2014 kde                            
-rwxrwxrwx   1 media  media   369 Jul 10  2014 login                          
-rwxrwxrwx   1 media  media   667 Jul 10  2014 other                          
-rwxrwxrwx   1 media  media   314 Jul 10  2014 passwd                          
-rwxrwxrwx   1 media  media   361 Jul 10  2014 pop3                            
-rwxrwxrwx   1 media  media   324 Jul 10  2014 rsh                            
-rwxrwxrwx   1 media  media   744 Jul 10  2014 sshd                            
-rwxrwxrwx   1 media  media   380 Jul 10  2014 su                              
-rwxrwxrwx   1 media  media   710 Jul 10  2014 system                          
-rwxrwxrwx   1 media  media   760 Jul 10  2014 telnetd                        
-rwxrwxrwx   1 media  media   525 Jul 10  2014 xdm 

Output of "ls -al /etc/pam.d/" for KodiLibrary jail:

Code:

root@KodiLibrary:/ # ls -al /etc/pam.d/                                        
total 178                                                                      
drwxrwxrwx   2 816  816    19 Jul 11  2014 .                                  
drwxrwxrwx  21 816  816   104 Aug  8 09:23 ..                                  
-rwxrwxrwx   1 816  816  2907 Jul 11  2014 README                              
-rwxrwxrwx   1 816  816   318 Jul 11  2014 atrun                              
-rwxrwxrwx   1 816  816   195 Jul 11  2014 cron                                
-rwxrwxrwx   2 816  816   527 Jul 11  2014 ftp                                
-rwxrwxrwx   2 816  816   527 Jul 11  2014 ftpd                                
-rwxrwxrwx   1 816  816   361 Jul 11  2014 imap                                
-rwxrwxrwx   1 816  816   473 Jul 11  2014 kde                                
-rwxrwxrwx   1 816  816   369 Jul 11  2014 login                              
-rwxrwxrwx   1 816  816   667 Jul 11  2014 other                              
-rwxrwxrwx   1 816  816   314 Jul 11  2014 passwd                              
-rwxrwxrwx   1 816  816   361 Jul 11  2014 pop3                                
-rwxrwxrwx   1 816  816   324 Jul 11  2014 rsh                                
-rwxrwxrwx   1 816  816   744 Jul 11  2014 sshd                                
-rwxrwxrwx   1 816  816   380 Jul 11  2014 su                                  
-rwxrwxrwx   1 816  816   710 Jul 11  2014 system                              
-rwxrwxrwx   1 816  816   760 Jul 11  2014 telnetd                            
-rwxrwxrwx   1 816  816   525 Jul 11  2014 xdm       

 

[styno]

Somehow your permissions in /etc/pam.d are ******-up and I am afraid that is beyond repair. Did you follow a guide that has a 'chmod 777' somewhere?

A healthy output looks like this:

Code:

[root@plexmediaserver_1 /]# ls -al /etc/pam.d/
total 178
drwxr-xr-x   2 root  wheel    19 Jul 11  2014 .
drwxr-xr-x  21 root  wheel   104 Aug  7 14:06 ..
-r--r--r--   1 root  wheel  2907 Jul 11  2014 README
-rw-r--r--   1 root  wheel   318 Jul 11  2014 atrun
-rw-r--r--   1 root  wheel   195 Jul 11  2014 cron
-rw-r--r--   2 root  wheel   527 Jul 11  2014 ftp
-rw-r--r--   2 root  wheel   527 Jul 11  2014 ftpd
-rw-r--r--   1 root  wheel   361 Jul 11  2014 imap
-rw-r--r--   1 root  wheel   473 Jul 11  2014 kde
-rw-r--r--   1 root  wheel   369 Jul 11  2014 login
-rw-r--r--   1 root  wheel   667 Jul 11  2014 other
-rw-r--r--   1 root  wheel   314 Jul 11  2014 passwd
-rw-r--r--   1 root  wheel   361 Jul 11  2014 pop3
-rw-r--r--   1 root  wheel   324 Jul 11  2014 rsh
-rw-r--r--   1 root  wheel   744 Jul 11  2014 sshd
-rw-r--r--   1 root  wheel   380 Jul 11  2014 su
-rw-r--r--   1 root  wheel   710 Jul 11  2014 system
-rw-r--r--   1 root  wheel   760 Jul 11  2014 telnetd
-rw-r--r--   1 root  wheel   525 Jul 11  2014 xdm

 

[28061]

Did you follow a guide that has a 'chmod 777' somewhere?

No, I've not done anything like this at all. System has been working fine for awhile and I've not had need to change anything.

I've just spoken with the wife, and we watched some content on Kodi on the morning of Thursday 4th August (so MySql must have been working well). According to the FreeNas logs, I updated to 9.10.1 at 1330 on the same day. I think it's fair to say that the problems therefore came after the update. The only thing I can think of that I've done differently, is I pressed update whilst ddns to access FreeNas - that's it!

So what now? I have to delete the jails and start again, reconfigure everything?

 

[Bidule0hm]

Is your FreeNAS exposed on the internet?

 

[styno]

So what now? I have to delete the jails and start again, reconfigure everything?

As long as we have no clue what caused this I am not sure how to mitigate....

 

[28061]

Is your FreeNAS exposed on the internet?

Yes... Using DDNS. Was set up about 10 minutes prior to me pressing update on Thursday for testing mainly. Disabled now.

As long as we have no clue what caused this I am not sure how to mitigate....

Ok. Thank you. I didn't set any replication tasks for the jail pool either... Lesson learned. It's going to take me a few hours to re-configure it all!

 

[styno]

Ok. Thank you. I didn't set any replication tasks for the jail pool either... Lesson learned. It's going to take me a few hours to re-configure it all!

Do you have snapshots configured? If so you should be able to mount them and review the permissions at some points in the past and have an idea when it exactly happened.

 

[28061]

Do you have snapshots configured?

I do now... But it's a bit late! I'd only set snapshots on my data shares and not for the jail pool... Another lesson learned.

 

[Jailer]

As you've discovered snapshots aren't just for critical data, they're nice to have for data you don't necessarily want to have to reconfigure or restore. They are great for jails for trying something new or updating software. If something goes wrong just revert.

I've only got scheduled snapshots on one jail the rest, such as Plex, I do manually before a software update.

 

[28061]

Sensible idea that is. Thank you - I'll do the same.
I've just began the process of rebuilding it all. Just glad it's not my data. And still without a clue what happened!

 

[styno]

Actually, I do have hourly snapshots just everywhere. Better safe then sorry.

 

[Bidule0hm]

And still without a clue what happened!

You may have been hacked. FreeNAS is not meant to be exposed to the net; with the exception of the jails behind a firewall.

 

[28061]

You may have been hacked.

Ok, I'm now fairly confident that's not the case, and it's instead caused by the new update...

So, I deleted the jails and started again. New jails->Uploaded plugins->Configured storage->Fully configured plugins (took hours)->Re-indexed all media (also hours)->Confirmed all fully working and happy.

Changed from 9.10 to 9.10.1 & rebooted. Everything's a mess. Plugins are running but are full of permissions errors & don't show in the GUI. The whole thing is fubar'd again.

Reverted back to 9.10 & rebooted -> Working config restored... Think I'll stay away from the update for a while!

 

[Bidule0hm]

Given the result of your last test I'd change my mind to say it's the update too.

That's why I stay on 9.3.0, rock stable :D

 

[styno]

Whoa, that is scary! I guess I am lucky with my updates now!
Can you please create a bug report so we are sure this has the attention of support&development?

 

[28061]

Can you please create a bug report

Reported - Bug # 16775.