Console not working. iSCSI works, other services inaccessible.

[Nicholas Laird]

Currently experiencing an issue where the FreeNAS server is running but I cannot log into it via any means. WebGUI is dead (service does not respond to requests), SSH is dead, and the local console will not load (pressing 9 for shell hangs), CTRL + ALT + F2 for a new console works, but logging in hangs after the password.

Basically the system is alive on the iSCSI interfaces for our VM farm to use, but everything else seems dead. CIFS is no longer responding, AFP appears down.

Suggestions on how to administer this system? We don't have iLO activated on it, it's a FreeNAS system purchased from iX Systems so the hardware meets the specifications for FreeNAS/TrueNAS.

If it comes down to it, I can power cycle it, but it seems very odd it is this unresponsive. We have had issues in the past where IO Waits have appeared and caused CIFS to become hung up. We can cause this issue by aborting a move between two CIFS folders on the server. (Client is moving files and it dies, leaving the server waiting.)

Can't seem to find any FreeBSD/FreeNAS articles that aren't specifically SSH related when it comes to the console hanging up.

 

[dlavigne]

That is weird... have you found a way to access it yet?

 

[Nicholas Laird]

No, by the end of the day I ended up having to hold the power button to force it off, then boot back up. Everything came up again normal and the system is running fine again. I can cause a zombie thread/IO Wait by trying to move large files over CIFS from my Mac. For example if I drag a 4GB DVD ISO between folders (on the server) from my client, then abort mid-transfer I can usually get it to hang up the CIFS thread on the server and it ends up a zombie process. Enough of these seem to eventually choke up the system.

While I was down and rebooting it, I applied the latest update to see if there is any change. I was runnign 9.2.1.2 (or something close to that) and I am now current.

I think eventually I am going to use the FreeNAS server as just iSCSI storage and run the CIFS server on one of our Virtual Machines. That way if a client screws up the service I don't have to take the NAS down to restore functionality as it only appears to be CIFS causing my woes.