FreeNAS unresponsive

[scotthm]

I'm running the latest FreeNAS 9.10.2 stable release on a Dell PowerEdge T430. It's a very basic setup and all I use it for is as a Plex server and as a Veeam backup target for a couple of PCs.

I tried to log into FreeNAS yesterday but the GUI wouldn't come up. I still have access to the ZFS file system (Plex and my backups are still running OK) and I can ping the FreeNAS management IP address. So I plugged in a monitor and keyboard, but I'm sorry to say I hit the Enter key on the keyboard before I recorded any error messages, and the screen went back to the menu.

Unfortunately, when I select an item from the menu nothing happens. I've selected 13 (reboot) and 9 (shell) and nothing happens other than the numbers showing up at the bottom of the screen. I'm not sure how long the server has been in this state as it's been a week or more since I've logged onto the UI.

So at this point I cannot manage FreeNAS at all. I'm beginning to think my only option is to power off the server, but I'm not sure what kind of file system damage that might cause. I'm hoping I can get some suggestions on how I might perform a graceful shutdown, or at the very least reassurance that my data isn't in danger.

Any suggestions?

 

[Robert Trevellyan]

Hitting the power button briefly should trigger a graceful shutdown on any moderately modern platform.

 

[scotthm]

That got me a response:

May 8 21:00:25 plexmediaserver_1 syslogd: exiting on signal 15
May 8 21:00:25 HomeNAS upsd[2075]: mainloop: Interrupted system call

Now Plex doesn't work. But still no reboot.

 

[Stux]

How long did you wait for a response before assuming the web server wasn't responding?

There has been a long-standing FreeBSD bug (not sure if this is still an issue in 9.10.2/11) which causes all the resources to get paged out. Its possible the entire webserver was paged out, and it was busy paging it in, and it would've eventually responded.

 

[scotthm]

How long did you wait for a response before assuming the web server wasn't responding?

About a minute. I've had this server running for about six months and this is the first time the web page hasn't come up almost immediately.

 

[Stux]

About a minute. I've had this server running for about six months and this is the first time the web page hasn't come up almost immediately.

Assuming its back up, have a look at your memory logs and see if you had any swap active. Also, if you have swap... and a disk fails/errors then it can crash the kernel and other processes.

Which is why I've written this script:
https://forums.freenas.org/index.ph...ny-used-swap-to-prevent-kernel-crashes.46206/

 

[scotthm]

Assuming its back up, have a look at your memory logs and see if you had any swap active.

It's not "back up" because it won't shut down. I hesitate to just power it off because I don't know what that's going to do to the file system. However, it's looking like I have little choice.

 

[Stux]

It's not "back up" because it won't shut down. I hesitate to just power it off because I don't know what that's going to do to the file system. However, it's looking like I have little choice.

Can you SSH in or get to the console?

If not then you really have no alternative. You can try pushing the power button, but if that makes no difference, and it won't if the kernels hung up, then you might as well power it down. Wait 15 seconds, and power it back up.

It *should* come back fine.

 

[scotthm]

you might as well power it down. Wait 15 seconds, and power it back up.

That's what I had to end up doing, and at a glance everything looks fine. I'll have to investigate the cause of the problem after I've had some sleep.

 

[Robert Trevellyan]

I don't know what that's going to do to the file system

For reference, ZFS is designed to handle power loss gracefully, at worst losing a few seconds worth of transactions but never becoming corrupted as a direct result. The fact that power loss sometimes breaks ZFS on FreeNAS is a puzzle.

 

[rs225]

If there is still disk activity happening, it is safer to hit the reset button before removing power.

 

[joeschmuck]

Awe just pull the cord out of the wall and beat the computer with it for giving you a hard time.

You should perform a scrub on your pool now just for peace of mind and while you're at it, whip that computer again.

Serious question... Any idea when the last time the system was rebooted? I'm thinking memory leak since the console screen was non-responsive, unless you say it was rebooted 1 month ago or sooner. And how much RAM do you have? Is that a factor?

 

[scotthm]

Serious question... Any idea when the last time the system was rebooted? I'm thinking memory leak since the console screen was non-responsive, unless you say it was rebooted 1 month ago or sooner. And how much RAM do you have? Is that a factor?

The last time it was rebooted (before yesterday) was a week or two ago when I updated to FreeNAS-9.10.2-U3 (e1497f269). This system has 32 GB of RAM. I've been running this server for about six months but never had a problem with it before. Maybe the latest update doesn't like something about my hardware.

 

[joeschmuck]

Keep an eye on it and if it happens again, note everything that you can and then also grab the log file, maybe there will be something there.