SOLVED FreeNAS Unresponsive every morning

[IronRobi]

Still relatively new to FreeNAS, learning as I go. Have had my system up and running for months now with no issues. Right around Christmas I noticed one morning everything was unresponsive so I had to do a hard reboot holding the power button. Now since Dec 30th every morning I wake up and everything is unresponsive. My first step was to check for updates and update everything. It didn't fix it. A hard reboot to start the day and everything runs fine until the next morning. Something happens between midnight and 7am causing it to go unresponsive. I can still ping the box, but nothing else.

I've looked under var/log/messages and there's no message appearing between the times it goes unresponsive. I'm not sure where to begin troubleshooting. Any suggestions?

FreeNAS v. 9.10.2
Hardware:
Motherboard: Asus Z77 Pro 4
NIC: Intel NIC
Ram: 16gb
HD: 8x WD Red 4tb

I've attached a pdf of the messages file

 

[SweetAndLow]

Memory test. Then CPU burn in test. Then verify temps and power usage. You didn't tell us what power supply you have. Could also be a bad install media. It looks like you have some plugin issues but that probably not related.

Sent from my Nexus 5X using Tapatalk

 

[IronRobi]

Memory test. Then CPU burn in test. Then verify temps and power usage. You didn't tell us what power supply you have. Could also be a bad install media. It looks like you have some plugin issues but that probably not related.

Sent from my Nexus 5X using Tapatalk

Power Supply is a Rosewill RP600 600w

Would a memory stress test put the data at risk at this point?

 

[gpsguy]

Do you have any ports on your FreeNAS server exposed to the internet? Perhaps you have some uninvited guests trying to access it, late at night.

 

[IronRobi]

Do you have any ports on your FreeNAS server exposed to the internet? Perhaps you have some uninvited guests trying to access it, late at night.

Yes I do for emby, transmission, sonarr, etc... And come to think of it I did have an access attempt to my main pc before the holidays. I will attempt to disconnect it from the web overnight.

 

[IronRobi]

Here's where I'm at on this so far.
Memtest OK
CPU Diagnostic OK
Replaced both the install media and backup with 2 brand new USB sticks, fresh install w/ config copied
Tried disconnecting internet access overnight, also tried plugging it in to the onboard realtek NIC instead of the Intel card

It doesn't seem to matter when I boot the system, it will always be unresponsive come morning. I can boot it at 6am and it is completely fine all day and then dead come morning or I can boot it at 11pm and it's dead come morning.

since var/log/messages was giving me no indication of what was happening or when, I configured Sonarr's drone factory to scan every minute. This would at least show me in the Sonarr log when it's crashing. Now here's where it get's weird.... 12am ET every night the system becomes unresponsive. Could be coincidence but it's 3 nights in a row right at midnight.

I'm baffled here

 

[IronRobi]

So after a few more days of 12am crashes, I'm grasping at straws here. I changed the system timezone back by 4 hours to see if it's at all related to it being midnight or if it's just some weird coincidence that it's crashed at the exact same time every day for a week.

 

[SweetAndLow]

Anything in the ipmi logs? If there is nothing in the freenas logs it is likely a hardware problem.

Sent from my Nexus 5X using Tapatalk

 

[IronRobi]

Anything in the ipmi logs? If there is nothing in the freenas logs it is likely a hardware problem.

Sent from my Nexus 5X using Tapatalk

I don't have the IPMI option which from my understanding means my hardware doesn't support it.

Where I have a hard time thinking hardware is the fact that the system goes down at the exact same time every day. To the minute. Regardless of when it was booted. That would be one hell of a coincidence. It has to be something software related that triggers it, which may then cause some sort of hardware failure.

Are there any processes in freenas that automatically run at midnight every night by default?

 

[SweetAndLow]

Nothing by default but I have no clue what your setup is like. Did you setup smart tests, scrubs, jails, snapshots or replication. There are too many things to guess at. Can you reinstall and don't configure anything? This way we can eliminate your modifications.

Sent from my Nexus 5X using Tapatalk

 

[IronRobi]

Nothing by default but I have no clue what your setup is like. Did you setup smart tests, scrubs, jails, snapshots or replication. There are too many things to guess at. Can you reinstall and don't configure anything? This way we can eliminate your modifications.

Sent from my Nexus 5X using Tapatalk

I installed, setup plugins for my media server (emby, sonarr, couchpotato, headphones, transmission, sabnzbd) and walked away. Didn't see a need to push further into it since it was suiting my needs.

I'll try a fresh install on a different usb stick and not configure anything and let it sit overnight and see what happens.

 

[SweetAndLow]

You have lots of plugins and not much memory. What does your memory graph look like? Are you using swap?

Sent from my Nexus 5X using Tapatalk

 

[IronRobi]

You have lots of plugins and not much memory. What does your memory graph look like? Are you using swap?

Sent from my Nexus 5X using Tapatalk

More memory is definitely on the shopping list, but at this point if I'm going to have to start replacing hardware I'd likely change the motherboard to one that supports ECC and then increase the memory. The graph typically looks fine though. Not using swap at all.

I'll have some time over the weekend I'm going to try that unconfigured install and see if it lasts the night.

 

[IronRobi]

Here's a screenshot of the memory graph

 

[IronRobi]

I just found a task scheduled to run in an Emby plugin every night at 12am ET. Last successful completion was on Dec. 29th, which also happens to be the exact day Emby updated to version 3.1. Dec 30th at 12am ET, when it should have run next is when my daily crashes started.

In the emby log

Code:

2017-01-13 00:00:03.8437 Error HttpClient: Error getting response from https://api-v2launch.trakt.tv/sync/watched/movies
	*** Error Report ***
	Version: 3.1.0.0
	Command line: /usr/pbi/emby-amd64/lib/emby-server/MediaBrowser.Server.Mono.exe -ffmpeg /usr/pbi/emby-amd64/bin/ffmpeg -ffprobe /usr/pbi/emby-amd64/bin/ffprobe -programdata /var/db/emby-server
	Operating system: Unix 10.3.0.0
	64-Bit OS: True
	64-Bit Process: True
	Mono: 4.4.2 (Stable 4.4.2.11/f72fe45 Mon Aug 22 04:48:30 UTC 2016)
	Processor count: 4
	Program data path: /var/db/emby-server
	Application directory: /usr/pbi/emby-amd64/lib/emby-server
	System.AggregateException: One or more errors occurred. ---> System.Net.WebException: Error: NameResolutionFailure
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x80c79b410 + 0x0020d> in <filename unknown>:0
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x80c79af60 + 0x00093> in <filename unknown>:0
	  --- End of inner exception stack trace ---
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x80e6822b0 + 0x00038> in <filename unknown>:0
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x80e681ec0 + 0x00118> in <filename unknown>:0
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x80e681df0 + 0x000b3> in <filename unknown>:0
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x80a0b8ca0 + 0x00042> in <filename unknown>:0
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x80a0d4f30 + 0x0001f> in <filename unknown>:0
	  at Emby.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__20.MoveNext () <0x80c738620 + 0x00f54> in <filename unknown>:0
	---> (Inner Exception #0) System.Net.WebException: Error: NameResolutionFailure
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x80c79b410 + 0x0020d> in <filename unknown>:0
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x80c79af60 + 0x00093> in <filename unknown>:0 <---
	
	System.AggregateException
	  at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw () <0x80e6822b0 + 0x00038> in <filename unknown>:0
	  at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess (System.Threading.Tasks.Task task) <0x80e681ec0 + 0x00118> in <filename unknown>:0
	  at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification (System.Threading.Tasks.Task task) <0x80e681df0 + 0x000b3> in <filename unknown>:0
	  at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd (System.Threading.Tasks.Task task) <0x80a0b8ca0 + 0x00042> in <filename unknown>:0
	  at System.Runtime.CompilerServices.ConfiguredTaskAwaitable`1+ConfiguredTaskAwaiter[TResult].GetResult () <0x80a0d4f30 + 0x0001f> in <filename unknown>:0
	  at Emby.Common.Implementations.HttpClientManager.HttpClientManager+<SendAsyncInternal>d__20.MoveNext () <0x80c738620 + 0x00f54> in <filename unknown>:0
	InnerException: System.Net.WebException
	System.Net.WebException: Error: NameResolutionFailure
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x80c79b410 + 0x0020d> in <filename unknown>:0
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x80c79af60 + 0x00093> in <filename unknown>:0
	  at System.Net.HttpWebRequest.EndGetResponse (IAsyncResult asyncResult) <0x80c79b410 + 0x0020d> in <filename unknown>:0
	  at System.Threading.Tasks.TaskFactory`1[TResult].FromAsyncCoreLogic (IAsyncResult iar, System.Func`2 endFunction, System.Action`1 endAction, System.Threading.Tasks.Task`1 promise, Boolean requiresSynchronization) <0x80c79af60 + 0x00093> in <filename unknown>:0

I've removed the plugin, disabled the task and now my fingers are crossed.

 

[IronRobi]

Happy to say that since disabling this task through emby the system has been up and stable for 48hrs. Just wish I had made this connection weeks earlier!