FreeNAS stuck on reboot(logs inside)

[ceofreak]

Hi guys,

I sporadically have the issue that my FreeNAS gets stuck when rebooting over the webUI.

I have logfiles from the moment the shutdown command was sent.


Dec 14 09:36:03 ce0nas notifier: shutdown: [pid 73077]
Dec 14 09:36:03 ce0nas notifier: Shutdown NOW!
Dec 14 09:36:04 ce0nas syslog-ng[73574]: syslog-ng starting up; version='3.6.4'
Dec 14 09:36:04 ce0nas GEOM_ELI: Device ada1p1.eli destroyed.
Dec 14 09:36:04 ce0nas GEOM_ELI: Detached ada1p1.eli on last close.
Dec 14 09:36:04 ce0nas kernel: pid 1468 (syslog-ng), uid 0: exited on signal 6 (core dumped)
Dec 14 09:36:04 ce0nas GEOM_ELI: Device ada2p1.eli destroyed.
Dec 14 09:36:04 ce0nas GEOM_ELI: Detached ada2p1.eli on last close.
Dec 14 09:36:04 ce0nas GEOM_ELI: Device ada3p1.eli destroyed.
Dec 14 09:36:04 ce0nas GEOM_ELI: Detached ada3p1.eli on last close.
Dec 14 09:36:04 ce0nas GEOM_ELI: Device ada4p1.eli destroyed.
Dec 14 09:36:04 ce0nas GEOM_ELI: Detached ada4p1.eli on last close.
Dec 14 09:36:04 ce0nas kernel: epair0a: link state changed to DOWN
Dec 14 09:36:04 ce0nas kernel: epair0a: link state changed to DOWN
Dec 14 09:36:04 ce0nas kernel: epair0b: link state changed to DOWN
Dec 14 09:36:04 ce0nas kernel: epair0b: link state changed to DOWN
Dec 14 09:36:04 ce0nas kernel: em0: link state changed to DOWN
Dec 14 09:36:04 ce0nas kernel: em0: link state changed to DOWN
Dec 14 09:36:04 ce0nas kernel: em0: promiscuous mode disabled
Dec 14 09:36:04 ce0nas kernel: bridge0: link state changed to DOWN
Dec 14 09:36:04 ce0nas kernel: bridge0: link state changed to DOWN
Dec 14 09:36:08 ce0nas smbd: dnssd_clientstub DNSServiceProcessResult called with DNSServiceRef with no ProcessReply function
Dec 14 09:36:08 ce0nas kernel: em0: link state changed to UP
Dec 14 09:36:08 ce0nas kernel: em0: link state changed to UP
Dec 14 09:36:08 ce0nas ntpd[1949]: ntpd exiting on signal 15 (Terminated)
Dec 14 09:36:09 ce0nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 14 09:36:10 ce0nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 14 09:36:11 ce0nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 14 09:36:12 ce0nas python: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:4 Err:-1 Errno:61 Conne
ction refused
Dec 14 09:36:13 ce0nas dhclient: New IP Address (em0): 192.168.10.22
Dec 14 09:36:13 ce0nas dhclient: New Subnet Mask (em0): 255.255.255.0
Dec 14 09:36:13 ce0nas dhclient: New Broadcast Address (em0): 192.168.10.255
Dec 14 09:36:13 ce0nas dhclient: New Routers (em0): 192.168.10.254
Dec 14 09:36:14 ce0nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Dec 14 09:36:15 ce0nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Dec 14 09:36:16 ce0nas python: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Dec 14 09:36:17 ce0nas syslog-ng[73574]: syslog-ng shutting down; version='3.6.4'
Dec 14 09:59:20 ce0nas syslog-ng[1468]: syslog-ng starting up; version='3.6.4'


I then switched it off manually by holding down the power button and started it up again at 09:59:20 where it booted up normally again.

It seem to somehow has something to do with DHCP oder DNS, can't really figure it out by myself...

Any help highly appreciated!

 

[anodos]

What version of freenas?

 

[ceofreak]

What version of FreeNAS?

FreeNAS-9.10.1-U4 (ec9a7d3)

 

[joeschmuck]

Please follow our forum rules in order for us to help you. Post all the requested information and we will be better prepared to give you an intelligent answer and get you and your system back on track. Also, this could be a hardware issue where FreeNAS hangs only during reboot and requires a power off and power on (cold boot) to work properly. If you used the Console to force a reboot, I'll bet you get the same issue.

EDIT: Also, when you reboot, what is displayed on the console when it hangs?

 

[ceofreak]

Yes same issue with the console. You can see the error above. I think it has something to do with the connection refused dns message, whatever that means...

 

[joeschmuck]

Please provide all the requested information per the forum rules. Also describe what you mean by the computer is hanging. The log file you provided has nothing of value. What I'm interested in is what your FreeNAS computer screen says and what you hear.

 

[ceofreak]

Please provide all the requested information per the forum rules. Also describe what you mean by the computer is hanging. The log file you provided has nothing of value. What I'm interested in is what your FreeNAS computer screen says and what you hear.

Alright, here you go:

• motherboard make and model

ASRock C236 WSI Mini-ITX

• CPU make and model

Intel Core i3-6100@3.70 GHz

• RAM quantity

16GB Crucial DDR4 PC4-17000 Single ECC RAM x 1

• hard drives, quantity, model numbers, and RAID configuration

WD Red 3TB NAS Hard Disk Drive 5400 RPM x 4

When it's stuck in the shutdown process, I get the following on-screen output: http://imgur.com/a/jdzEc

I hope this helps.

 

[joeschmuck]

Unfortunately the screen photo didn't help much but at least I see what you are seeing. Right now I still believe it's a BIOS hangup. Ensure you backup your config file before doing anything. It will save you some time if you need to restore it.

Question Time:
1) Did this ever work properly? If Yes, did something happen or change such as maybe a FreeNAS software update? If this never worked then we need to go down a different path/mind set.
2) Are you overclocking the system at all? If Yes, restore to factory defaults.
3) What is your Boot Device? You left that off the hardware list.
4) Is your motherboard at the latest BIOS firmware version? Crap, your motherboard only has the initial release which is version 1.00 dated 12/07/15, but check it anyway. BTW, that is a nice looking motherboard.

Things to try in the meantime:
1) Reset your BIOS to Factory Defaults. Note that you may need to go back into the BIOS and re-select the boot drive/boot order.
2) Save your configuration file. If you are using a USB Flash Drive for the boot device move it and install a different USB Flash drive. Install FreeNAS 9.3.1 from this path: http://download.freenas.org/9.3.1/STABLE/201605170422/x64/ Do not restore your configuration file. All we are shooting for here is to see if you can boot it without issue and then try to reboot it via the GUI. If this works properly then try an early version of FreeNAS 9.10 and see how that works.
3) Instead of doing step #2 above you could also try the FreeBSD 10 Live CD, boot your system from it and see if it works and then reboot and see if it works.
4) If you are booting from a USB Flash Drive, you could try a real hard drive/SSD and see if that makes any difference. I don't get the sense that this is the issue but if there is some reason it's not interfacing properly with the USB drive during a reboot, well...
5) If you are using a USB Flash Drive as the boot device, try moving it to a different USB port, preferably USB 2.0 port (black ones on the left).
6) If you are using UEFI, change the BIOS setting to Legacy and reinstall FreeNAS as Legacy (not UEFI).

While you are doing all the testing above you can disconnect the power from the hard drives if you don't want them spinning up and down a lot, just power down before making any connection changes.

Good luck and please post any thing odd you see, any more data you see on the monitor, etc...

P.S. I'm looking at your user manual right now and if I see anything worth trying, I'll let you know.

 

[joeschmuck]

In your BIOS is an event log called Smbios Event Log. View it and see if there is anything listed. Refer to your user manual as well for settings.

Also I found on the internet that this could be a watchdog(8) issue or some other watchdog issue.

 

[ceofreak]

Question Time:
1) Did this ever work properly? If Yes, did something happen or change such as maybe a FreeNAS software update? If this never worked then we need to go down a different path/mind set.
No, I encountered this from the beginning. I installed the system this summer freshly.
2) Are you overclocking the system at all? If Yes, restore to factory defaults.
No, no bios settings have been changed.
3) What is your Boot Device? You left that off the hardware list.
The boot device is an internal 128gb SSD drive.
4) Is your motherboard at the latest BIOS firmware version? Crap, your motherboard only has the initial release which is version 1.00 dated 12/07/15, but check it anyway. BTW, that is a nice looking motherboard.
I will check it. But this motherboard already failed on me a few weeks ago and I needed to send it in to get a new one. It didn't boot up anymore at all after a reboot. Just blew up the fans for a few seconds and then restarted. I got a completely new one now.

Hi there! First of all thanks for the time you spent to write and research this!

Answering your question above.

Now since yesterday, also the HDD light permanently is lit up. That didn't happen before. It just started lighting yesterday and doesn't stop anymore. I don't have power save enabled, but already changed the APM to 127 like you suggested in another thread, didn't help tho.

My complete build is also to see on my website (https://www.ceos3c.com/2016/08/25/diy-nas-freenas-server/), if that somehow helps.

 

[ceofreak]

Maybe I should also add, sometimes it reboots normally. The error does occur maybe 50-70% of the time.

I probably will not be able to test your steps until next Wednesday, will be off for work. Soon as I get back I will go through all of them.

 

[joeschmuck]

Nice photos. With respect to cleaning up the cables in the case, maybe a few right angled SATA and Power connectors and then small zip-ties.

I'd change the APM back to Disabled. I've been running it that way for a very long time without issues.

Since you are now having issues with the HDD light, I think it's time to stop and choose only one problem to troubleshoot. The HDD light could be because a SCRUB is in progress or maybe you have something else running? Look at your Reports -> Disk, see if there is any activity.

 

[ceofreak]

Nice photos. With respect to cleaning up the cables in the case, maybe a few right angled SATA and Power connectors and then small zip-ties.

I'd change the APM back to Disabled. I've been running it that way for a very long time without issues.

Since you are now having issues with the HDD light, I think it's time to stop and choose only one problem to troubleshoot. The HDD light could be because a SCRUB is in progress or maybe you have something else running? Look at your Reports -> Disk, see if there is any activity.

Sorry for the late reply, got busy with work. Well, the last crash of that sort is a while back now, actually, I haven't had any since the update before the last. Now I run the newest version, just did a few reboots and no hangings. Maybe it got fixed through an update. I will keep checking on it and report back if it comes back.

But, now I got the HDD LED permanently lighting issue. It just started from one day to another, was working normally before and just lit up when there was activity. I already did some steps you recommended in another post without success, let me summarize what I had tried:

• Unplugged the LAN cable for several minutes - no change.
• Unplugged all storage HDD's - no change.
• Double checked on the HDDLED pin on the mainboard and even swapped +- without success, it is and was plugged correctly, so - no change

What I learned:

• I recognized that the HDD LED immediately lights up as soon as I switch the machine on, even before there is any hard drive activity at all.

I mean I can just live with it, it doesn't bother me too much but curiosity got me wanting to find out what causes this behavior.

Any clue from your side?

Ceo

 

[joeschmuck]

Since I don't have your motherboard it's a bit difficult to know if the HDD light is caused by the OS or hardware. To test this out, power off your machine, unplug the boot device (128GB SSD), power on the machine, it will not boot up. If the HDD light stays on then it's the hardware. If the HDD light stays off then it's the OS and likely something happening like reading/writing. Since your FreeNAS OS is on a the SSD, this could be causing the light to come on if there is some constant activity.

But as you said, you can live with this but it would be nice to know what is going on.