FreeNAS freezing frequently (daily) after upgrade to 9.10

[ysaule]

Hi everyone,

Following the upgrade to 9.10, I started having regular freeze of the system.
More specifically the box is still answering ping request but no SSH, CIFS, NFS, AFS or HTTP access.

After hard reboot everything is fine again for a few hours.

I saw nothing relevant in the message or dmesg logs.
And I tried changing a few hardware parts (motherboard, memory).

FreeNAS version is : FreeNAS-9.10-STABLE-201606270534 (dd17351)

Hardware is :
- ASROCK C2750D4
- G.Skill Kit Extreme3 2 x 8 Go PC12800 Ripjaws X CAS9
- 9 drives

camcontrol devlist
<WDC WD20EARX-00PASB0 51.0AB51> at scbus0 target 0 lun 0 (ada0,pass0)
<WDC WD40EZRX-00SPEB0 80.00A80> at scbus1 target 0 lun 0 (ada1,pass1)
<WDC WD30EFRX-68EUZN0 82.00A82> at scbus2 target 0 lun 0 (ada2,pass2)
<ST2000DL001-9VT156 CC98> at scbus3 target 0 lun 0 (ada3,pass3)
<WDC WD40EZRX-00SPEB0 80.00A80> at scbus4 target 0 lun 0 (ada4,pass4)
<WDC WD30EZRX-00D8PB0 80.00A80> at scbus5 target 0 lun 0 (ada5,pass5)
<Marvell Console 1.01> at scbus9 target 0 lun 0 (pass6)
<WDC WD30EFRX-68EUZN0 82.00A82> at scbus11 target 0 lun 0 (ada6,pass7)
<SanDisk SDSSDHP064G X2306RL> at scbus13 target 0 lun 0 (ada7,pass8)
<ST2000DL003-9VT166 CC3C> at scbus14 target 0 lun 0 (ada8,pass9)
< USB DISK 2.0 PMAP> at scbus17 target 0 lun 0 (pass10,da0)

Any idea what is going on ?

Should I report a formal bug ?

Thanks

 

[JoshDW19]

Hey ysaule. We had another user reporting that their machine was also freezing, but I'm not sure what version of FreeNAS he's running. It might be worth filing a bug report. Are you familiar with how to do so? We also have instructions on attaching a debug here in our forum rules that may help: https://forums.freenas.org/index.php?threads/updated-forum-rules-8-3-16.45124/

Thanks for posting!

 

[ysaule]

Thanks you for the reply. I will post a bug report.

EDIT:
bug report #16823 created

 

[blackangus]

Sounds VERY similar to my issue :) https://forums.freenas.org/index.php?threads/consistent-system-freezes-every-week.45271/

It's very interesting that this was not resolved for you even after swapping the motherboard and memory. I was just about to try and swap out my stuff for something else...

Do you get a really hard freeze of the whole system? No crashes of any kind or kernel panics? That's my main problem. There's almost nothing in the logs that indicate somethings wrong, and when it does freeze it just completely locks up so I have to hard power it off and on. Looking at the logs afterwards shows nothing, it just looks like if you had a system running for a while which was then booted up with no indication or logs from when it was taken down.

FWIW I had this issue with 9.10 and still have it with 9.10.1

 

[joeschmuck]

You could try to update to version 9.10.1 Stable to see if this fixes your problem. If it doesn't then I'd update the bug report to note that you upgraded and it still didn't fix it.

How often do you notice it failing after a reboot?

Also, run MemTest86 for at least 24 hours, select all CPUs to run, this will try to ensure that you do not have any other issues like a power supply failure for instance. Also run a CPU burn in test for about 20 minutes (some like longer tests but I feel 20 minutes is enough). These tests will rule out hardware issues for the most part.

Are you using an UPS? If no then you know my answer, if yes then either try another UPS or connect directly into the wall outlet.

 

[ysaule]

I have the same behavior (hard freeze of the system, no crashes or kernel panics) despite my hardware change (changed 3 time the motherboard using the same model which included also the CPUs, and twice the memory). ans nothing in the logs....

I don't have an UPS (not a good thing but it is makes clear the crash are not UPS linked)
And I already running 9.10 stable (last update availble on the 9.10-Stable train)

It worth mentionning that this issue was not present on 9.3-stable and only arised with 9.10-stable

following my various experiement, I don't think it is hardware related more likely either a configuration issue specific to my setup or a bug.
I also did a test by runing a live image from linux and got not freeze. (but linux is quite useless for my ZFS based NAS)

I am a bit lost there ...

 

[blackangus]

I'm going to perform an experiment on my end. I'm going to install FreeNAS 9.3 and see how it goes for the next two weeks. Will report back once I have something to talk about :)

 

[joeschmuck]

I am a bit lost there ...

Me too. The good thing is you submitted a bug report and hopefully someone will be contacting you for additional information or to tell you they found the issue.

Since it didn't happen in 9.3, maybe you should roll back. But that is up to you.

EDIT: I just created a debug log (I'm running the current 9.10.1 as well) and in the file "ixdiagnose\fndebug\Verify\dump.txt" it states the below message however my system is still running strong...

+--------------------------------------------------------------------------------+
+ Verifier Results @1471126465 +
+--------------------------------------------------------------------------------+
Running Verifier....
The following inconsistencies were found in your current install:

List of Files/Directories/Symlinks not Found:

/compat/linux/proc/.donotremove

List of Permission Errors:

/compat/linux/proc Expected MODE: 0755, Got: 0555
debug finished in 24 seconds for Verifier Results

 

[Morpheus187]

I had similiar issues with a monthly crash or just unresponsive machine. The first issue was a unreliable power supply, as the asrock board uses very little power, that can be a problem for the 5V rail of the supply. The sensor showed a clear drop to under 3v sometimes which caused the system to crash.

The other issue was a faulty system ssd drive which seemded to randomly become unresponsive.

 

[blackangus]

This is sort of a cross post from my other thread but I wanted to let everyone know that downgrading to 9.3 has resolved the issue for me. I will see if this continues to happen with FreeNAS 10 when that comes out.