FreeNAS 11.2-BETA2 crashing nightly

[sulliwan]

Every night, my FreeNAS installation freezes. Responds to ping, no response from ssh or webui. If I have an active SSH session, the shell still works, however certain commands (such as ls -lha in /var/log) also cause that to freeze. Have to hard reset the system to get it back operational.

The problem, as best I could guess with my troubleshooting ability:
When the periodic daily tasks run, collectd tries to run smartctl -a -n standby /dev/daX for all drives to gather data for the report. However, since my disks are behind a 3ware raid controller, smartctl can not access then by probing the disk devices directly. Probably a bug in smartctl leads this to just hang (STAT DL+), which causes the entire system to hang.

How can I either: a) disable the daily report collection or b) direct collectd to use the correct smartctl commands to get the disk summary?

Another interesting symptom with this system was that smartd service was not able to run, as smartd.conf was empty. After I edited it manually, smartd service is now able to start. When looking into why it might have been empty, I found references to /etc/ix.rc.d/ix-smartd - which was not present on my system. (However, after reboot both /etc/local/smartd.conf and /usr/local/etc/smartd.conf are both empty)

Sorry if this is the wrong forum, I hope some mod can move to the correct one if so.

 

[Alecmascot]

I suggest you raise a bug report and revert to the current stable firmware : 11.1-U6

 

[sulliwan]

I suggest you raise a bug report and revert to the current stable firmware : 11.1-U6

I forgot to add, the same problem was present with 11.1-U6, I switched to the beta version to see if this would fix it.

 

[KrisBee]

However, since my disks are behind a 3ware raid controller, smartctl can not access then by probing the disk devices directly. Probably a bug in smartctl leads this to just hang (STAT DL+), which causes the entire system to hang .....

Are you sure your RAID card is not the problem? Does it work in IT mode?

 

[sulliwan]

Are you sure your RAID card is not the problem? Does it work in IT mode?

If using smartctl -a -d 3ware,X /dev/tws0 , I get smart results from the drives. The disks are presented as single disks to FreeNAS, JBOD mode is not enabled. I am not 100% sure that my troubleshooting identified the root cause, however after a crash, it was the smartctl commands that were in uninterruptable sleep state indicating some kind of weirdness going on. I just tried the same smartctl that hangs during the night after restarting the system and it gracefully exits and tells me to use 3ware syntax, so there must be something else happening during the daily tasks that makes the command hang.

The card is 9750-8i, I'm not sure if it supports IT mode.

 

[KrisBee]

According to this ref: the 9750-8i supposedly can work in IT mode which is always the recommendation for FreeNAS/ZFS, i.e as HBA .

https://www.servethehome.com/lsi-sas-2108-raid-controller-information-listing/


But this old thread casts doubt on that: https://forums.freenas.org/index.ph...-disks-attach-to-raid-card-lsisas-2108.13078/ and there's the https://forums.freenas.org/index.php?threads/confused-about-that-lsi-card-join-the-crowd.11901/ thread to consult.