continual smbd exit signal 4 , after upgrading from 9.2.1.9 to 9.3

[skimon]

Hi There,

I had freenas running smoothly on 9.2.1.9 and I upgraded last week to 9.3 stable. Since upgrading I get an email everyday showing that smbd is exiting (and presumably restarting continually). See log below. Does anyone know if this is a known issue or if there are any steps i can take to remedy this. Rebooting didnt help, nor did taking an upgrade on stable line (201503070129).

freenas.local kernel log messages:
> pid 49568 (smbd), uid 1005: exited on signal 4
> pid 49569 (smbd), uid 1005: exited on signal 4
> pid 49570 (smbd), uid 1005: exited on signal 4
> pid 49571 (smbd), uid 1005: exited on signal 4
> pid 49572 (smbd), uid 1005: exited on signal 4
> pid 49573 (smbd), uid 1005: exited on signal 4
> pid 49574 (smbd), uid 1005: exited on signal 4
> pid 49575 (smbd), uid 1005: exited on signal 4
> pid 49576 (smbd), uid 1005: exited on signal 4
> pid 49577 (smbd), uid 1005: exited on signal 4
> pid 49578 (smbd), uid 1005: exited on signal 4
> pid 49579 (smbd), uid 1005: exited on signal 4
> pid 49580 (smbd), uid 1005: exited on signal 4
> pid 49586 (smbd), uid 1005: exited on signal 4
> pid 49587 (smbd), uid 1005: exited on signal 4
> pid 49588 (smbd), uid 1005: exited on signal 4
> pid 49589 (smbd), uid 1005: exited on signal 4
> pid 49590 (smbd), uid 1005: exited on signal 4
> pid 49591 (smbd), uid 1005: exited on signal 4
> pid 49592 (smbd), uid 1005: exited on signal 4
> pid 49593 (smbd), uid 1005: exited on signal 4
> pid 49594 (smbd), uid 1005: exited on signal 4
> pid 49595 (smbd), uid 1005: exited on signal 4
> pid 49596 (smbd), uid 1005: exited on signal 4
> pid 49597 (smbd), uid 1005: exited on signal 4
> pid 49598 (smbd), uid 1005: exited on signal 4
> pid 49599 (smbd), uid 1005: exited on signal 4
> pid 49600 (smbd), uid 1005: exited on signal 4
> pid 49601 (smbd), uid 1005: exited on signal 4
> pid 49602 (smbd), uid 1005: exited on signal 4
> pid 49603 (smbd), uid 1005: exited on signal 4
> pid 49604 (smbd), uid 1005: exited on signal 4
> pid 49605 (smbd), uid 1005: exited on signal 4
> pid 49606 (smbd), uid 1005: exited on signal 4
> pid 49607 (smbd), uid 1005: exited on signal 4
> pid 49608 (smbd), uid 1005: exited on signal 4
> pid 49609 (smbd), uid 1005: exited on signal 4
> pid 49610 (smbd), uid 1005: exited on signal 4
> pid 49611 (smbd), uid 1005: exited on signal 4
> pid 49612 (smbd), uid 1005: exited on signal 4
> pid 49613 (smbd), uid 1005: exited on signal 4
> pid 49614 (smbd), uid 1005: exited on signal 4
> pid 49615 (smbd), uid 1005: exited on signal 4
> pid 49616 (smbd), uid 1005: exited on signal 4
> pid 49617 (smbd), uid 1005: exited on signal 4
> pid 49618 (smbd), uid 1005: exited on signal 4
> pid 49619 (smbd), uid 1005: exited on signal 4
> pid 49620 (smbd), uid 1005: exited on signal 4
> pid 49625 (smbd), uid 1005: exited on signal 4
> pid 49642 (smbd), uid 1005: exited on signal 4
> pid 49659 (smbd), uid 1005: exited on signal 4
> pid 49671 (smbd), uid 1005: exited on signal 4
> pid 49672 (smbd), uid 1005: exited on signal 4
> pid 49673 (smbd), uid 1005: exited on signal 4
> pid 49674 (smbd), uid 1005: exited on signal 4
> pid 49683 (smbd), uid 1005: exited on signal 4
> pid 49687 (smbd), uid 1005: exited on signal 4
> pid 49695 (smbd), uid 1005: exited on signal 4
> pid 49696 (smbd), uid 1005: exited on signal 4
> pid 49697 (smbd), uid 1005: exited on signal 4
> pid 49698 (smbd), uid 1005: exited on signal 4
> pid 49699 (smbd), uid 1005: exited on signal 4
> pid 49706 (smbd), uid 1005: exited on signal 4
> pid 49707 (smbd), uid 1005: exited on signal 4
> pid 49708 (smbd), uid 1005: exited on signal 4
> pid 49709 (smbd), uid 1005: exited on signal 4
> pid 49710 (smbd), uid 1005: exited on signal 4
> pid 49711 (smbd), uid 1005: exited on signal 4
> pid 49712 (smbd), uid 1005: exited on signal 4
> pid 49713 (smbd), uid 1005: exited on signal 4
> pid 49714 (smbd), uid 1005: exited on signal 4
> pid 49715 (smbd), uid 1005: exited on signal 4
> pid 49716 (smbd), uid 1005: exited on signal 4
> pid 49717 (smbd), uid 1005: exited on signal 4
> pid 49718 (smbd), uid 1005: exited on signal 4
> pid 49719 (smbd), uid 1005: exited on signal 4
> pid 49720 (smbd), uid 1005: exited on signal 4
> pid 49721 (smbd), uid 1005: exited on signal 4
> pid 49722 (smbd), uid 1005: exited on signal 4
> pid 49723 (smbd), uid 1005: exited on signal 4
> pid 49724 (smbd), uid 1005: exited on signal 4
> pid 49725 (smbd), uid 1005: exited on signal 4
> pid 49726 (smbd), uid 1005: exited on signal 4
> pid 49727 (smbd), uid 1005: exited on signal 4
> pid 49728 (smbd), uid 1005: exited on signal 4
> pid 49729 (smbd), uid 1005: exited on signal 4
> pid 49730 (smbd), uid 1005: exited on signal 4
> pid 49731 (smbd), uid 1005: exited on signal 4
> pid 49732 (smbd), uid 1005: exited on signal 4
> pid 49734 (smbd), uid 1005: exited on signal 4
> pid 49735 (smbd), uid 1005: exited on signal 4
> pid 49736 (smbd), uid 1005: exited on signal 4
> pid 49737 (smbd), uid 1005: exited on signal 4
> pid 49738 (smbd), uid 1005: exited on signal 4
> pid 49739 (smbd), uid 1005: exited on signal 4
> pid 49740 (smbd), uid 1005: exited on signal 4
> pid 49741 (smbd), uid 1005: exited on signal 4
> pid 49742 (smbd), uid 1005: exited on signal 4
> pid 49743 (smbd), uid 1005: exited on signal 4
> pid 49744 (smbd), uid 1005: exited on signal 4
> pid 49745 (smbd), uid 1005: exited on signal 4
> pid 49746 (smbd), uid 1005: exited on signal 4
> pid 49747 (smbd), uid 1005: exited on signal 4
> pid 49748 (smbd), uid 1005: exited on signal 4
> pid 49749 (smbd), uid 1005: exited on signal 4
> pid 49750 (smbd), uid 1005: exited on signal 4
> pid 49751 (smbd), uid 1005: exited on signal 4
> pid 49752 (smbd), uid 1005: exited on signal 4
> pid 49764 (smbd), uid 1005: exited on signal 4
> pid 49765 (smbd), uid 1005: exited on signal 4
> pid 49766 (smbd), uid 1005: exited on signal 4
> pid 49767 (smbd), uid 1005: exited on signal 4
> pid 49768 (smbd), uid 1005: exited on signal 4
> pid 49769 (smbd), uid 1005: exited on signal 4
> pid 49770 (smbd), uid 1005: exited on signal 4
> pid 49771 (smbd), uid 1005: exited on signal 4
> pid 49772 (smbd), uid 1005: exited on signal 4
> pid 49773 (smbd), uid 1005: exited on signal 4
> pid 49774 (smbd), uid 1005: exited on signal 4
> pid 49775 (smbd), uid 1005: exited on signal 4
> pid 49776 (smbd), uid 1005: exited on signal 4
> pid 49790 (smbd), uid 1005: exited on signal 4
> pid 49791 (smbd), uid 1005: exited on signal 4
> pid 49792 (smbd), uid 1005: exited on signal 4
> pid 49793 (smbd), uid 1005: exited on signal 4
> pid 49794 (smbd), uid 1005: exited on signal 4
> pid 49795 (smbd), uid 1005: exited on signal 4
> pid 49796 (smbd), uid 1005: exited on signal 4
> pid 49797 (smbd), uid 1005: exited on signal 4
> pid 49798 (smbd), uid 1005: exited on signal 4
> pid 49799 (smbd), uid 1005: exited on signal 4
> pid 49800 (smbd), uid 1005: exited on signal 4
> pid 49801 (smbd), uid 1005: exited on signal 4
> pid 49802 (smbd), uid 1005: exited on signal 4
> pid 49803 (smbd), uid 1005: exited on signal 4
> pid 49804 (smbd), uid 1005: exited on signal 4
> pid 49805 (smbd), uid 1005: exited on signal 4
> pid 49806 (smbd), uid 1005: exited on signal 4
> pid 49807 (smbd), uid 1005: exited on signal 4
> pid 49809 (smbd), uid 1005: exited on signal 4
> pid 49810 (smbd), uid 1005: exited on signal 4
> pid 49811 (smbd), uid 1005: exited on signal 4
> pid 49812 (smbd), uid 1005: exited on signal 4
> pid 49813 (smbd), uid 1005: exited on signal 4
> pid 49814 (smbd), uid 1005: exited on signal 4
> pid 49815 (smbd), uid 1005: exited on signal 4
> pid 49816 (smbd), uid 1005: exited on signal 4
> pid 49817 (smbd), uid 1005: exited on signal 4
> pid 49818 (smbd), uid 1005: exited on signal 4
> pid 49819 (smbd), uid 1005: exited on signal 4
> pid 49820 (smbd), uid 1005: exited on signal 4

-- End of security output --

 

[cyberjock]

Can you provide a debug for your box? I'm going to guess your hardware is inadequate for FreeNAS, but the debug will provide more info.

 

[skimon]

Can you provide a debug for your box? I'm going to guess your hardware is inadequate for FreeNAS, but the debug will provide more info.

Sorry for the late reply , here it is.

 

[skimon]

File was too big , i removed some of the older logs otherwise it wouldnt upload.
----

 

[cyberjock]

Well, sig 4 generally means illegal CPU instruction per some Google searching. So maybe try a BIOS and IPMI update and see if that helps. If not, check your BIOS settings. Nobody else is having that problem, so you have to figure out if this is a bad hardware problem or a bad setting, outdated BIOS, etc.

 

[Ericloewe]

Verifying the installation might also help. For Samba to throw an illegal instruction something unusual has to be going on. It'd be bigger than FreeNAS and on the front page of every deep tech site.

 

[DrKK]

Verifying the installation might also help. For Samba to throw an illegal instruction something unusual has to be going on. It'd be bigger than FreeNAS and on the front page of every deep tech site.

You're dead right, Eric.

You have inspired me: https://bugs.freenas.org/issues/8530

 

[skimon]

Verify installation passed.

I was also seeing a dns resolve error in the smbd log (192.168.x.x != NULL ) also caused by an old machine trying to backup , so i disabled dns resolution in the service. This didnt immediately quiet the log but perhaps it needed to restart the service. I restarted the service and now smbd does not exit.

 

[CoteRL]

I know I'm resurrecting an old thread but it is the exact issue I'm currently seeing after applying the latest updates. On the 9th I started getting emails filled with similar to what's in my console now:

Code:

Sep 23 22:37:21 nas kernel: pid 4943 (smbd), uid 65534: exited on signal 4
Sep 23 22:37:31 nas kernel: pid 4954 (smbd), uid 65534: exited on signal 4
Sep 23 22:37:42 nas kernel: pid 4993 (smbd), uid 65534: exited on signal 4
Sep 23 22:37:52 nas kernel: pid 5017 (smbd), uid 65534: exited on signal 4
Sep 23 22:38:03 nas kernel: pid 5025 (smbd), uid 65534: exited on signal 4
Sep 23 22:38:13 nas kernel: pid 5033 (smbd), uid 65534: exited on signal 4
Sep 23 22:38:24 nas kernel: pid 5040 (smbd), uid 65534: exited on signal 4
Sep 23 22:38:34 nas kernel: pid 5049 (smbd), uid 65534: exited on signal 4
Sep 23 22:38:45 nas kernel: pid 5054 (smbd), uid 65534: exited on signal 4

I believe it started with FreeNAS-9.3-STABLE-201509022158 but is still present on FreeNAS-9.3-STABLE-201509220011. Everything appears to be working fine but that console spew can't be a good thing.

What information could I provide that would help point in the right direction for tracking this down?

 

[Nofun]

I'm getting the same issue's and have been running FreeNAS on the same box since 2012 and never seen this issue.

I did a clean install of 9.3 less then 3 weeks ago and have more recently done an update.

These floods of smbd messages don't occur all the time but seem rather randomly.

EDIT: Unsure if this applies to anyone else but I've found what causes this message in my environment here.

I have two CentOS 6.5 boxes connected to the nas with the latest cifs-utils installed.
Drives are mounted and as soon as I start doing any IO from these centos boxes these messages start spewing out.

If I connect the boxes via NFS all is ok, if I go back to cifs more message spam.

 

[Antoine Roy-Gobeil]

I had similar issues. My files are stored on Freenas and are served by CIFS.

From my Ubuntu machine, I got several IO errors: removing certain files would sometimes hang, writes would fail leaving files truncated. Pretty horrible. On the FreeNAS host I got a bunch of lines like the following in the logs:

Mar 11 14:43:11 neumann kernel: pid 53160 (smbd), uid 1001: exited on signal 4

Ubuntu host: Linux ubuntu 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux after a fresh apt-get upgrade.
FreeNAS host: FreeNAS-9.3-STABLE-201602031011

I now connect using NFS and all is OK. I hope to use CIFS again in the future so my Windows machine can connect as well.

 

[danno]

I had similar issues. My files are stored on Freenas and are served by CIFS.

From my Ubuntu machine, I got several IO errors: removing certain files would sometimes hang, writes would fail leaving files truncated. Pretty horrible. On the FreeNAS host I got a bunch of lines like the following in the logs:

Mar 11 14:43:11 neumann kernel: pid 53160 (smbd), uid 1001: exited on signal 4

Ubuntu host: Linux ubuntu 4.2.0-27-generic #32~14.04.1-Ubuntu SMP Fri Jan 22 15:32:26 UTC 2016 x86_64 x86_64 x86_64 GNU/Linux after a fresh apt-get upgrade.
FreeNAS host: FreeNAS-9.3-STABLE-201602031011

I now connect using NFS and all is OK. I hope to use CIFS again in the future so my Windows machine can connect as well.

There is another forum thread I think you'd be interested to read: https://forums.freenas.org/index.php?threads/smbd-crashing-signal-4-on-file-delete.34865/. The symptoms you describe sound similar to those I am encountering from my Linux Mint laptop. The issue seems to me to be a FreeNAS (CIFS) bug.