after upgrade from 8.3 to 9.2 => periodic reboot (crash?)

[kamuzon]

After upgrade from FreeNAS x64 8.3 to 9.2 (9.2.1.5) at this moment 2 (two) of my freenas systems go to reboot while copying to it (writing to its zpool). I cant see that it "saying" on console. In /var/log/messages is nothing about it.
At one of these two systems I drop and create zpool clearly. But it reboots again then I copy files to it.
SWAT is empty before crash.
No memory peaks.
How to fix it?
What to do?

 

[cyberjock]

How about hardware specs?

 

[kamuzon]

AMD Phenom(tm) 9750 Quad-Core Processor
and 4GB of memory on both systems
5 and 6 1TB seagate disks

 

[kamuzon]

Here is boot log

Code:

May  9 02:18:02 freenas118 syslogd: kernel boot file is /boot/kernel-debug/kernel
May  9 02:18:02 freenas118 kernel: Copyright (c) 1992-2013 The FreeBSD Project.
May  9 02:18:02 freenas118 kernel: Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
May  9 02:18:02 freenas118 kernel: The Regents of the University of California. All rights reserved.
May  9 02:18:02 freenas118 kernel: FreeBSD is a registered trademark of The FreeBSD Foundation.
May  9 02:18:02 freenas118 kernel: FreeBSD 9.2-RELEASE-p4 #0 r262572+17a4d3d: Wed Apr 23 10:11:41 PDT 2014
May  9 02:18:02 freenas118 kernel: root@build3.ixsystems.com:/tank/home/jkh/build/9.2.1/freenas/os-base/amd64/fusion/jkh/9.2.1/freenas/FreeBSD/src/sys
/FREENAS.amd64-DEBUG amd64
May  9 02:18:02 freenas118 kernel: gcc version 4.2.1 20070831 patched [FreeBSD]
May  9 02:18:02 freenas118 kernel: WARNING: WITNESS option enabled, expect reduced performance.
May  9 02:18:02 freenas118 kernel: CPU: AMD Phenom(tm) 9750 Quad-Core Processor (2411.03-MHz K8-class CPU)
May  9 02:18:02 freenas118 kernel: Origin = "AuthenticAMD"  Id = 0x100f23  Family = 0x10  Model = 0x2  Stepping = 3
May  9 02:18:02 freenas118 kernel: Features=0x178bfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,MMX,FXSR,SSE,SS
E2,HTT>
May  9 02:18:02 freenas118 kernel: Features2=0x802009<SSE3,MON,CX16,POPCNT>
May  9 02:18:02 freenas118 kernel: AMD Features=0xee500800<SYSCALL,NX,MMX+,FFXSR,Page1GB,RDTSCP,LM,3DNow!+,3DNow!>
May  9 02:18:02 freenas118 kernel: AMD Features2=0x7ff<LAHF,CMP,SVM,ExtAPIC,CR8,ABM,SSE4A,MAS,Prefetch,OSVW,IBS>
May  9 02:18:02 freenas118 kernel: TSC: P-state invariant
May  9 02:18:02 freenas118 kernel: real memory  = 4294967296 (4096 MB)
May  9 02:18:02 freenas118 kernel: avail memory = 3964735488 (3781 MB)
May  9 02:18:02 freenas118 kernel: Event timer "LAPIC" quality 400
May  9 02:18:02 freenas118 kernel: ACPI APIC Table: <GBT    GBTUACPI>
May  9 02:18:02 freenas118 kernel: FreeBSD/SMP: Multiprocessor System Detected: 4 CPUs
May  9 02:18:02 freenas118 kernel: FreeBSD/SMP: 1 package(s) x 4 core(s)
May  9 02:18:02 freenas118 kernel: cpu0 (BSP): APIC ID:  0
May  9 02:18:02 freenas118 kernel: cpu1 (AP): APIC ID:  1
May  9 02:18:02 freenas118 kernel: cpu2 (AP): APIC ID:  2
May  9 02:18:02 freenas118 kernel: cpu3 (AP): APIC ID:  3
May  9 02:18:02 freenas118 kernel: WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
May  9 02:18:02 freenas118 kernel: ioapic0: Changing APIC ID to 2
May  9 02:18:02 freenas118 kernel: ioapic0 <Version 2.1> irqs 0-23 on motherboard
May  9 02:18:02 freenas118 kernel: kbd1 at kbdmux0
May  9 02:18:02 freenas118 kernel: cryptosoft0: <software crypto> on motherboard
May  9 02:18:02 freenas118 kernel: aesni0: No AESNI support.
May  9 02:18:02 freenas118 kernel: padlock0: No ACE support.
May  9 02:18:02 freenas118 kernel: acpi0: <GBT GBTUACPI> on motherboard
May  9 02:18:02 freenas118 kernel: acpi0: Power Button (fixed)
May  9 02:18:02 freenas118 kernel: acpi0: reservation of 0, a0000 (3) failed
May  9 02:18:02 freenas118 kernel: acpi0: reservation of 100000, cfde0000 (3) failed
May  9 02:18:02 freenas118 kernel: cpu0: <ACPI CPU> on acpi0
May  9 02:18:02 freenas118 kernel: cpu1: <ACPI CPU> on acpi0
May  9 02:18:02 freenas118 kernel: cpu2: <ACPI CPU> on acpi0
May  9 02:18:02 freenas118 kernel: cpu3: <ACPI CPU> on acpi0
May  9 02:18:02 freenas118 kernel: attimer0: <AT timer> port 0x40-0x43 on acpi0
May  9 02:18:02 freenas118 kernel: Timecounter "i8254" frequency 1193182 Hz quality 0
May  9 02:18:02 freenas118 kernel: Event timer "i8254" frequency 1193182 Hz quality 100
May  9 02:18:02 freenas118 kernel: hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff irq 0,8 on acpi0
May  9 02:18:02 freenas118 kernel: Timecounter "HPET" frequency 14318180 Hz quality 950
May  9 02:18:02 freenas118 kernel: Event timer "HPET" frequency 14318180 Hz quality 450
May  9 02:18:02 freenas118 kernel: Event timer "HPET1" frequency 14318180 Hz quality 450
May  9 02:18:02 freenas118 kernel: Event timer "HPET2" frequency 14318180 Hz quality 450
May  9 02:18:02 freenas118 kernel: atrtc0: <AT realtime clock> port 0x70-0x73 on acpi0
May  9 02:18:02 freenas118 kernel: Event timer "RTC" frequency 32768 Hz quality 0
May  9 02:18:02 freenas118 kernel: Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
May  9 02:18:02 freenas118 kernel: acpi_timer0: <32-bit timer at 3.579545MHz> port 0x4008-0x400b on acpi0
May  9 02:18:02 freenas118 kernel: acpi_button0: <Power Button> on acpi0
May  9 02:18:02 freenas118 kernel: pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
May  9 02:18:02 freenas118 kernel: pci0: <ACPI PCI bus> on pcib0
May  9 02:18:02 freenas118 kernel: pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
May  9 02:18:02 freenas118 kernel: pci1: <ACPI PCI bus> on pcib1
May  9 02:18:02 freenas118 kernel: vgapci0: <VGA-compatible display> port 0xee00-0xeeff mem 0xd8000000-0xdfffffff,0xfdee0000-0xfdeeffff,0xfdd00000-0xf
ddfffff irq 18 at device 5.0 on pci1
May  9 02:18:02 freenas118 kernel: pci1: <multimedia, HDA> at device 5.1 (no driver attached)
May  9 02:18:02 freenas118 kernel: pcib2: <ACPI PCI-PCI bridge> irq 18 at device 10.0 on pci0
May  9 02:18:02 freenas118 kernel: pci2: <ACPI PCI bus> on pcib2
May  9 02:18:02 freenas118 kernel: re0: <RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet> port 0xde00-0xdeff mem 0xfdfff000-0xfdffffff,0xfdf
e0000-0xfdfeffff irq 18 at device 0.0 on pci2
May  9 02:18:02 freenas118 kernel: re0: Using 1 MSI-X message
May  9 02:18:02 freenas118 kernel: re0: Chip rev. 0x3c000000
May  9 02:18:02 freenas118 kernel: re0: MAC rev. 0x00200000
May  9 02:18:02 freenas118 kernel: miibus0: <MII bus> on re0
May  9 02:18:02 freenas118 kernel: rgephy0: <RTL8169S/8110S/8211 1000BASE-T media interface> PHY 1 on miibus0
May  9 02:18:02 freenas118 kernel: rgephy0:  none, 10baseT, 10baseT-FDX, 10baseT-FDX-flow, 100baseTX, 100baseTX-FDX, 100baseTX-FDX-flow, 1000baseT, 10
00baseT-master, 1000baseT-FDX, 1000baseT-FDX-master, 1000baseT-FDX-flow, 1000baseT-FDX-flow-master, auto, auto-flow
May  9 02:18:02 freenas118 kernel: re0: Ethernet address: 00:1f:d0:51:c9:77
May  9 02:18:02 freenas118 kernel: ahci0: <ATI IXP700 AHCI SATA controller> port 0xff00-0xff07,0xfe00-0xfe03,0xfd00-0xfd07,0xfc00-0xfc03,0xfb00-0xfb0f
mem 0xfe02f000-0xfe02f3ff irq 22 at device 17.0 on pci0
May  9 02:18:02 freenas118 kernel: ahci0: AHCI v1.10 with 4 3Gbps ports, Port Multiplier supported
May  9 02:18:02 freenas118 kernel: ahcich0: <AHCI channel> at channel 0 on ahci0
May  9 02:18:02 freenas118 kernel: ahcich1: <AHCI channel> at channel 1 on ahci0
May  9 02:18:02 freenas118 kernel: ahcich2: <AHCI channel> at channel 2 on ahci0
May  9 02:18:02 freenas118 kernel: ahcich3: <AHCI channel> at channel 3 on ahci0
May  9 02:18:02 freenas118 kernel: ohci0: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02e000-0xfe02efff irq 16 at device 18.0 on pci0
May  9 02:18:02 freenas118 kernel: usbus0 on ohci0
May  9 02:18:02 freenas118 kernel: ohci1: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02d000-0xfe02dfff irq 16 at device 18.1 on pci0
May  9 02:18:02 freenas118 kernel: usbus1 on ohci1
May  9 02:18:02 freenas118 kernel: ehci0: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe02c000-0xfe02c0ff irq 17 at device 18.2 on pci0
May  9 02:18:02 freenas118 kernel: ehci0: AMD SB600/700 quirk applied
May  9 02:18:02 freenas118 kernel: usbus2: EHCI version 1.0
May  9 02:18:02 freenas118 kernel: usbus2 on ehci0
May  9 02:18:02 freenas118 kernel: ohci2: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02b000-0xfe02bfff irq 18 at device 19.0 on pci0
May  9 02:18:02 freenas118 kernel: usbus3 on ohci2
May  9 02:18:02 freenas118 kernel: ohci3: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe02a000-0xfe02afff irq 18 at device 19.1 on pci0
May  9 02:18:02 freenas118 kernel: usbus4 on ohci3
May  9 02:18:02 freenas118 kernel: ehci1: <AMD SB7x0/SB8x0/SB9x0 USB 2.0 controller> mem 0xfe029000-0xfe0290ff irq 19 at device 19.2 on pci0
May  9 02:18:02 freenas118 kernel: ehci1: AMD SB600/700 quirk applied
May  9 02:18:02 freenas118 kernel: usbus5: EHCI version 1.0
May  9 02:18:02 freenas118 kernel: usbus5 on ehci1
May  9 02:18:02 freenas118 kernel: pci0: <serial bus, SMBus> at device 20.0 (no driver attached)
May  9 02:18:02 freenas118 kernel: atapci0: <ATI IXP700/800 UDMA133 controller> port 0x1f0-0x1f7,0x3f6,0x170-0x177,0x376,0xfa00-0xfa0f at device 20.1
on pci0
May  9 02:18:02 freenas118 kernel: ata0: <ATA channel> at channel 0 on atapci0
May  9 02:18:02 freenas118 kernel: ata1: <ATA channel> at channel 1 on atapci0
May  9 02:18:02 freenas118 kernel: isab0: <PCI-ISA bridge> at device 20.3 on pci0
May  9 02:18:02 freenas118 kernel: isa0: <ISA bus> on isab0
May  9 02:18:02 freenas118 kernel: pcib3: <ACPI PCI-PCI bridge> at device 20.4 on pci0
May  9 02:18:02 freenas118 kernel: pci3: <ACPI PCI bus> on pcib3
May  9 02:18:02 freenas118 kernel: fwohci0: <Texas Instruments TSB43AB23> mem 0xfdcff000-0xfdcff7ff,0xfdcf8000-0xfdcfbfff irq 22 at device 14.0 on pci
3
May  9 02:18:02 freenas118 kernel: fwohci0: OHCI version 1.10 (ROM=0)
May  9 02:18:02 freenas118 kernel: fwohci0: No. of Isochronous channels is 4.
May  9 02:18:02 freenas118 kernel: fwohci0: EUI64 00:de:4c:0c:00:00:1d:7d
May  9 02:18:02 freenas118 kernel: fwohci0: Phy 1394a available S400, 3 ports.
May  9 02:18:02 freenas118 kernel: fwohci0: Link S400, max_rec 2048 bytes.
May  9 02:18:02 freenas118 kernel: firewire0: <IEEE1394(FireWire) bus> on fwohci0
May  9 02:18:02 freenas118 kernel: sbp0: <SBP-2/SCSI over FireWire> on firewire0
May  9 02:18:02 freenas118 kernel: fwohci0: Initiate bus reset
May  9 02:18:02 freenas118 kernel: fwohci0: fwohci_intr_core: BUS reset
May  9 02:18:02 freenas118 kernel: fwohci0: fwohci_intr_core: node_id=0x00000000, SelfID Count=1, CYCLEMASTER mode
May  9 02:18:02 freenas118 kernel: ohci4: <AMD SB7x0/SB8x0/SB9x0 USB controller> mem 0xfe028000-0xfe028fff irq 18 at device 20.5 on pci0
May  9 02:18:02 freenas118 kernel: usbus6 on ohci4
May  9 02:18:02 freenas118 kernel: amdtemp0: <AMD CPU On-Die Thermal Sensors> on hostb4
May  9 02:18:02 freenas118 kernel: atkbdc0: <Keyboard controller (i8042)> port 0x60,0x64 irq 1 on acpi0
May  9 02:18:02 freenas118 kernel: atkbd0: <AT Keyboard> irq 1 on atkbdc0
May  9 02:18:02 freenas118 kernel: kbd0 at atkbd0
May  9 02:18:02 freenas118 kernel: atkbd0: [GIANT-LOCKED]
May  9 02:18:02 freenas118 kernel: amdsbwd0: <AMD SB600/SB7xx Watchdog Timer> at iomem 0xfec000f0-0xfec000f3,0xfec000f4-0xfec000f7 on isa0
May  9 02:18:02 freenas118 kernel: sc0: <System console> at flags 0x100 on isa0
May  9 02:18:02 freenas118 kernel: sc0: VGA <16 virtual consoles, flags=0x300>
May  9 02:18:02 freenas118 kernel: vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
May  9 02:18:02 freenas118 kernel: wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
May  9 02:18:02 freenas118 kernel: hwpstate0: <Cool`n'Quiet 2.0> on cpu0
May  9 02:18:02 freenas118 kernel: firewire0: 1 nodes, maxhop <= 0 cable IRM irm(0)  (me)
May  9 02:18:02 freenas118 kernel: firewire0: bus manager 0
May  9 02:18:02 freenas118 kernel: WITNESS: unable to allocate a new witness object
May  9 02:18:02 freenas118 kernel: Timecounters tick every 1.000 msec
May  9 02:18:02 freenas118 kernel: ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
May  9 02:18:02 freenas118 kernel: DUMMYNET 0xfffffe00022bfd40 with IPv6 initialized (100409)
May  9 02:18:02 freenas118 kernel: load_dn_sched dn_sched RR loaded
May  9 02:18:02 freenas118 kernel: load_dn_sched dn_sched WF2Q+ loaded
May  9 02:18:02 freenas118 kernel: load_dn_sched dn_sched FIFO loaded
May  9 02:18:02 freenas118 kernel: load_dn_sched dn_sched PRIO loaded
May  9 02:18:02 freenas118 kernel: load_dn_sched dn_sched QFQ loaded
May  9 02:18:02 freenas118 kernel: usbus0: 12Mbps Full Speed USB v1.0
May  9 02:18:02 freenas118 kernel: usbus1: 12Mbps Full Speed USB v1.0
May  9 02:18:02 freenas118 kernel: usbus2: 480Mbps High Speed USB v2.0
May  9 02:18:02 freenas118 kernel: usbus3: 12Mbps Full Speed USB v1.0
May  9 02:18:02 freenas118 kernel: usbus4: 12Mbps Full Speed USB v1.0
May  9 02:18:02 freenas118 kernel: usbus5: 480Mbps High Speed USB v2.0
May  9 02:18:02 freenas118 kernel: usbus6: 12Mbps Full Speed USB v1.0
May  9 02:18:02 freenas118 kernel: ugen0.1: <ATI> at usbus0
May  9 02:18:02 freenas118 kernel: uhub0: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
May  9 02:18:02 freenas118 kernel: ugen1.1: <ATI> at usbus1
May  9 02:18:02 freenas118 kernel: uhub1: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
May  9 02:18:02 freenas118 kernel: ugen2.1: <ATI> at usbus2
May  9 02:18:02 freenas118 kernel: uhub2: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus2
May  9 02:18:02 freenas118 kernel: ugen3.1: <ATI> at usbus3
May  9 02:18:02 freenas118 kernel: uhub3: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus3
May  9 02:18:02 freenas118 kernel: ugen4.1: <ATI> at usbus4
May  9 02:18:02 freenas118 kernel: uhub4: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
May  9 02:18:02 freenas118 kernel: ugen5.1: <ATI> at usbus5
May  9 02:18:02 freenas118 kernel: uhub5: <ATI EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus5
May  9 02:18:02 freenas118 kernel: ugen6.1: <ATI> at usbus6
May  9 02:18:02 freenas118 kernel: uhub6: <ATI OHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
May  9 02:18:02 freenas118 kernel: (aprobe0:ahcich0:0:15:0): NOP. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
May  9 02:18:02 freenas118 kernel: (aprobe0:ahcich0:0:15:0): CAM status: Command timeout
May  9 02:18:02 freenas118 kernel: (aprobe0:ahcich0:0:15:0): Error 5, Retries exhausted
May  9 02:18:02 freenas118 kernel: (aprobe1:ahcich1:0:15:0): NOP. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
May  9 02:18:02 freenas118 kernel: (aprobe1:ahcich1:0:15:0): CAM status: Command timeout
May  9 02:18:02 freenas118 kernel: (aprobe1:ahcich1:0:15:0): Error 5, Retries exhausted
May  9 02:18:02 freenas118 kernel: (aprobe2:ahcich2:0:15:0): NOP. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
May  9 02:18:02 freenas118 kernel: (aprobe2:ahcich2:0:15:0): CAM status: Command timeout
May  9 02:18:02 freenas118 kernel: (aprobe2:ahcich2:0:15:0): Error 5, Retries exhausted
May  9 02:18:02 freenas118 kernel: (aprobe3:ahcich3:0:15:0): NOP. ACB: 00 00 00 00 00 00 00 00 00 00 00 00
May  9 02:18:02 freenas118 kernel: (aprobe3:ahcich3:0:15:0): CAM status: Command timeout
May  9 02:18:02 freenas118 kernel: (aprobe3:ahcich3:0:15:0): Error 5, Retries exhausted
May  9 02:18:02 freenas118 kernel: uhub6: 2 ports with 2 removable, self powered
May  9 02:18:02 freenas118 kernel: uhub0: 3 ports with 3 removable, self powered
May  9 02:18:02 freenas118 kernel: uhub1: 3 ports with 3 removable, self powered
May  9 02:18:02 freenas118 kernel: uhub3: 3 ports with 3 removable, self powered
May  9 02:18:02 freenas118 kernel: uhub4: 3 ports with 3 removable, self powered
May  9 02:18:02 freenas118 kernel: uhub2: 6 ports with 6 removable, self powered
May  9 02:18:02 freenas118 kernel: uhub5: 6 ports with 6 removable, self powered
May  9 02:18:02 freenas118 kernel: ugen2.2: <JetFlash> at usbus2
May  9 02:18:02 freenas118 kernel: umass0: <JetFlash Mass Storage Device, class 0/0, rev 2.00/1.00, addr 2> on usbus2
May  9 02:18:02 freenas118 kernel: umass0:  SCSI over Bulk-Only; quirks = 0x4000
May  9 02:18:02 freenas118 kernel: umass0:8:0:-1: Attached to scbus8
May  9 02:18:02 freenas118 kernel: da0 at umass-sim0 bus 0 scbus8 target 0 lun 0
May  9 02:18:02 freenas118 kernel: da0: <JetFlash Transcend 4GB 8.01> Removable Direct Access SCSI-2 device
May  9 02:18:02 freenas118 kernel: da0: Serial Number PPC7NM0T
May  9 02:18:02 freenas118 kernel: da0: 40.000MB/s transfers
May  9 02:18:02 freenas118 kernel: da0: 3818MB (7821300 512 byte sectors: 255H 63S/T 486C)
May  9 02:18:02 freenas118 kernel: da0: quirks=0x12<NO_6_BYTE,NO_RC16>
May  9 02:18:02 freenas118 kernel: ada0 at ahcich0 bus 0 scbus0 target 0 lun 0
May  9 02:18:02 freenas118 kernel: ada0: <ST1000VX000-1CU162 CV22> ATA-8 SATA 3.x device
May  9 02:18:02 freenas118 kernel: ada0: Serial Number S1D736WV
May  9 02:18:02 freenas118 kernel: ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May  9 02:18:02 freenas118 kernel: ada0: Command Queueing enabled
May  9 02:18:02 freenas118 kernel: cd0 at umass-sim0 bus 0 scbus8 target 0 lun 1
May  9 02:18:02 freenas118 kernel: cd0: <Generic AutoRun Disk 8.00> Removable CD-ROM SCSI-2 device
May  9 02:18:02 freenas118 kernel: cd0: Serial Number PPC7NM0T
May  9 02:18:02 freenas118 kernel: cd0: 40.000MB/s transfers
May  9 02:18:02 freenas118 kernel: cd0: cd present [22540 x 512 byte records]
May  9 02:18:02 freenas118 kernel: cd0: quirks=0x10<10_BYTE_ONLY>
May  9 02:18:02 freenas118 kernel: ada0: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
May  9 02:18:02 freenas118 kernel: ada0: Previously was known as ad4
May  9 02:18:02 freenas118 kernel: ada1 at ahcich1 bus 0 scbus1 target 0 lun 0
May  9 02:18:02 freenas118 kernel: ada1: <ST31000528AS CC38> ATA-8 SATA 2.x device
May  9 02:18:02 freenas118 kernel: ada1: Serial Number 9VP2QSWW
May  9 02:18:02 freenas118 kernel: ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May  9 02:18:02 freenas118 kernel: ada1: Command Queueing enabled
May  9 02:18:02 freenas118 kernel: ada1: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
May  9 02:18:02 freenas118 kernel: ada1: Previously was known as ad6
May  9 02:18:02 freenas118 kernel: ada2 at ahcich2 bus 0 scbus2 target 0 lun 0
May  9 02:18:02 freenas118 kernel: ada2: <ST31000528AS CC38> ATA-8 SATA 2.x device
May  9 02:18:02 freenas118 kernel: ada2: Serial Number 9VP2X1E4
May  9 02:18:02 freenas118 kernel: ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May  9 02:18:02 freenas118 kernel: ada2: Command Queueing enabled
May  9 02:18:02 freenas118 kernel: ada2: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
May  9 02:18:02 freenas118 kernel: ada2: Previously was known as ad8
May  9 02:18:02 freenas118 kernel: ada3 at ahcich3 bus 0 scbus3 target 0 lun 0
May  9 02:18:02 freenas118 kernel: ada3: <ST31000528AS CC38> ATA-8 SATA 2.x device
May  9 02:18:02 freenas118 kernel: ada3: Serial Number 9VP2XL4M
May  9 02:18:02 freenas118 kernel: ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
May  9 02:18:02 freenas118 kernel: ada3: Command Queueing enabled
May  9 02:18:02 freenas118 kernel: ada3: 953869MB (1953525168 512 byte sectors: 16H 63S/T 16383C)
May  9 02:18:02 freenas118 kernel: ada3: Previously was known as ad10
May  9 02:18:02 freenas118 kernel: ada4 at ata1 bus 0 scbus5 target 0 lun 0
May  9 02:18:02 freenas118 kernel: ada4: <ST1000VX000-1CU162 CV22> ATA-8 SATA 3.x device
May  9 02:18:02 freenas118 kernel: ada4: Serial Number S1D7370X
May  9 02:18:02 freenas118 kernel: ada4: 150.000MB/s transfers (SATA, UDMA6, PIO 8192bytes)
May  9 02:18:02 freenas118 kernel: ada4: 953865MB (1953516911 512 byte sectors: 16H 63S/T 16383C)
May  9 02:18:02 freenas118 kernel: ada4: Previously was known as ad2
May  9 02:18:02 freenas118 kernel: SMP: AP CPU #1 Launched!
May  9 02:18:02 freenas118 kernel: SMP: AP CPU #3 Launched!
May  9 02:18:02 freenas118 kernel: SMP: AP CPU #2 Launched!
May  9 02:18:02 freenas118 kernel: Timecounter "TSC-low" frequency 1205515130 Hz quality 800
May  9 02:18:02 freenas118 kernel: WARNING: WITNESS option enabled, expect reduced performance.
May  9 02:18:02 freenas118 kernel: Trying to mount root from ufs:/dev/ufs/FreeNASs1a [ro]...
May  9 02:18:02 freenas118 kernel: WARNING: /data was not properly dismounted
May  9 02:18:02 freenas118 kernel: GEOM_RAID5: Module loaded, version 1.1.20130907.44 (rev 5c6d2a159411)
May  9 02:18:02 freenas118 kernel: ZFS NOTICE: Prefetch is disabled by default if less than 4GB of RAM is present;
May  9 02:18:02 freenas118 kernel: to enable, add "vfs.zfs.prefetch_disable=0" to /boot/loader.conf.
May  9 02:18:02 freenas118 kernel: ZFS filesystem version: 5
May  9 02:18:02 freenas118 kernel: ZFS storage pool version: features support (5000)
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Device ada0p1.eli created.
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Encryption: AES-XTS 256
May  9 02:18:02 freenas118 kernel: GEOM_ELI:    Crypto: software
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Device ada1p1.eli created.
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Encryption: AES-XTS 256
May  9 02:18:02 freenas118 kernel: GEOM_ELI:    Crypto: software
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Device ada2p1.eli created.
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Encryption: AES-XTS 256
May  9 02:18:02 freenas118 kernel: GEOM_ELI:    Crypto: software
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Device ada3p1.eli created.
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Encryption: AES-XTS 256
May  9 02:18:02 freenas118 kernel: GEOM_ELI:    Crypto: software
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Device ada4p1.eli created.
May  9 02:18:02 freenas118 kernel: GEOM_ELI: Encryption: AES-XTS 256
May  9 02:18:02 freenas118 kernel: GEOM_ELI:    Crypto: software
May  9 02:18:02 freenas118 root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
May  9 02:18:03 freenas118 ntpd[2095]: ntpd 4.2.4p5-a (1)
May  9 02:18:03 freenas118 kernel: ....+++
May  9 02:18:04 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD 1.3.4d (maint) (built Wed Apr 23 2014 17:41:28 UTC) standalone mode STARTUP
May  9 02:18:04 freenas118 generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: /sbin/sysctl -n 'kern.maxfilesperproc'
May  9 02:18:04 freenas118 generate_smb4_conf.py: [common.pipesubr:58] Popen()ing: /usr/local/bin/pdbedit -d 0 -i smbpasswd:/tmp/tmpwxC50k -e tdbsam:/
var/etc/private/passdb.tdb -s /usr/local/etc/smb4.conf
May  9 02:18:06 freenas118 smartd[2334]: Device: /dev/ada1, WARNING: A firmware update for this drive may be available,
May  9 02:18:06 freenas118 smartd[2334]: see the following Seagate web pages:
May  9 02:18:06 freenas118 smartd[2334]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en
May  9 02:18:06 freenas118 smartd[2334]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en
May  9 02:18:06 freenas118 smartd[2334]: Device: /dev/ada2, WARNING: A firmware update for this drive may be available,
May  9 02:18:06 freenas118 smartd[2334]: see the following Seagate web pages:
May  9 02:18:06 freenas118 smartd[2334]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en
May  9 02:18:06 freenas118 smartd[2334]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en
May  9 02:18:06 freenas118 smartd[2334]: Device: /dev/ada3, WARNING: A firmware update for this drive may be available,
May  9 02:18:06 freenas118 smartd[2334]: see the following Seagate web pages:
May  9 02:18:06 freenas118 smartd[2334]: http://knowledge.seagate.com/articles/en_US/FAQ/207931en
May  9 02:18:06 freenas118 smartd[2334]: http://knowledge.seagate.com/articles/en_US/FAQ/213891en
May  9 02:18:11 freenas118 mDNSResponder: mDNSResponder (Engineering Build) (Apr 23 2014 17:17:29) starting
May  9 02:18:11 freenas118 mDNSResponder:  8: Listening for incoming Unix Domain Socket client requests
May  9 02:18:11 freenas118 mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
May  9 02:18:11 freenas118 mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
May  9 02:18:13 freenas118 kernel: done.
May  9 02:18:13 freenas118 mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000800C2FD60 freenas118.local. (Addr) that'
s already in the list
May  9 02:18:13 freenas118 mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000800C30180 118.18.18.172.in-addr.arpa. (P
TR) that's already in the list

 

[cyberjock]

Well, you don't meet the minimum hardware requirements. Please upgrade to 8GB minimum as required per the manual and see if that fixes your problem.

 

[kamuzon]

But why 8.3 works fine??
BTW It's not a solution because I have three more FreeNASes which work fine:
AMD Athlon(tm) Dual Core Processor 4850e with 4GB
AMD Athlon(tm) 64 X2 Dual Core Processor 4200+ with 6GB
and again
AMD Athlon(tm) Dual Core Processor 4850e with 4GB

 

[cyberjock]

Great. Fantastic. That doesn't change the fact that you have a problem and don't meet the minimum requirments. Those requirements are set because there's common problems that manifest randomly if you don't have that minimum. FreeNAS 8.x doesn't have the same requirements as 9.x. As a result a whole bunch of people start asking why.

Didn't you wonder why I asked for hardware specs right off the bat? Because I knew what was coming....

Please be courteous to people that read and respond to your posts and at least meet the minimum requirements before posting. It's considered bad taste to create threads for problems that could very well be caused by insufficient hardware. It's not particularly fun to invest huge amounts of time writing responses for you when we already provide minimum requirements that you don't meet.

 

[galaxsat]

I have had the same problem since getting into 9.2 a few months ago. Most recently, instead of upgrading to 9.2.1.5, I tried a fresh install (on a different USB 2GB flash drive) and restored my 9.2.1.4 config. It reboots less frequently now, but still has rebooted twice in the past week. Like zamuzon, I also have nothing in the logs indicating a problem, and it's even logging to a remote syslog server. I didn't post until now because this thread is the first that matches my issue, and I hadn't tried a fresh install until just last week.

FreeNAS-9.2.1.5-RELEASE-x64 (80c1d35)
AMD Athlon(tm) II X2 4450e Processor
7913MB reported memory (8GB)
4 1TB HD's in a 1TB RAIDZ2 array using ZFS

 

[kamuzon]

Well done.
Now there is 8GB in both problem NAS.

When I copy data from one to another. Receiver go to reboot:

after that it will up

What should I do to fix this?
Downgrade to 8.3 via fresh install?

BTW system load before reboot.

 

[kamuzon]

BTW in logs I see:
(on another FreeNAS'es not)
What's this?

Code:

mDNSResponder: mDNSResponder (Engineering Build) (Apr 23 2014 17:17:29) starting
May 13 23:38:00 freenas118 mDNSResponder:  8: Listening for incoming Unix Domain Socket client requests
May 13 23:38:00 freenas118 mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
May 13 23:38:00 freenas118 mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
May 13 23:38:02 freenas118 kernel: done.
May 13 23:38:02 freenas118 mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000800C2FD60 freenas118.local. (Addr) that's already in the list
May 13 23:38:02 freenas118 mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000800C30180 118.18.18.172.in-addr.arpa. (PTR) that's already in the list

 

[cyberjock]

Ok, those errors are normal for 9.2.1.x. You can safely ignore those.

What happens if you do some dd reads? Does memory utilization go up? Something like dd if=/mnt/poolname/somebigfile of=/devnull bs=1M should show your memory utilization going up.

Do you do scrubs regularly? When was the last time one completed(If you aren't sure zpool status will tell you).

How long has the problem system been in use? Have you changed anything recently besides upgrading?

I will say that Realteks have been a major problem for a lot of people. Seemingly unrelated problems have manifested themselves and as soon as people went to an Intel NIC the problem suddenly went away. Do you have spare Intel NICs you could put in just to rule out the Realtek?

Also can you provide the model number for your motherboard? Are you using ECC RAM? Have you tested your RAM recently?

 

[kamuzon]

...and it in down again. :(
Here how it looks from transferer NAS:
at 21:06 down
at 21:18 there is traffic - it ups?! I dont know.
21:44 transfer is again in progress
21:53 again in down

as I see NFS is in stail and mc stop copy

at 23:25 I resume copy and start writing today posts.
at 23:34 118 is down again.
at 23:42 is again up and copy continue.

I dont understand anything - it copy only 6 minutes and then goes to reboot and then it works 20 minutes.
How to fix it?!

 

[kamuzon]

What happens if you do some dd reads? Does memory utilization go up? Something like dd if=/mnt/poolname/somebigfile of=/devnull bs=1M should show your memory utilization going up.

I'll try so.
On pics I see memory util doesnt go upper then 1.8GB
Loot at first image from my today posts.
And one more:

Do you do scrubs regularly? When was the last time one completed(If you aren't sure zpool status will tell you).

I'd create this zpool a few days ago. Before one day then I start this tread.

How long has the problem system been in use? Have you changed anything recently besides upgrading?

A long.
It was my hardworkusage NAS with a lot of sites. I'd upgrade it last. After others NASes. I was looking how it works.
And I press upload "firmware". It reboots and "fun" starts.

I will say that Realteks have been a major problem for a lot of people. Seemingly unrelated problems have manifested themselves and as soon as people went to an Intel NIC the problem suddenly went away. Do you have spare Intel NICs you could put in just to rule out the Realtek?

Here is new created NAS that I'd build after "fun" starts.
It is re0 too.

Here is graph then I copying data from justUpgradedNAS to freshBuildedNAS.

And here is uptime

Also can you provide the model number for your motherboard? Are you using ECC RAM? Have you tested your RAM recently?

How can I look it from inside NAS?
I thinks it's Gigabyte mATX motherboard.
No ECC RAM.
I just bought new 4x2GB DDR2 800MHz CL5 today. Twice.

 

[kamuzon]

BTW!
justUpgradedNAS didnt have ZFS pools. Only UFS gmirrors.
It works fine on 8.3 and starts rebootings on 9.2. And as you know it checks UFS long time after each crash. After that gmirror mirror all disk. Terrible.
It was hard threeDaysCopying.

After that I drop all volumes and create raidz1 from 5x1 TB disks. And start copy another data from another NAS to justUpgradedNAS and it starts reboot again. F... I was thinking. And now I here looking for help.

 

[kamuzon]

BTW.
It doesnt rotate log file messages.
Maybe some old scripts (from 8.3) somehow stay in system directories?!

Code:

[kamuzon@freenas118] /var/log# ls -la
total 395
drwxr-xr-x  4 root  wheel      31 May 14 00:00 ./
drwxr-xr-x  3 root  wheel      3 May  9 03:35 ../
-rw-r--r--  1 root  wheel      1 May 13 03:01 3ware_raid_alarms.today
-rw-r--r--  1 root  wheel      1 May 12 03:01 3ware_raid_alarms.yesterday
-rw-------  1 root  wheel    1988 May 14 00:57 auth.log
-rw-------  1 root  wheel  421013 May 14 00:55 cron
-rw-------  1 root  wheel  12402 May 10 03:01 dmesg.today
-rw-------  1 root  wheel      0 May  9 03:35 lpd-errs
-rw-r-----  1 root  wheel      64 May 14 00:00 maillog
-rw-r-----  1 root  wheel    123 May 14 00:00 maillog.0.bz2
-rw-r-----  1 root  wheel    119 May 13 00:00 maillog.1.bz2
-rw-r-----  1 root  wheel    118 May 12 00:00 maillog.2.bz2
-rw-r-----  1 root  wheel    120 May 11 00:00 maillog.3.bz2
-rw-r-----  1 root  wheel      99 May 10 00:00 maillog.4.bz2
-rw-------  1 root  wheel  208753 May 14 00:57 messages
-rw-------  1 root  wheel    533 May 10 03:01 mount.today
-rw-r--r--  1 www  www    390477 May 14 00:58 nginx-access.log
-rw-r--r--  1 www  www    21726 May 14 00:00 nginx-access.log.0.bz2
-rw-r--r--  1 www  www        0 May 14 00:19 nginx-error.log
-rw-r--r--  1 root  wheel    162 May 14 00:19 pbid.log
-rw-------  1 root  wheel      0 May 10 03:01 pf.today
-rw-------  1 root  wheel      0 May  9 03:35 ppp.log
drwxr-xr-x  2 root  wheel      2 May  9 03:39 proftpd/
drwxr-xr-x  2 root  wheel      2 May  9 03:35 samba4/
-rw-------  1 root  wheel      0 May  9 03:35 security
-rw-r--r--  1 uucp  uucp      66 May  9 03:39 ups.log
-rw-------  1 root  wheel    7728 May 14 00:19 userlog
-rw-r--r--  1 root  wheel    197 May 14 00:57 utx.lastlogin
-rw-r--r--  1 root  wheel    216 May 14 00:57 utx.log
-rw-r--r--  1 root  wheel      0 May  9 03:39 wtmp
-rw-------  1 root  wheel    1643 May 14 00:19 xferlog

BTW
todays reboots after installation of 8GB:

Code:

May 13 20:36:14 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD killed (signal 15)
May 13 20:36:14 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD 1.3.4d standalone mode SHUTDOWN
May 13 20:44:38 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD 1.3.4d (maint) (built Wed Apr 23 2014 17:41:28 UTC) standalone mode STARTUP
May 13 21:57:16 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD 1.3.4d (maint) (built Wed Apr 23 2014 17:41:28 UTC) standalone mode STARTUP
May 13 23:37:52 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD 1.3.4d (maint) (built Wed Apr 23 2014 17:41:28 UTC) standalone mode STARTUP
May 14 00:07:33 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD 1.3.4d (maint) (built Wed Apr 23 2014 17:41:28 UTC) standalone mode STARTUP
May 14 00:19:26 freenas118 proftpd[2298]: 127.0.0.1 - ProFTPD 1.3.4d (maint) (built Wed Apr 23 2014 17:41:28 UTC) standalone mode STARTUP

 

[kamuzon]

What happens if you do some dd reads? Does memory utilization go up? Something like dd if=/mnt/poolname/somebigfile of=/devnull bs=1M should show your memory utilization going up.

dd: /devnull: Read-only file system
/dev# dd if=/mnt/zpool5tb/httpd-access.log of=/dev/null bs=1M
17396+1 records in
17396+1 records out
18241068212 bytes transferred in 28.387099 secs (642583034 bytes/sec)
last pid: 5522; load averages: 0.58, 0.28, 0.25 up 0+01:13:10 01:31:52 40 processes: 1 running, 39 sleeping CPU: 0.9% user, 0.0% nice, 0.4% system, 0.1% interrupt, 98.6% idle Mem: 192M Active, 97M Inact, 1278M Wired, 3664K Cache, 105M Buf, 6209M Free ARC: 831M Total, 64M MFU, 754M MRU, 144K Anon, 5417K Header, 7900K Other Swap: 10G Total, 10G Free
Mem Util doesnt go up.

AND
Then I copy 17GB httpd-access.log )) from server (not from another NAS) to justUpdatedNAS - its not reboot.

And now I'll go to copy many bzip2 files....

 

[cyberjock]

Well, I'd do a RAM test first.

Second, I'd check to see if you are using the x86 or x64 version of FreeNAS. If you are using ZFS you should only be using the x64 version. Your memory utilization makes it look like you are using the x86 version.

 

[kamuzon]

Well, I'd do a RAM test first.

How?
Maybe motherboard chipset?! Cause it reboots with preinstalled 2x2GB and with newFreshBought 4x2GB RAM.

Second, I'd check to see if you are using the x86 or x64 version of FreeNAS. If you are using ZFS you should only be using the x64 version. Your memory utilization makes it look like you are using the x86 version.

[kamuzon@freenas118] /dev# uname -a
FreeBSD freenas118... 9.2-RELEASE-p4 FreeBSD 9.2-RELEASE-p4 #0 r262572+17a4d3d: Wed Apr 23 10:11:41 PDT 2014 root@build3.ixsystems.com:/tank/home/jkh/build/9.2.1/freenas/os-base/amd64/fusion/jkh/9.2.1/freenas/FreeBSD/src/sys/FREENAS.amd64-DEBUG amd64

 

[kamuzon]

....9:39 from start copy - it is down.

UPD. 3 minutes to reboot.

 

[kamuzon]

...and it lost graphic data.