FreeNAS 11-U4 crashes regularly

[gegtor]

My FreeNAS server crashes regularly.

It freezes and then I have to do hardware restart.

It happens every 3 days today was the 4 time it crashed and I was expecting it :)

But today's crash was different because I received critical alert:

Code:

Oct. 10, 2017, 8:13 p.m. - The boot volume state is ONLINE: One or more devices has experienced an unrecoverable error. An attempt was made to correct the error. Applications are unaffected.

Everything is working just fine when i restart it but this worries me

Here is /var/log/messages before the crash and when it was started again

Code:

Oct 10 14:53:08 SKA-SERVER collectd[2901]: utils_vl_lookup: The user object callback failed with status 2.

Oct 10 14:58:49 SKA-SERVER afpd[97640]: AFP logout by PReifur

Oct 10 14:58:49 SKA-SERVER afpd[97640]: AFP statistics: 2738.50 KB read, 6067.67 KB written

Oct 10 14:58:49 SKA-SERVER afpd[97640]: done

Oct 10 15:17:48 SKA-SERVER collectd[2901]: aggregation plugin: Unable to read the current rate of "SKA-SERVER.local/cpu-3/cpu-user".

Oct 10 15:17:48 SKA-SERVER collectd[2901]: utils_vl_lookup: The user object callback failed with status 2.

Oct 10 16:33:21 SKA-SERVER afpd[76527]: afp_zzz: entering extended sleep

Oct 10 16:57:11 SKA-SERVER afpd[76527]: afp_zzz: waking up from extended sleep

Oct 10 16:59:55 SKA-SERVER afpd[76527]: afp_zzz: entering extended sleep

Oct 10 16:59:57 SKA-SERVER afpd[76527]: afp_zzz: waking up from extended sleep

Oct 10 17:01:59 SKA-SERVER afpd[76527]: afp_zzz: entering extended sleep

Oct 10 18:06:18 SKA-SERVER collectd[2901]: aggregation plugin: Unable to read the current rate of "SKA-SERVER.local/cpu-4/cpu-user".

Oct 10 18:06:18 SKA-SERVER collectd[2901]: utils_vl_lookup: The user object callback failed with status 2.

Oct 10 18:26:48 SKA-SERVER collectd[2901]: aggregation plugin: Unable to read the current rate of "SKA-SERVER.local/cpu-5/cpu-user".

Oct 10 18:26:48 SKA-SERVER collectd[2901]: utils_vl_lookup: The user object callback failed with status 2.

Oct 10 18:37:18 SKA-SERVER collectd[2901]: aggregation plugin: Unable to read the current rate of "SKA-SERVER.local/cpu-3/cpu-user".

Oct 10 18:37:18 SKA-SERVER collectd[2901]: utils_vl_lookup: The user object callback failed with status 2.

Oct 10 18:37:35 SKA-SERVER tap2: Ethernet address: 00:bd:67:18:13:02

Oct 10 18:37:35 SKA-SERVER kernel: tap2: promiscuous mode enabled

Oct 10 18:37:35 SKA-SERVER kernel: tap2: link state changed to UP

Oct 10 18:37:35 SKA-SERVER kernel: tap2: link state changed to UP

Oct 10 18:49:27 SKA-SERVER dhclient[21948]: Interface tap2 is down, dhclient exiting

Oct 10 18:49:27 SKA-SERVER dhclient[21796]: connection closed

Oct 10 18:49:27 SKA-SERVER dhclient[21796]: exiting.

Oct 10 18:49:27 SKA-SERVER kernel: tap2: link state changed to DOWN

Oct 10 18:49:27 SKA-SERVER kernel: tap2: link state changed to DOWN

Oct 10 18:49:27 SKA-SERVER tap2: Ethernet address: 00:bd:60:f4:1d:02

Oct 10 18:49:27 SKA-SERVER kernel: tap2: promiscuous mode enabled

Oct 10 18:49:27 SKA-SERVER kernel: tap2: link state changed to UP

Oct 10 18:49:27 SKA-SERVER kernel: tap2: link state changed to UP

Oct 10 19:36:08 SKA-SERVER collectd[2901]: aggregation plugin: Unable to read the current rate of "SKA-SERVER.local/cpu-3/cpu-user".

Oct 10 19:36:08 SKA-SERVER collectd[2901]: utils_vl_lookup: The user object callback failed with status 2.

Oct 10 20:13:09 SKA-SERVER syslog-ng[1538]: syslog-ng starting up; version='3.7.3'

Oct 10 20:13:09 SKA-SERVER Copyright (c) 1992-2017 The FreeBSD Project.

Oct 10 20:13:09 SKA-SERVER Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994

Oct 10 20:13:09 SKA-SERVER	 The Regents of the University of California. All rights reserved.

Oct 10 20:13:09 SKA-SERVER FreeBSD is a registered trademark of The FreeBSD Foundation.

Oct 10 20:13:09 SKA-SERVER FreeBSD 11.0-STABLE #0 r321665+25fe8ba8d06(freenas/11.0-stable): Mon Sep 25 06:24:11 UTC 2017

Oct 10 20:13:09 SKA-SERVER root@gauntlet:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64

Oct 10 20:13:09 SKA-SERVER FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)

Oct 10 20:13:09 SKA-SERVER SRAT: No memory found for CPU 0

Oct 10 20:13:09 SKA-SERVER VT(efifb): resolution 1024x768

Oct 10 20:13:09 SKA-SERVER CPU: AMD Ryzen 7 1700 Eight-Core Processor		   (3400.08-MHz K8-class CPU)

Oct 10 20:13:09 SKA-SERVER Origin="AuthenticAMD"  Id=0x800f11  Family=0x17  Model=0x1  Stepping=1

It looks okay to me.

I don't know what log files are important here so just ask me for them :)

 

[m0nkey_]

Please can you post details of your hardware, as it makes it easier to troubleshoot.

Have you run a memory test to verify the RAM is good? How about the PSU, have you tested to ensure that's also fine?

 

[siconic]

Have you made any changes?

I have actually had a weird crash every few days as well. My system was always stable as hell, never crashing and never stopping jails without me doing it. But lately, ALL the jails stop themselves, I get errors on the console scrolling as fast as the eye can see, system hangups, and an unresponsive GUI.

However, in all fairness, I have made a TON of changes over the last 2 weeks. All new disks for my pool, consolidated all my disks to be a little more "neat", changed configs, all while updating to the newest 11.0_U4. From all those changes, I found I had 2 bad disks (one would actually power down randomly, and JUST barely caught it in a log), a cron job that was creating errors because of 1.5 million lines of text, and the need for a bigger VM disk.

So, now that I have eliminated a lot of errors, bottlenecks, and hardware issues, it is down to a waiting game. I originally suspected there wass something wrong with U3 and U4, but after doing all this work to figure out the problems, it may very well have been a configs and hardware.

So any changes you made could be a cause, or even a failing disk that is not throwing any errors yet. You may need to dig into archived logs in order to find the culprit. Between the users here, and digging into logs, it took time, but thats what you gotta do. The more info the better!

 

[SweetAndLow]

Hardware problem or failing boot device.

 

[wblock]

It says right there in the first message that the boot device had a problem. It also says the processor is a Ryzen, which as far as I know is still an unknown for use with FreeNAS or FreeBSD.

 

[Ericloewe]

It says right there in the first message that the boot device had a problem.

Well, but it does say "applications are unaffected", so presumably it's mirrored and the mirror kept things going. Definitely something to check, but it's not a guaranteed cause of the problem to a decent degree.

 

[gegtor]

FreeNAS is located on internal SSD not an USB stick and also i'm using non-ECC ram

I will move OS to USB stick in the weekend and then perform some hardware checks :) because now server is in use and i can't work on it

 

[gegtor]

My system:

OS: FreeNAS 11-U4
CPU: AMD Ryzen 1700
RAM: 16GB DDR4 2400mHz GoodRam 2x sticks 8GB non-ECC
MB: MSI B350M GAMING PRO
HDD: 2x WD 1TB RED
GPU: Gigabyte Radeon RX 550
PSU: 550W corsair

 

[m0nkey_]

Ah, Ryzen.

At the moment, FreeNAS is very unpredictable on Ryzen systems. FreeBSD is about 6-12 months behind current hardware, so running on newer architectures can produce weird scenarios.

 

[wackymole]

My Ryzen build is also experiencing regular restarts.... It looks like every 5 days or so. -- 1700 Ryzen, 32 GB ECC, 2x 4TB mirror, 11.0- U4 -- Blank screen-- I guess I will schedule some restarts until resolved

 

[ykhodo]

@gegtor I stumbled upon this thread on reddit https://www.reddit.com/r/Amd/commen...faults_andor_lockups/?st=j968ndsq&sh=5d98b5cb

Basically, disable c6 in BIOS and see if it helps.

 

[gegtor]

@gegtor I stumbled upon this thread on reddit https://www.reddit.com/r/Amd/commen...faults_andor_lockups/?st=j968ndsq&sh=5d98b5cb

Basically, disable c6 in BIOS and see if it helps.

Thanks i will try this today and report if the problem is solved

 

[ykhodo]

I just had a crash. Found another thread http://forum.asrock.com/forum_posts...en-1700-on-taichi-x370-no-oc-random-hard-lock

Now I disabled global cstatus and I'll see if it makes a difference. I also read setting the voltages for the RAM could help. It could also be bad slots on the motherboard for the RAM.

 

[gpsguy]

According to the 11.1-Beta docs, "Support has been added for the Ryzen processors". There are no details, just a comment.

 

[gegtor]

But how far away is 11.1?

For example this is still unresolved and i can't run windows vm under AMD procesor
https://bugs.freenas.org/issues/25131

 

[Ericloewe]

But how far away is 11.1?

A few weeks.

 

[Afrojoe]

I too am experiencing crashing. But mine points evidently to a memory leak issue. I have 16GB ram for a 13TB RAIDZ2 volume. Should be more than enough.

The NAS is connected to three VMware Hosts presented via iSCSI. I noticed in the messages.log file there's some errors to that reguard...

When the system is idle, no issues. Once I put a load on the box, memory gets consumed (which is supposed to be normal), but typically it leaves 1GB free. Now it sucks it up right down to 400MB! and then slowly lags on, then crashes from not freeing up memory by the looks of things.

Can I have an extra pair of eyes have a look?

Any tweaking I can do so that FreeNAS doesn't suck up ALL the memory?

Hardware:

CPU: Intel(R) Xeon(R) E5620@2.40GHz
HBA: LSI 9210-8i (IT Mode)
RAM: 16GB ECC
MB: Super Micro X8DTN (Server)
HDDs: 2TB WD RED 10x (RAIDZ2)
SSDs: 2x120GB KINGSTON

zpool status

Code:

root@freenas_01:~ # zpool status
  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Mon Oct 16 03:45:17 2017
config:

  NAME  STATE  READ WRITE CKSUM
  freenas-boot  ONLINE  0  0  0
  mirror-0  ONLINE  0  0  0
  ada1p2  ONLINE  0  0  0
  ada2p2  ONLINE  0  0  0

errors: No known data errors

  pool: savage
 state: ONLINE
  scan: none requested
config:

  NAME  STATE  READ WRITE CKSUM
  savage  ONLINE  0  0  0
  raidz2-0  ONLINE  0  0  0
  gptid/cb229fda-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/cbd5ae2d-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/cc717692-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/cd0a6a77-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/d05db1be-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/d4a0dc23-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/d80616a5-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/db7243ac-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/ded96150-abd2-11e7-972d-003048f17660  ONLINE  0  0  0
  gptid/e242d7b4-abd2-11e7-972d-003048f17660  ONLINE  0  0  0

errors: No known data errors

top output

Code:

last pid: 87302;  load averages:  0.42,  0.52,  0.98  up 0+07:01:30  20:08:27
53 processes:  2 running, 51 sleeping
CPU:  0.6% user,  0.0% nice,  0.5% system,  0.1% interrupt, 98.9% idle
Mem: 71M Active, 267M Inact, 15G Wired, 412M Free
ARC: 13G Total, 504M MFU, 13G MRU, 728K Anon, 101M Header, 9548K Other
Swap: 20G Total, 310M Used, 20G Free, 1% Inuse, 20K In

  PID USERNAME  THR PRI NICE  SIZE  RES STATE  C  TIME  WCPU COMMAND
 4355 root  15  52  0  471M 88880K umtxn  2  0:05  0.00% uwsgi
  242 root  8  20  0  408M  9668K kqread  4  0:08  0.00% python3.6
 3271 root  1  20  0  407M 39784K kqread  3  0:08  0.01% uwsgi
 3259 root  1  52  0  293M 49264K select  4  0:27  0.00% python3.6
 4411 root  1  52  0  236M 43924K ttyin  1  0:02  0.00% python3.6
 3379 root  12  20  0  209M 13124K nanslp  6  1:58  0.00% collectd
 4157 root  6  52  0  104M 14252K usem  4  0:00  0.00% python3.6
 2675 root  6  28  0 89344K 10824K select 15  1:54  0.30% python3.6
 3304 root  32  43  0 84324K 16960K uwait  2  0:40  0.66% consul
87298 root  1  24  0 82848K  7236K select 11  0:00  0.00% sshd
10522 root  1  20  0 82848K  5968K select  4  0:03  0.01% sshd
87280 root  1  22  0 81500K 25908K uwait  10  0:00  0.03% dtrace
87297 root  1  36  0 81500K 25904K uwait  2  0:00  8.71% dtrace
87266 root  1  20  0 79228K 23628K CPU13  13  0:00  1.19% dtrace
 3308 root  34  20  0 65296K  7984K uwait  4  0:03  0.00% consul-alerts
 1678 root  2  20  0 61880K  2680K kqread 13  0:02  0.02% syslog-ng
 2679 root  1  20  0 60468K  6368K select  0  0:10  0.05% snmpd
 2652 root  1  20  0 57992K  4884K select  7  0:01  0.02% sshd
 4171 root  1  20  0 47864K  3184K select 13  0:00  0.00% zfsd
 1677 root  1  52  0 42132K  0K wait  5  0:00  0.00% <syslog-ng>
 4458 root  8  26  0 39708K  6704K uwait  6  0:00  0.00% consul
 4457 root  9  20  0 38044K  6840K uwait  6  0:00  0.00% consul
 3203 www  1  20  0 30980K  1544K kqread  6  0:00  0.00% nginx
 3202 root  1  52  0 30980K  0K pause  7  0:00  0.00% <nginx>
 2987 root  1  20  0 29060K  1576K nanslp  0  0:00  0.00% smartd
87302 root  1  27  0 27204K  4196K cbwait  8  0:00  0.00% smartctl
10680 root  1  20  0 26320K  3620K CPU11  11  0:23  0.12% top
 2308 root  1  20  0 24700K 12524K select 10  0:01  0.00% ntpd
87300 root  1  29  0 23848K  3020K pause  11  0:00  0.00% csh
10666 root  1  20  0 23848K  0K pause  9  0:00  0.00% <csh>
 2035 root  1  20  0 21368K  2144K select  4  0:00  0.00% ctld
 3297 messagebus  1  52  0 17488K  1936K select  1  0:00  0.00% dbus-daemon
 4181 root  1  20  0 16704K  668K nanslp  6  0:00  0.00% cron
87296 root  1  21  0 15272K  3700K wait  15  0:00  0.26% ksh93
87279 root  1  20  0 15272K  3700K wait  1  0:00  0.00% ksh93
87265 root  1  21  0 15272K  3700K wait  10  0:00  0.00% ksh93
 3014 root  1  21  0 13216K  976K wait  8  0:00  0.06% sh
 3210 nobody  1  20  0 12976K  2032K select  7  0:00  0.00% mdnsd
 4414 root  1  52  0 10440K  2024K ttyin  1  0:00  0.00% getty
 4415 root  1  52  0 10440K  2024K ttyin  11  0:00  0.00% getty
 4416 root  1  52  0 10440K  2016K ttyin  4  0:00  0.00% getty
 4418 root  1  52  0 10440K  1988K ttyin  3  0:00  0.00% getty
 4413 root  1  52  0 10440K  1988K ttyin  8  0:00  0.00% getty
 4417 root  1  52  0 10440K  1964K ttyin  14  0:00  0.00% getty
 4412 root  1  52  0 10440K  1944K ttyin  12  0:00  0.00% getty
 3013 root  1  52  0 10412K  0K sigwai  9  0:00  0.00% <daemon>
  240 root  1  52  0 10412K  0K sigwai  8  0:00  0.00% <daemon>
 3307 root  1  52  0 10412K  0K sigwai  2  0:00  0.00% <daemon>
 4156 root  1  52  0 10412K  0K sigwai 14  0:00  0.00% <daemon>
 3303 root  1  52  0 10412K  0K sigwai  0  0:00  0.00% <daemon>
 1470 root  1  20  0  9560K  136K select  2  0:00  0.00% devd
 1806 root  1 -52  r0  7600K  3572K nanslp  0  0:04  0.00% watchdogd
87295 root  1  21  0  6224K  1980K nanslp 12  0:00  0.03% sleep

/var/log/dmseg.today

Code:

Copyright (c) 1992-2017 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
  The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 11.0-STABLE #0 r321665+25fe8ba8d06(freenas/11.0-stable): Mon Sep 25 06:24:11 UTC 2017
  root@gauntlet:/freenas-11-releng/freenas/_BE/objs/freenas-11-releng/freenas/_BE/os/sys/FreeNAS.amd64 amd64
FreeBSD clang version 4.0.0 (tags/RELEASE_400/final 297347) (based on LLVM 4.0.0)
CPU: Intel(R) Xeon(R) CPU  E5620  @ 2.40GHz (2400.13-MHz K8-class CPU)
  Origin="GenuineIntel"  Id=0x206c2  Family=0x6  Model=0x2c  Stepping=2
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x9ee3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x1<LAHF>
  VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
  TSC: P-state invariant, performance statistics
real memory  = 22548578304 (21504 MB)
avail memory = 20751052800 (19789 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <102811 APIC1450>
FreeBSD/SMP: Multiprocessor System Detected: 16 CPUs
FreeBSD/SMP: 2 package(s) x 4 core(s) x 2 hardware threads
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20170303/tbfadt-748)
ioapic0: Changing APIC ID to 6
ioapic1: Changing APIC ID to 7
ioapic0 <Version 2.0> irqs 0-23 on motherboard
ioapic1 <Version 2.0> irqs 24-47 on motherboard
ioapic2 <Version 2.0> irqs 48-71 on motherboard
ioapic3 <Version 2.0> irqs 72-95 on motherboard
random: entropy device external interface
kbd1 at kbdmux0
nexus0
cryptosoft0: <software crypto> on motherboard
aesni0: No AESNI support.
padlock0: No ACE support.
acpi0: <SMCI > on motherboard
acpi0: Power Button (fixed)
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
cpu8: <ACPI CPU> on acpi0
cpu9: <ACPI CPU> on acpi0
cpu10: <ACPI CPU> on acpi0
cpu11: <ACPI CPU> on acpi0
cpu12: <ACPI CPU> on acpi0
cpu13: <ACPI CPU> on acpi0
cpu14: <ACPI CPU> on acpi0
cpu15: <ACPI CPU> on acpi0
attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Event timer "RTC" frequency 32768 Hz quality 0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 350
Event timer "HPET1" frequency 14318180 Hz quality 340
Event timer "HPET2" frequency 14318180 Hz quality 340
Event timer "HPET3" frequency 14318180 Hz quality 340
Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pcib0: _OSC returned error 0x10
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xe880-0xe89f mem 0xfbe60000-0xfbe7ffff,0xfbe40000-0xfbe5ffff,0xfbe1c000-0xfbe1ffff irq 28 at device 0.0 on pci1
igb0: Using MSIX interrupts with 9 vectors
igb0: Ethernet address: 00:30:48:f1:76:60
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
igb0: Bound queue 4 to cpu 4
igb0: Bound queue 5 to cpu 5
igb0: Bound queue 6 to cpu 6
igb0: Bound queue 7 to cpu 7
igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xec00-0xec1f mem 0xfbee0000-0xfbefffff,0xfbec0000-0xfbedffff,0xfbe9c000-0xfbe9ffff irq 40 at device 0.1 on pci1
igb1: Using MSIX interrupts with 9 vectors
igb1: Ethernet address: 00:30:48:f1:76:61
igb1: Bound queue 0 to cpu 8
igb1: Bound queue 1 to cpu 9
igb1: Bound queue 2 to cpu 10
igb1: Bound queue 3 to cpu 11
igb1: Bound queue 4 to cpu 12
igb1: Bound queue 5 to cpu 13
igb1: Bound queue 6 to cpu 14
igb1: Bound queue 7 to cpu 15
pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
pci2: <ACPI PCI bus> on pcib2
pcib3: <ACPI PCI-PCI bridge> at device 5.0 on pci0
pci3: <ACPI PCI bus> on pcib3
pcib4: <ACPI PCI-PCI bridge> at device 7.0 on pci0
pci4: <ACPI PCI bus> on pcib4
pcib5: <ACPI PCI-PCI bridge> at device 9.0 on pci0
pci5: <ACPI PCI bus> on pcib5
pcib6: <ACPI PCI-PCI bridge> at device 0.0 on pci5
pci6: <ACPI PCI bus> on pcib6
pcib7: <ACPI PCI-PCI bridge> at device 0.2 on pci5
pci7: <ACPI PCI bus> on pcib7
pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
pci0: <base peripheral, interrupt controller> at device 20.3 (no driver attached)
uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0x9400-0x941f irq 16 at device 26.0 on pci0
uhci0: LegSup = 0x2f00
usbus0 on uhci0
uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0x9480-0x949f irq 21 at device 26.1 on pci0
uhci1: LegSup = 0x2f00
usbus1 on uhci1
uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0x9800-0x981f irq 19 at device 26.2 on pci0
uhci2: LegSup = 0x2f00
usbus2 on uhci2
ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0xfb9f4000-0xfb9f43ff irq 18 at device 26.7 on pci0
usbus3: EHCI version 1.0
usbus3 on ehci0
pcib8: <ACPI PCI-PCI bridge> irq 17 at device 28.0 on pci0
pcib8: [GIANT-LOCKED]
pci8: <ACPI PCI bus> on pcib8
mps0: <Avago Technologies (LSI) SAS2008> port 0xd000-0xd0ff mem 0xfbc3c000-0xfbc3ffff,0xfbc40000-0xfbc7ffff irq 16 at device 0.0 on pci8
mps0: Firmware: 20.00.07.00, Driver: 21.01.00.00-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib9: <ACPI PCI-PCI bridge> irq 17 at device 28.4 on pci0
pcib9: [GIANT-LOCKED]
pci9: <ACPI PCI bus> on pcib9
atapci0: <JMicron JMB368 UDMA133 controller> port 0xc400-0xc407,0xcc00-0xcc03,0xc880-0xc887,0xc800-0xc803,0xc480-0xc48f irq 16 at device 0.0 on pci9
ata2: <ATA channel> at channel 0 on atapci0
uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0x9880-0x989f irq 23 at device 29.0 on pci0
uhci3: LegSup = 0x2f00
usbus4 on uhci3
uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0x9c00-0x9c1f irq 19 at device 29.1 on pci0
uhci4: LegSup = 0x2f00
usbus5 on uhci4
uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xa000-0xa01f irq 18 at device 29.2 on pci0
uhci5: LegSup = 0x2f00
usbus6 on uhci5
ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfb9f6000-0xfb9f63ff irq 23 at device 29.7 on pci0
usbus7: EHCI version 1.0
usbus7 on ehci1
pcib10: <ACPI PCI-PCI bridge> at device 30.0 on pci0
pci10: <ACPI PCI bus> on pcib10
vgapci0: <VGA-compatible display> port 0xb000-0xb0ff mem 0xd8000000-0xdfffffff,0xfbab0000-0xfbabffff irq 18 at device 1.0 on pci10
vgapci0: Boot video device
em0: <Intel(R) PRO/1000 Legacy Network Connection 1.1.0> port 0xbc00-0xbc3f mem 0xfbae0000-0xfbafffff irq 16 at device 2.0 on pci10
em0: Ethernet address: 00:30:48:9e:f6:29
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel ICH10 AHCI SATA controller> port 0xa400-0xa407,0xac00-0xac03,0xa880-0xa887,0xa800-0xa803,0xa480-0xa49f mem 0xfb9fe000-0xfb9fe7ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
acpi_button0: <Power Button> on acpi0
uart0: <16550 or compatible> port 0x3f8-0x3ff irq 4 flags 0x10 on acpi0
uart1: <16550 or compatible> port 0x2f8-0x2ff irq 3 on acpi0
qpi0: <QPI system bus> on motherboard
pcib11: <QPI Host-PCI bridge> pcibus 255 on qpi0
pci11: <PCI bus> on pcib11
pcib12: <QPI Host-PCI bridge> pcibus 254 on qpi0
pci12: <PCI bus> on pcib12
ichwd0: <Intel ICH10R watchdog timer> on isa0
wbwd0: <Winbond 83627HF/F/HG/G (0x52/0x41) Watchdog Timer> at port 0x2e-0x2f on isa0
orm0: <ISA Option ROMs> at iomem 0xc0000-0xcafff,0xce000-0xcffff,0xd6000-0xd6fff,0xd7000-0xd7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
atkbd0: <AT Keyboard> irq 1 on atkbdc0
kbd0 at atkbd0
atkbd0: [GIANT-LOCKED]
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
coretemp8: <CPU On-Die Thermal Sensors> on cpu8
est8: <Enhanced SpeedStep Frequency Control> on cpu8
coretemp9: <CPU On-Die Thermal Sensors> on cpu9
est9: <Enhanced SpeedStep Frequency Control> on cpu9
coretemp10: <CPU On-Die Thermal Sensors> on cpu10
est10: <Enhanced SpeedStep Frequency Control> on cpu10
coretemp11: <CPU On-Die Thermal Sensors> on cpu11
est11: <Enhanced SpeedStep Frequency Control> on cpu11
coretemp12: <CPU On-Die Thermal Sensors> on cpu12
est12: <Enhanced SpeedStep Frequency Control> on cpu12
coretemp13: <CPU On-Die Thermal Sensors> on cpu13
est13: <Enhanced SpeedStep Frequency Control> on cpu13
coretemp14: <CPU On-Die Thermal Sensors> on cpu14
est14: <Enhanced SpeedStep Frequency Control> on cpu14
coretemp15: <CPU On-Die Thermal Sensors> on cpu15
est15: <Enhanced SpeedStep Frequency Control> on cpu15
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
Timecounters tick every 1.000 msec
freenas_sysctl: adding account.
freenas_sysctl: adding directoryservice.
freenas_sysctl: adding middlewared.
freenas_sysctl: adding network.
freenas_sysctl: adding services.
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
nvme cam probe device init
usbus0: 12Mbps Full Speed USB v1.0
usbus1: 12Mbps Full Speed USB v1.0
usbus2: 12Mbps Full Speed USB v1.0
usbus3: 480Mbps High Speed USB v2.0
usbus4: 12Mbps Full Speed USB v1.0
usbus5: 12Mbps Full Speed USB v1.0
usbus6: 12Mbps Full Speed USB v1.0
usbus7: 480Mbps High Speed USB v2.0
ugen0.1: <Intel UHCI root HUB> at usbus0
uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
ugen3.1: <Intel EHCI root HUB> at usbus3
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
ugen2.1: <Intel UHCI root HUB> at usbus2
uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
ugen1.1: <Intel UHCI root HUB> at usbus1
uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
ugen5.1: <Intel UHCI root HUB> at usbus5
uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
ugen4.1: <Intel UHCI root HUB> at usbus4
uhub5: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
ugen7.1: <Intel EHCI root HUB> at usbus7
uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
ugen6.1: <Intel UHCI root HUB> at usbus6
uhub7: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
uhub0: 2 ports with 2 removable, self powered
uhub2: 2 ports with 2 removable, self powered
uhub4: 2 ports with 2 removable, self powered
uhub3: 2 ports with 2 removable, self powered
uhub5: 2 ports with 2 removable, self powered
uhub7: 2 ports with 2 removable, self powered
uhub1: 6 ports with 6 removable, self powered
uhub6: 6 ports with 6 removable, self powered
usb_alloc_device: set address 2 failed (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
usbd_req_re_enumerate: addr=2, set address failed! (USB_ERR_IOERROR, ignored)
usbd_setup_device_desc: getting device descriptor at addr 2 failed, USB_ERR_IOERROR
ugen3.2: <Unknown > at usbus3 (disconnected)
uhub_reattach_port: could not allocate new device
ada0 at ahcich0 bus 0 scbus2 target 0 lun 0
ada0: <WDC WD2003FYYS-02W0B1 01.01D02> ATA8-ACS SATA 2.x device
ada0: Serial Number WD-WMAY04844406
ada0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 1907729MB (3907029168 512 byte sectors)
ada1 at ahcich1 bus 0 scbus3 target 0 lun 0
ada1: <SAMSUNG MZ7PA128HMCD-010L1 AXM08L1Q> ATA8-ACS SATA 2.x device
ada1: Serial Number S0MUNEABA13792
ada1: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 122104MB (250069680 512 byte sectors)
ada1: quirks=0x3<4K,NCQ_TRIM_BROKEN>
ada2 at ahcich2 bus 0 scbus4 target 0 lun 0
ada2: <KINGSTON SA400S37120G SBFK71E0> ACS-4 ATA SATA 3.x device
ada2: Serial Number 50026B767702CBBF
ada2: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada2: Command Queueing enabled
ada2: 114473MB (234441648 512 byte sectors)
ada3 at ahcich3 bus 0 scbus5 target 0 lun 0
ada3: <WDC WD2003FZEX-00SRLA0 01.01A01> ACS-3 ATA SATA 3.x device
ada3: Serial Number WD-WMC6N0F3Y49V
ada3: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada3: Command Queueing enabled
ada3: 1907729MB (3907029168 512 byte sectors)
ada3: quirks=0x1<4K>
ada4 at ahcich4 bus 0 scbus6 target 0 lun 0
ada4: <WDC WD2003FZEX-00SRLA0 01.01A01> ACS-3 ATA SATA 3.x device
ada4: Serial Number WD-WMC6N0F7Y8E6
ada4: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada4: Command Queueing enabled
ada4: 1907729MB (3907029168 512 byte sectors)
ada4: quirks=0x1<4K>
ada5 at ahcich5 bus 0 scbus7 target 0 lun 0
ada5: <WDC WD2003FZEX-00SRLA0 01.01A01> ACS-3 ATA SATA 3.x device
ada5: Serial Number WD-WMC6N0FA2R41
ada5: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)
ada5: Command Queueing enabled
ada5: 1907729MB (3907029168 512 byte sectors)
ada5: quirks=0x1<4K>
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <SEAGATE ST32000SSSUN2.0T 0514> Fixed Direct Access SPC-3 SCSI device
da0: Serial Number 001048L2LZNW  9WM2LZNW
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 1907729MB (3907029168 512 byte sectors)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <IBM-XIV ST2000NM0001  A2 BC42> Fixed Direct Access SPC-4 SCSI device
da3: Serial Number Z1P13T940000C22706WD
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 1907729MB (3907029168 512 byte sectors)
da5 at mps0 bus 0 scbus0 target 6 lun 0
da5: <IBM-XIV HUS723020ALS64A4 J210> Fixed Direct Access SPC-4 SCSI device
da5: Serial Number YGJS66PD
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 1907729MB (3907029168 512 byte sectors)
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <IBM-ESXS ST2000NM0001 BC42> Fixed Direct Access SPC-4 SCSI device
da2: Serial Number Z1P11F3X0000C21724XT
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 1907729MB (3907029168 512 byte sectors)
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <IBM-ESXS ST32000444SS BC26> Fixed Direct Access SPC-3 SCSI device
da1: Serial Number 9WM3MCA20000C1286CSN
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 1907729MB (3907029168 512 byte sectors)
da4 at mps0 bus 0 scbus0 target 4 lun 0
da4: <IBM-ESXS ST32000444SS BC2D> Fixed Direct Access SPC-3 SCSI device
da4: Serial Number 9WM26CYL0000C1183A57
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 1907729MB (3907029168 512 byte sectors)
random: unblocking device.
SMP: AP CPU #1 Launched!
SMP: AP CPU #14 Launched!
SMP: AP CPU #8 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #13 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #9 Launched!
SMP: AP CPU #11 Launched!
SMP: AP CPU #12 Launched!
SMP: AP CPU #15 Launched!
SMP: AP CPU #10 Launched!
Timecounter "TSC-low" frequency 1200066924 Hz quality 1000
Trying to mount root from zfs:freenas-boot/ROOT/11.0-U4 []...
GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
ipmi0: <IPMI System Interface> on isa0
ipmi0: KCS mode found at io 0xca2 alignment 0x1 on isa
ipmi0: IPMI device rev. 1, firmware rev. 1.66, version 2.0
ipmi0: Number of channels 8
ipmi0: Attached watchdog
hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> UCP/8/48/0x3f8<EDG,THR,REA,WRI,INV,QUA,PRC> UCF/1/48/0x60<REA,WRI>
lagg0: link state changed to DOWN
igb0: link state changed to UP
lagg0: link state changed to UP
igb1: link state changed to UP
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device ada3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device ada4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device ada5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device da0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device da1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device da3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device da4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
GEOM_ELI: Device da5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 128
GEOM_ELI:  Crypto: software
(da2:mps0:0:2:0): READ(10). CDB: 28 00 2d 40 25 50 00 00 08 00
(da2:mps0:0:2:0): CAM status: SCSI Status Error
(da2:mps0:0:2:0): SCSI status: Check Condition
(da2:mps0:0:2:0): SCSI sense: ABORTED COMMAND asc:4b,4 (NAK received)
(da2:mps0:0:2:0): Info: 0x2d402557
(da2:mps0:0:2:0): Retrying command (per sense data)

/var/log/messages

Code:

Oct 26 13:07:59 freenas_01 GEOM_ELI: Encryption: AES-XTS 128
Oct 26 13:07:59 freenas_01 GEOM_ELI:  Crypto: software
Oct 26 13:07:59 freenas_01 GEOM_ELI: Device da0p1.eli created.
Oct 26 13:07:59 freenas_01 GEOM_ELI: Encryption: AES-XTS 128
Oct 26 13:07:59 freenas_01 GEOM_ELI:  Crypto: software
Oct 26 13:07:59 freenas_01 GEOM_ELI: Device da1p1.eli created.
Oct 26 13:07:59 freenas_01 GEOM_ELI: Encryption: AES-XTS 128
Oct 26 13:07:59 freenas_01 GEOM_ELI:  Crypto: software
Oct 26 13:07:59 freenas_01 GEOM_ELI: Device da2p1.eli created.
Oct 26 13:07:59 freenas_01 GEOM_ELI: Encryption: AES-XTS 128
Oct 26 13:07:59 freenas_01 GEOM_ELI:  Crypto: software
Oct 26 13:07:59 freenas_01 GEOM_ELI: Device da3p1.eli created.
Oct 26 13:07:59 freenas_01 GEOM_ELI: Encryption: AES-XTS 128
Oct 26 13:07:59 freenas_01 GEOM_ELI:  Crypto: software
Oct 26 13:07:59 freenas_01 GEOM_ELI: Device da4p1.eli created.
Oct 26 13:07:59 freenas_01 GEOM_ELI: Encryption: AES-XTS 128
Oct 26 13:07:59 freenas_01 GEOM_ELI:  Crypto: software
Oct 26 13:07:59 freenas_01 GEOM_ELI: Device da5p1.eli created.
Oct 26 13:07:59 freenas_01 GEOM_ELI: Encryption: AES-XTS 128
Oct 26 13:07:59 freenas_01 GEOM_ELI:  Crypto: software
Oct 26 13:08:01 freenas_01 ntpd[2307]: ntpd 4.2.8p10-a (1): Starting
Oct 26 13:08:04 freenas_01 root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Oct 26 13:08:15 freenas_01 root: /etc/rc: WARNING: failed precmd routine for minio
Oct 26 17:08:30 freenas_01 ctld[4464]: 10.0.2.12: read: Connection reset by peer
Oct 26 13:08:30 freenas_01 ctld[2035]: child process 4464 terminated with exit status 1
Oct 26 17:08:30 freenas_01 ctld[4465]: 10.0.2.14: read: Connection reset by peer
Oct 26 13:08:30 freenas_01 ctld[2035]: child process 4465 terminated with exit status 1
Oct 26 17:08:31 freenas_01 ctld[4466]: 10.0.2.8: read: Connection reset by peer
Oct 26 13:08:31 freenas_01 ctld[2035]: child process 4466 terminated with exit status 1
Oct 26 14:37:37 freenas_01 collectd[3379]: aggregation plugin: Unable to read the current rate of "freenas_01.kayna.grp/cpu-2/cpu-user".
Oct 26 14:37:37 freenas_01 collectd[3379]: utils_vl_lookup: The user object callback failed with status 2.
Oct 26 15:41:27 freenas_01 collectd[3379]: aggregation plugin: Unable to read the current rate of "freenas_01.kayna.grp/cpu-3/cpu-user".
Oct 26 15:41:27 freenas_01 collectd[3379]: utils_vl_lookup: The user object callback failed with status 2.
Oct 26 17:53:27 freenas_01 collectd[3379]: aggregation plugin: Unable to read the current rate of "freenas_01.kayna.grp/cpu-6/cpu-user".
Oct 26 17:53:27 freenas_01 collectd[3379]: utils_vl_lookup: The user object callback failed with status 2.

ifconfig

Code:

igb0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 00:30:48:f1:76:60
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
igb1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 00:30:48:f1:76:60
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect (1000baseT <full-duplex>)
  status: active
em0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
  options=209b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,WOL_MAGIC>
  ether 00:30:48:9e:f6:29
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect
  status: no carrier
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
  options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
  inet6 ::1 prefixlen 128
  inet6 fe80::1%lo0 prefixlen 64 scopeid 0x4
  inet 127.0.0.1 netmask 0xff000000
  nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
  groups: lo
lagg0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
  options=6403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
  ether 00:30:48:f1:76:60
  inet 10.0.2.40 netmask 0xffffff00 broadcast 10.0.2.255
  nd6 options=9<PERFORMNUD,IFDISABLED>
  media: Ethernet autoselect
  status: active
  groups: lagg
  laggproto lacp lagghash l2,l3,l4
  laggport: igb0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
  laggport: igb1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>

 

[gpsguy]

The other users have an AMD Ryzen CPU.

I too am experiencing crashing.

While 16GB RAM, might be adequate for general storage, with iSCSI, it's recommended to have 32GB+. See -> http://doc.freenas.org/11/intro.html#ram

You probably need a SLOG and possibly L2ARC.

And, rather than using RAIDz2, you should be running striped mirrors.

I have 16GB ram

 

[gegtor]

Hardware:

CPU: Intel(R) Xeon(R) E5620@2.40GHz
HBA: LSI 9210-8i (IT Mode)
RAM: 16GB ECC

Please create your own thread
This thread is for AMD users that have stability problems

 

[Afrojoe]

Please create your own thread
This thread is for AMD users that have stability problems

The subject does not say exclusively for AMD users. I posted because it mentioned v11-U4 which is what I was running. You may want to update your subject to reflect that fact.