Yet another Cifs is slow thread;)

[Rand]

Hi,

So i thought I'd start on of those as well;)

Specs : E3-1230v3, 16GB, in the current setup for max disk speed 8 drives in Stripe (

dd if=/dev/zero of=/mnt/test/testStripe bs=2048k count=50000
50000+0 records in
50000+0 records out
104857600000 bytes transferred in 96.693578 secs (1084431895 bytes/sec)

)
No Zil, no L2Arc
Autotune on after having the issues, tried
"smb.conf :
aio read size = 1
aio write size =1"
as well - does help but not too much.

One noteworthy thing maybe - I run FreeNas 9.2-RELEASE on a small 2GB eUSB drive which probably is not really fast - could that be an issue?

So the problem is (fairly) slow CIFS write speed, averaging around 50MB/s, sometimes spiking up to 80 but seldom.

-CIFS Read Speed is about 110MB/s, GbE limit.
-Couldnt check nfs (yet) as windows had some weird error while trying to write after connecting
-FTP Write Speed is 103 MB/s so I'd assume
network - check, array check?

For completeness -
Client 1: Win 7 64, 16Gb, i5-3330, SSD, Realtek Nic (did ftp test)
Client 2: Win8-64, 16GB, i7-27 something (laptop), SSD (Intel Nic)

I am running over 2 switches atm but will run another test from a win server in the same net in a few.
Edit:
So Client 3 connected on the same switch
Client3: W2k8R2, 8Gb, some older Quad Core Xeon, Broadcom Nic, 15kSAS drive
Read >110 MB, Write about 70-75MB (not stable per TaskManager/network but Explorer always >70)

So the question is - why can cyberjock get GbE speed out of his default CIFS installation and I can't ?;)


CJ, are you running 9.2?
If tuning Samba on FreeNas is not required b/c its not the culprit - what else might impact the performance?
*Something* has to be different...

Thanks:)

 

[cyberjock]

AUtotune shouldn't be used on your system.. disable it, delete any sysctls/tunables, and reboot.

There's lots of stuff that can be different. For starters, I'm an Intel NIC only household! You clearly aren't. :P

Your USB stick being slow won't affect CIFS performance.

Not a clue, but 8 disks doesn't tell me how big your pool is at all. 8x4TB drives will need more than 16GB of RAM....

 

[Rand]

Autotune is gone, Speed has gotten worse - large VMWare File -> between 15 and 60 MB, massive fluctuation (in the one large file);
FTP max 90MB/s

8disks , 3TB drives; just did that to have maximum speed for testing, final layout is either 1x7 Z3 or 4x2 mirror (original plan), depending on what has better write speed.

Whats the correlation between poolsize and ram requirement? Is there agood read on that?

 

[cyberjock]

What are you using the pool for? iscsi? NFS? CIFS? ESXI datastore?

Post the output of dmesg, zpool status, and zpool list.

 

[Rand]

This particular pool is planned to be used with CIFS as main storage around the house.

dmesg:

Code:

dmesg
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.2-RELEASE #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013
    root@build.ixsystems.com:/tank/home/jkh/checkout/freenas/os-base/amd64/tank/home/jkh/checkout/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz (3292.44-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x306c3  Family = 0x6  Model = 0x3c  Stepping = 3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,<b11>,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Standard Extended Features=0x2fbb<GSFSBASE,TSCADJ,SMEP,ENHMOVSB,INVPCID>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16452767744 (15690 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  DENLOW>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP): APIC ID:  3
cpu4 (AP): APIC ID:  4
cpu5 (AP): APIC ID:  5
cpu6 (AP): APIC ID:  6
cpu7 (AP): APIC ID:  7
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
acpi0: <INTEL DENLOW> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of ff000000, 1000fff (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 18 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mps0: <LSI SAS2008> port 0x6000-0x60ff mem 0xc1840000-0xc184ffff,0xc1800000-0xc183ffff irq 16 at device 0.0 on pci1
mps0: Firmware: 07.15.08.00, Driver: 14.00.00.02-fbsd
mps0: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
mps1: <LSI SAS2008> port 0x5000-0x50ff mem 0xc1640000-0xc164ffff,0xc1600000-0xc163ffff irq 17 at device 0.0 on pci2
mps1: Firmware: 07.15.08.00, Driver: 14.00.00.02-fbsd
mps1: IOCCapabilities: 185c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,IR>
pci0: <serial bus, USB> at device 20.0 (no driver attached)
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xc1940000-0xc19403ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci3: <ACPI PCI bus> on pcib3
vgapci0: <VGA-compatible display> mem 0xc2000000-0xc2ffffff,0xc1010000-0xc1013fff,0xc0800000-0xc0ffffff irq 16 at device 0.0 on pci3
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci4: <ACPI PCI bus> on pcib4
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x4000-0x401f mem 0xc1300000-0xc13fffff,0xc1400000-0xc1403fff irq 17 at device 0.0 on pci4
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:1e:67:58:fa:f3
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib5: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci5: <ACPI PCI bus> on pcib5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x3000-0x301f mem 0xc1100000-0xc11fffff,0xc1200000-0xc1203fff irq 18 at device 0.0 on pci5
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 00:1e:67:58:fa:f4
igb1: Bound queue 0 to cpu 4
igb1: Bound queue 1 to cpu 5
igb1: Bound queue 2 to cpu 6
igb1: Bound queue 3 to cpu 7
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xc1930000-0xc19303ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Lynx Point AHCI SATA controller> port 0x7070-0x7077,0x7060-0x7063,0x7050-0x7057,0x7040-0x7043,0x7020-0x703f mem 0xc1920000-0xc19207ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pci0: <dasp> at device 31.6 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
battery1: <ACPI Control Method Battery> on acpi0
battery2: <ACPI Control Method Battery> on acpi0
ppc1: cannot reserve I/O port range
orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc87ff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0xfffffe0008345d80 with IPv6 initialized (100409)
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub0: 2 ports with 2 removable, self powered
uhub1: 2 ports with 2 removable, self powered
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 110 completed ioc 8047 scsi 0 state 4 xfer 0
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 110 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 115 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 115 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 120 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
ugen0.2: <vendor 0x8087> at usbus0
uhub2: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.04, addr 2> on usbus0
ugen1.2: <vendor 0x8087> at usbus1
uhub3: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.04, addr 2> on usbus1
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 116 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 121 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 121 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 126 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Error 5, Retries exhausted
uhub2: 4 ports with 4 removable, self powered
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 122 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 127 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): Error 5, Retries exhausted
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
uhub3: 6 ports with 6 removable, self powered
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 127 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 136 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 128 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 137 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 137 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 138 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Retrying command
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 138 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 143 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): Retrying command
(probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe3:mps0:0:3:0): CAM status: CCB request completed with an error
(probe3:mps0:0:3:0): Error 5, Retries exhausted
ugen0.3: <American Megatrends Inc.> at usbus0
ukbd0: <Keyboard Interface> on usbus0
kbd0 at ukbd0
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 139 completed ioc 8047 scsi 0 state 4 xfer 0
(probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
(probe129:mps1:0:3:0): CAM status: CCB request completed with an error
(probe129:mps1:0:3:0): Error 5, Retries exhausted
ada0 at ahcich1 bus 0 scbus3 target 0 lun 0
ada0: <INTEL SSDSC2MH120A2 PPG4> ATA-8 SATA 3.x device
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad6
ada1 at ahcich3 bus 0 scbus5 target 0 lun 0
ada1: <KINGSTON SS100S216G 110512> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 15272MB (31277232 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad10
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
Timecounter "TSC-low" frequency 1646218782 Hz quality 1000
da1 at mps0 bus 0 scbus0 target 5 lun 0
da1: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da2 at mps0 bus 0 scbus0 target 6 lun 0
da2: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da0 at mps0 bus 0 scbus0 target 4 lun 0
da0: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da3 at mps0 bus 0 scbus0 target 7 lun 0
da3: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da5 at mps1 bus 0 scbus1 target 23 lun 0
da5: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da7 at mps1 bus 0 scbus1 target 25 lun 0
da7: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da4 at mps1 bus 0 scbus1 target 22 lun 0
da4: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da6 at mps1 bus 0 scbus1 target 24 lun 0
da6: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-5 device
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
ugen1.3: <vendor 0x0624> at usbus1
ukbd1: <vendor 0x0624 Virtual Media KVM Module, class 0/0, rev 2.00/1.00, addr 3> on usbus1
kbd2 at ukbd1
ums1: <vendor 0x0624 Virtual Media KVM Module, class 0/0, rev 2.00/1.00, addr 3> on usbus1
ums1: 5 buttons and [XYZ] coordinates ID=1
ums1: 5 buttons and [Z] coordinates ID=4
umass0: <vendor 0x0624 Virtual Media KVM Module, class 0/0, rev 2.00/1.00, addr 3> on usbus1
da8 at umass-sim0 bus 0 scbus9 target 0 lun 0
da8: <KVM vmDisk 0.01> Removable Direct Access SCSI-0 device
da8: 40.000MB/s transfers
da8: Attempt to query device size failed: NOT READY, Medium not present
da8: quirks=0x2<NO_6_BYTE>
ugen0.4: <Intel> at usbus0
umass1: <Intel Value SSD, class 0/0, rev 2.00/2.00, addr 4> on usbus0
da9 at umass-sim1 bus 1 scbus10 target 0 lun 0
da9: <Intel Value SSD 2.00> Fixed Direct Access SCSI-2 device
da9: 40.000MB/s transfers
da9: 1966MB (4026368 512 byte sectors: 255H 63S/T 250C)
da9: quirks=0x2<NO_6_BYTE>
Trying to mount root from ufs:/dev/ufs/FreeNASs1a [ro]...
GEOM_RAID5: Module loaded, version 1.1.20130907.44 (rev 5c6d2a159411)
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM_ELI: Device ada0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware

zpool status
pool: test
state: ONLINE
scan: none requested
config:

NAME STATE READ WRITE CKSUM
test ONLINE 0 0 0
gptid/6418a891-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0
gptid/66d345c6-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0
gptid/699dad92-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0
gptid/6c782039-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0
gptid/6f5622ec-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0
gptid/721bd4fd-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0
gptid/74ef8012-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0
gptid/77e6983d-7d52-11e3-9cec-001e6758faf3 ONLINE 0 0 0

errors: No known data errors


zpool list
NAME SIZE ALLOC FREE CAP DEDUP HEALTH ALTROOT
SSD2 12.9G 812K 12.9G 0% 1.00x ONLINE /mnt
SSD510 109G 97.7G 11.3G 89% 1.00x ONLINE /mnt
test 21.8T 176G 21.6T 0% 1.00x ONLINE /mnt

I omitted the SSD test pools in the status output

 

[cyberjock]

Uhhh.. you have a hardware problem....

1. (probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 110 completed ioc 8047 scsi 0 state 4 xfer 0
2. (probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
3. (probe3:mps0:0:3:0): CAM status: CCB request completed with an error
4. (probe3:mps0:0:3:0): Retrying command
5. (probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 110 completed ioc 8047 scsi 0 state 4 xfer 0
6. (probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
7. (probe129:mps1:0:3:0): CAM status: CCB request completed with an error
8. (probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 115 completed ioc 8047 scsi 0 state 4 xfer 0
9. (probe129:mps1:0:3:0): Retrying command
10. (probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
11. (probe3:mps0:0:3:0): CAM status: CCB request completed with an error
12. (probe3:mps0:0:3:0): Retrying command
13. (probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 115 completed ioc 8047 scsi 0 state 4 xfer 0
14. (probe129:mps1:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
15. (probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00 length 36 SMID 120 completed ioc 8047 scsi 0 state 4 xfer 0
16. (probe129:mps1:0:3:0): CAM status: CCB request completed with an error
17. (probe129:mps1:0:3:0): Retrying command
18. (probe3:mps0:0:3:0): INQUIRY. CDB: 12 00 00 00 24 00
19. (probe3:mps0:0:3:0): CAM status: CCB request completed with an error
20. (probe3:mps0:0:3:0): Retrying command

Those.. those are "bad things".

Got lots of them too. Might want to figure out what is causing errors with mps0 and mps1 before you go further...

 

[Rand]

Yes i've just seen them too;
not entirely sure what they are b/c its on both hbas...
All attached devices are available though, so I'll have to dig what those are...

 

[cyberjock]

Your HBAs might not be 100% compatible with FreeNAS. What are they? What firmware version?

 

[Rand]

Yes thats what i was thinking (that 03 is the hba sas address) looking for a way to verify that.

And those are Dell H310 flashed to IT mode, LSI 2008 based (rebranded LSI-9211-8i).
One is running
Dell
** 6GBPSAS Adapter with:
** Package version -------------- 07.03.05.00
** Firmware Version ------------- 07.15.08.00
** Option BIOS Version ---------- 7.11.10.00-Dellized_24
** Boot services Driver Version - 7.02.03.00
** NVDATA Version --------------- Version 19
**
(with Bios for staggered spin up config)
the other on P15 i think, not sure when i actually flashed it, just remember I couldnt update to latest version

 

[Rand]

All right, took me a while but now I've upgraded both cards to Phase 17 and that error is gone.

Code:

root@nas2] ~# dmesg
Copyright (c) 1992-2013 The FreeBSD Project.
Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
        The Regents of the University of California. All rights reserved.
FreeBSD is a registered trademark of The FreeBSD Foundation.
FreeBSD 9.2-RELEASE #0 r+2315ea3: Fri Dec 20 12:48:50 PST 2013
    root@build.ixsystems.com:/tank/home/jkh/checkout/freenas/os-base/amd64/tank/home/jkh/checkout/freenas/FreeBSD/src/sys/FREENAS.amd64 amd64
gcc version 4.2.1 20070831 patched [FreeBSD]
CPU: Intel(R) Xeon(R) CPU E3-1230 v3 @ 3.30GHz (3292.44-MHz K8-class CPU)
  Origin = "GenuineIntel"  Id = 0x306c3  Family = 0x6  Model = 0x3c  Stepping = 3
  Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
  Features2=0x7ffafbff<SSE3,PCLMULQDQ,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,<b11>,FMA,CX16,xTPR,PDCM,PCID,SSE4.1,SSE4.2,x2APIC,MOVBE,POPCNT,TSCDLT,AESNI,XSAVE,OSXSAVE,AVX,F16C,RDRAND>
  AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
  AMD Features2=0x21<LAHF,ABM>
  Standard Extended Features=0x2fbb<GSFSBASE,TSCADJ,SMEP,ENHMOVSB,INVPCID>
  TSC: P-state invariant, performance statistics
real memory  = 17179869184 (16384 MB)
avail memory = 16452763648 (15690 MB)
Event timer "LAPIC" quality 600
ACPI APIC Table: <INTEL  DENLOW>
FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
cpu0 (BSP): APIC ID:  0
cpu1 (AP): APIC ID:  1
cpu2 (AP): APIC ID:  2
cpu3 (AP): APIC ID:  3
cpu4 (AP): APIC ID:  4
cpu5 (AP): APIC ID:  5
cpu6 (AP): APIC ID:  6
cpu7 (AP): APIC ID:  7
WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
ioapic0 <Version 2.0> irqs 0-23 on motherboard
kbd1 at kbdmux0
cryptosoft0: <software crypto> on motherboard
aesni0: <AES-CBC,AES-XTS> on motherboard
padlock0: No ACE support.
acpi0: <INTEL DENLOW> on motherboard
acpi0: Power Button (fixed)
acpi0: reservation of ff000000, 1000fff (3) failed
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
cpu2: <ACPI CPU> on acpi0
cpu3: <ACPI CPU> on acpi0
cpu4: <ACPI CPU> on acpi0
cpu5: <ACPI CPU> on acpi0
cpu6: <ACPI CPU> on acpi0
cpu7: <ACPI CPU> on acpi0
hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Timecounter "HPET" frequency 14318180 Hz quality 950
Event timer "HPET" frequency 14318180 Hz quality 550
atrtc0: <AT realtime clock> port 0x70-0x77 irq 8 on acpi0
atrtc0: Warning: Couldn't map I/O.
Event timer "RTC" frequency 32768 Hz quality 0
attimer0: <AT timer> port 0x40-0x43,0x50-0x53 irq 0 on acpi0
Timecounter "i8254" frequency 1193182 Hz quality 0
Event timer "i8254" frequency 1193182 Hz quality 100
Timecounter "ACPI-fast" frequency 3579545 Hz quality 900
acpi_timer0: <24-bit timer at 3.579545MHz> port 0x408-0x40b on acpi0
pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
pci0: <ACPI PCI bus> on pcib0
pcib1: <ACPI PCI-PCI bridge> irq 18 at device 1.0 on pci0
pci1: <ACPI PCI bus> on pcib1
mps0: <LSI SAS2008> port 0x6000-0x60ff mem 0xc1840000-0xc184ffff,0xc1800000-0xc183ffff irq 16 at device 0.0 on pci1
mps0: Firmware: 17.00.01.00, Driver: 14.00.00.02-fbsd
mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pcib2: <ACPI PCI-PCI bridge> irq 18 at device 1.1 on pci0
pci2: <ACPI PCI bus> on pcib2
mps1: <LSI SAS2008> port 0x5000-0x50ff mem 0xc1640000-0xc164ffff,0xc1600000-0xc163ffff irq 17 at device 0.0 on pci2
mps1: Firmware: 17.00.01.00, Driver: 14.00.00.02-fbsd
mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
pci0: <serial bus, USB> at device 20.0 (no driver attached)
ehci0: <EHCI (generic) USB 2.0 controller> mem 0xc1940000-0xc19403ff irq 16 at device 26.0 on pci0
usbus0: EHCI version 1.0
usbus0 on ehci0
pcib3: <ACPI PCI-PCI bridge> irq 16 at device 28.0 on pci0
pci3: <ACPI PCI bus> on pcib3
vgapci0: <VGA-compatible display> mem 0xc2000000-0xc2ffffff,0xc1010000-0xc1013fff,0xc0800000-0xc0ffffff irq 16 at device 0.0 on pci3
pcib4: <ACPI PCI-PCI bridge> irq 17 at device 28.1 on pci0
pci4: <ACPI PCI bus> on pcib4
igb0: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x4000-0x401f mem 0xc1300000-0xc13fffff,0xc1400000-0xc1403fff irq 17 at device 0.0 on pci4
igb0: Using MSIX interrupts with 5 vectors
igb0: Ethernet address: 00:1e:67:58:fa:f3
igb0: Bound queue 0 to cpu 0
igb0: Bound queue 1 to cpu 1
igb0: Bound queue 2 to cpu 2
igb0: Bound queue 3 to cpu 3
pcib5: <ACPI PCI-PCI bridge> irq 18 at device 28.2 on pci0
pci5: <ACPI PCI bus> on pcib5
igb1: <Intel(R) PRO/1000 Network Connection version - 2.4.0> port 0x3000-0x301f mem 0xc1100000-0xc11fffff,0xc1200000-0xc1203fff irq 18 at device 0.0 on pci5
igb1: Using MSIX interrupts with 5 vectors
igb1: Ethernet address: 00:1e:67:58:fa:f4
igb1: Bound queue 0 to cpu 4
igb1: Bound queue 1 to cpu 5
igb1: Bound queue 2 to cpu 6
igb1: Bound queue 3 to cpu 7
ehci1: <EHCI (generic) USB 2.0 controller> mem 0xc1930000-0xc19303ff irq 23 at device 29.0 on pci0
usbus1: EHCI version 1.0
usbus1 on ehci1
isab0: <PCI-ISA bridge> at device 31.0 on pci0
isa0: <ISA bus> on isab0
ahci0: <Intel Lynx Point AHCI SATA controller> port 0x7070-0x7077,0x7060-0x7063,0x7050-0x7057,0x7040-0x7043,0x7020-0x703f mem 0xc1920000-0xc19207ff irq 19 at device 31.2 on pci0
ahci0: AHCI v1.30 with 6 6Gbps ports, Port Multiplier not supported
ahcich0: <AHCI channel> at channel 0 on ahci0
ahcich1: <AHCI channel> at channel 1 on ahci0
ahcich2: <AHCI channel> at channel 2 on ahci0
ahcich3: <AHCI channel> at channel 3 on ahci0
ahcich4: <AHCI channel> at channel 4 on ahci0
ahcich5: <AHCI channel> at channel 5 on ahci0
pci0: <serial bus, SMBus> at device 31.3 (no driver attached)
pci0: <dasp> at device 31.6 (no driver attached)
acpi_tz0: <Thermal Zone> on acpi0
acpi_tz1: <Thermal Zone> on acpi0
battery0: <ACPI Control Method Battery> on acpi0
battery1: <ACPI Control Method Battery> on acpi0
battery2: <ACPI Control Method Battery> on acpi0
ppc1: cannot reserve I/O port range
orm0: <ISA Option ROM> at iomem 0xc0000-0xc7fff on isa0
sc0: <System console> at flags 0x100 on isa0
sc0: VGA <16 virtual consoles, flags=0x300>
vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
ppc0: cannot reserve I/O port range
wbwd0: HEFRAS and EFER do not align: EFER 0x2e DevID 0xff DevRev 0xff CR26 0xff
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
est0: <Enhanced SpeedStep Frequency Control> on cpu0
p4tcc0: <CPU Frequency Thermal Control> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
est1: <Enhanced SpeedStep Frequency Control> on cpu1
p4tcc1: <CPU Frequency Thermal Control> on cpu1
coretemp2: <CPU On-Die Thermal Sensors> on cpu2
est2: <Enhanced SpeedStep Frequency Control> on cpu2
p4tcc2: <CPU Frequency Thermal Control> on cpu2
coretemp3: <CPU On-Die Thermal Sensors> on cpu3
est3: <Enhanced SpeedStep Frequency Control> on cpu3
p4tcc3: <CPU Frequency Thermal Control> on cpu3
coretemp4: <CPU On-Die Thermal Sensors> on cpu4
est4: <Enhanced SpeedStep Frequency Control> on cpu4
p4tcc4: <CPU Frequency Thermal Control> on cpu4
coretemp5: <CPU On-Die Thermal Sensors> on cpu5
est5: <Enhanced SpeedStep Frequency Control> on cpu5
p4tcc5: <CPU Frequency Thermal Control> on cpu5
coretemp6: <CPU On-Die Thermal Sensors> on cpu6
est6: <Enhanced SpeedStep Frequency Control> on cpu6
p4tcc6: <CPU Frequency Thermal Control> on cpu6
coretemp7: <CPU On-Die Thermal Sensors> on cpu7
est7: <Enhanced SpeedStep Frequency Control> on cpu7
p4tcc7: <CPU Frequency Thermal Control> on cpu7
Timecounters tick every 1.000 msec
ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
DUMMYNET 0xfffffe0008345d80 with IPv6 initialized (100409)
load_dn_sched dn_sched QFQ loaded
load_dn_sched dn_sched RR loaded
load_dn_sched dn_sched WF2Q+ loaded
load_dn_sched dn_sched FIFO loaded
load_dn_sched dn_sched PRIO loaded
usbus0: 480Mbps High Speed USB v2.0
usbus1: 480Mbps High Speed USB v2.0
ugen0.1: <Intel> at usbus0
uhub0: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus0
ugen1.1: <Intel> at usbus1
uhub1: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus1
uhub1: 2 ports with 2 removable, self powered
uhub0: 2 ports with 2 removable, self powered
ada0 at ahcich1 bus 0 scbus3 target 0 lun 0
ada0: <INTEL SSDSC2MH120A2 PPG4> ATA-8 SATA 3.x device
ada0: 600.000MB/s transfers (SATA 3.x, UDMA6, PIO 8192bytes)
ada0: Command Queueing enabled
ada0: 114473MB (234441648 512 byte sectors: 16H 63S/T 16383C)
ada0: quirks=0x1<4K>
ada0: Previously was known as ad6
ada1 at ahcich3 bus 0 scbus5 target 0 lun 0
ada1: <KINGSTON SS100S216G 110512> ATA-8 SATA 2.x device
ada1: 300.000MB/s transfers (SATA 2.x, UDMA5, PIO 8192bytes)
ada1: Command Queueing enabled
ada1: 15272MB (31277232 512 byte sectors: 16H 63S/T 16383C)
ada1: Previously was known as ad10
da8 at mps1 bus 0 scbus1 target 8 lun 0
da8: <HITACHI HUC10606 CLAR600 C330> Fixed Direct Access SCSI-6 device
da8: 600.000MB/s transfers
da8: Command Queueing enabled
da8: 572204MB (1171875000 512 byte sectors: 255H 63S/T 72945C)
da0 at mps0 bus 0 scbus0 target 0 lun 0
da0: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da0: 600.000MB/s transfers
da0: Command Queueing enabled
da0: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da1 at mps0 bus 0 scbus0 target 1 lun 0
da1: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da1: 600.000MB/s transfers
da1: Command Queueing enabled
da1: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da2 at mps0 bus 0 scbus0 target 2 lun 0
da2: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da2: 600.000MB/s transfers
da2: Command Queueing enabled
da2: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da3 at mps0 bus 0 scbus0 target 3 lun 0
da3: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da3: 600.000MB/s transfers
da3: Command Queueing enabled
da3: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da9 at mps1 bus 0 scbus1 target 9 lun 0
da9: <HITACHI HUC10606 CLAR600 C330> Fixed Direct Access SCSI-6 device
da9: 600.000MB/s transfers
da9: Command Queueing enabled
da9: 572204MB (1171875000 512 byte sectors: 255H 63S/T 72945C)
SMP: AP CPU #1 Launched!
SMP: AP CPU #5 Launched!
SMP: AP CPU #3 Launched!
SMP: AP CPU #7 Launched!
SMP: AP CPU #4 Launched!
SMP: AP CPU #6 Launched!
SMP: AP CPU #2 Launched!
Timecounter "TSC-low" frequency 1646221258 Hz quality 1000
da4 at mps1 bus 0 scbus1 target 0 lun 0
da4: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da4: 600.000MB/s transfers
da4: Command Queueing enabled
da4: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da6 at mps1 bus 0 scbus1 target 2 lun 0
da6: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da6: 600.000MB/s transfers
da6: Command Queueing enabled
da6: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da5 at mps1 bus 0 scbus1 target 1 lun 0
da5: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da5: 600.000MB/s transfers
da5: Command Queueing enabled
da5: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
da7 at mps1 bus 0 scbus1 target 3 lun 0
da7: <ATA TOSHIBA DT01ABA3 ABB0> Fixed Direct Access SCSI-6 device
da7: 600.000MB/s transfers
da7: Command Queueing enabled
da7: 2861588MB (5860533168 512 byte sectors: 255H 63S/T 364801C)
ugen1.2: <vendor 0x8087> at usbus1
uhub2: <vendor 0x8087 product 0x8000, class 9/0, rev 2.00/0.04, addr 2> on usbus1
ugen0.2: <vendor 0x8087> at usbus0
uhub3: <vendor 0x8087 product 0x8008, class 9/0, rev 2.00/0.04, addr 2> on usbus0
uhub3: 4 ports with 4 removable, self powered
uhub2: 6 ports with 6 removable, self powered
ugen0.3: <American Megatrends Inc.> at usbus0
ukbd0: <Keyboard Interface> on usbus0
kbd0 at ukbd0
ums0: <Mouse Interface> on usbus0
ums0: 3 buttons and [Z] coordinates ID=0
ugen1.3: <Avocent> at usbus1
ukbd1: <Avocent Virtual Media KVM Module, class 0/0, rev 2.00/1.00, addr 3> on usbus1
kbd2 at ukbd1
ums1: <Avocent Virtual Media KVM Module, class 0/0, rev 2.00/1.00, addr 3> on usbus1
ums1: 5 buttons and [XYZ] coordinates ID=1
ums1: 5 buttons and [Z] coordinates ID=4
umass0: <Avocent Virtual Media KVM Module, class 0/0, rev 2.00/1.00, addr 3> on usbus1
cd0 at umass-sim0 bus 0 scbus9 target 0 lun 0
cd0: <KVM vmDisk-CD 0.01> Removable CD-ROM SCSI-0 device
cd0: 40.000MB/s transfers
cd0: Attempt to query device size failed: NOT READY, Medium not present
cd0: quirks=0x10<10_BYTE_ONLY>
ugen0.4: <Intel> at usbus0
umass1: <Intel Value SSD, class 0/0, rev 2.00/2.00, addr 4> on usbus0
da10 at umass-sim1 bus 1 scbus10 target 0 lun 0
da10: <Intel Value SSD 2.00> Fixed Direct Access SCSI-2 device
da10: 40.000MB/s transfers
da10: 1966MB (4026368 512 byte sectors: 255H 63S/T 250C)
da10: quirks=0x2<NO_6_BYTE>
Trying to mount root from ufs:/dev/ufs/FreeNASs1a [ro]...
GEOM_RAID5: Module loaded, version 1.1.20130907.44 (rev 5c6d2a159411)
ZFS filesystem version: 5
ZFS storage pool version: features support (5000)
GEOM_ELI: Device da5p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da6p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da7p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da2p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da3p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da8p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device ada1p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da4p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da9p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware
GEOM_ELI: Device da0p1.eli created.
GEOM_ELI: Encryption: AES-XTS 256
GEOM_ELI:    Crypto: hardware

Do you think the driver version would be relevant?
"mps0: Firmware: 17.00.01.00, Driver: 14.00.00.02-fbsd"

But array performance is ok after all... recreated the 8 drive array, around 1050 MB write, 950 read (dd);
Cifs performance is still bad on write (max 60MB, even with multiple copies at once to possibly circumvent the single thread issue),
cifs read is good, >100MB

So - whats the difference between Cifs and say ftp on the write side?
Why would the pool size have an impact (as you indicated)?

Thanks:)

 

[cyberjock]

Multiple CIFS copies to cirvumvent the single thread issue doesn't do what you want and only makes everything slower.

Your firmware version should match the driver version, but people using v17 firmware haven't had problems.

 

[Rand]

Yes i thought so (multiple cifs writes), multiple write requests and head repositioning and so on;) Just thought i'd give it a try; but cpu never even broke a sweat so thats not a limit (3-4% usage)

 

[cyberjock]

Not to single you out or anything, but I've really given up on the "CIFS is slow" threads because they just aren't worth the amount of time I put into it. So you are kind-of on your own unless you want my consulation services or someone else here wants to help you out otherwise. I've spent far too many hours on these problems and they are always the result of:

1. improper/bad hardware
2. improper tweaks/settings
3. Choosing to use those stupid 3rd party programs to copy datas.

Sorry. I've just recently had a "come to god" moment where I'm finding I spent way to much time helping others(in excess of 50-60 hours a week) and as an unpaid volunteer I've made the choice to seriously cut back because that's just crazy.

 

[Rand]

Hey,
dont worry its not a problem:)

Bad for me but a good decision for you i think:)

Edit:
Almost forgot: Thanks for your help :)

 

[Rand]

So, to conclude this ...
I've tried playing around with smb settings and smaller drive setups to negate any "too large a disk/too little ram" problems which might have been; some other stuff with switches and windows but nothing major;
checked the current Beta too but nothing either.
Then I've installed Omni OS with Napp-It ... 4 Disk Mirror (300MB dd) maxes out GbEthernet out of the box...
Why? (Rethorical question i guess)

Thanks and good luck with *your* problem if your reading this;)

 

[Rand]

Just to conclude this really -
By now the following changes have been implemented and performance is very good.
Another thread
-Upgraded to 9.2.1. Release (unknown but large improvements promised)
-Upgraded Firmware to be in line with 9.2.1 Release (p16) (3-5MB/s)
-Changed a 'green' setting in my switch (seemingly removing the large variation i was experiencing in the thread above)
-Added an Intel Nic to the Desktop (~10-20MB/s)
-Disable HyperThreading as per this thread
I assume that will be fixed but didnt have any extreme influence for me (couple of mb/s), probably the E3-1230 is not *that* fast (as the 1270)

Added LACP to the system (2x1GB), and only 4 disks in a mirror config already saturate both links.

Going down to a single mirror disk -> Performance is at expected level of about 140MB/s (single disk write performance in dd) over 2 connections