10 Gig Networking Primer

[helloha]

Does anyone know if specific firmware versions of Chelsio S310E-CR cards are better supported then others?

I'm having some issues where network speed seems to come to a complete stop from time to time...

I've added my log messages below. I'm getting some errors but don't know if they are related?

K.

Apr 21 12:24:00 SERVER syslog-ng[1714]: syslog-ng starting up; version='3.6.4'
Apr 21 12:24:00 SERVER Copyright (c) 1992-2016 The FreeBSD Project.
Apr 21 12:24:00 SERVER Copyright (c) 1979, 1980, 1983, 1986, 1988, 1989, 1991, 1992, 1993, 1994
Apr 21 12:24:00 SERVER The Regents of the University of California. All rights reserved.
Apr 21 12:24:00 SERVER FreeBSD is a registered trademark of The FreeBSD Foundation.
Apr 21 12:24:00 SERVER FreeBSD 10.3-RELEASE #0 1b72f49(freebsd10): Mon Apr 11 10:51:27 PDT 2016
Apr 21 12:24:00 SERVER root@build.ixsystems.com:/tank/home/nightlies/build-freenas9/_BE/objs/tank/home/nightlies/build-freenas9/_BE/trueos/sys/FreeNAS.amd64 amd64
Apr 21 12:24:00 SERVER FreeBSD clang version 3.4.1 (tags/RELEASE_34/dot1-final 208032) 20140512
Apr 21 12:24:00 SERVER CPU: Intel(R) Xeon(R) CPU L5630 @ 2.13GHz (2128.05-MHz K8-class CPU)
Apr 21 12:24:00 SERVER Origin="GenuineIntel" Id=0x206c2 Family=0x6 Model=0x2c Stepping=2
Apr 21 12:24:00 SERVER Features=0xbfebfbff<FPU,VME,DE,PSE,TSC,MSR,PAE,MCE,CX8,APIC,SEP,MTRR,PGE,MCA,CMOV,PAT,PSE36,CLFLUSH,DTS,ACPI,MMX,FXSR,SSE,SSE2,SS,HTT,TM,PBE>
Apr 21 12:24:00 SERVER Features2=0x9ee3fd<SSE3,DTES64,MON,DS_CPL,VMX,SMX,EST,TM2,SSSE3,CX16,xTPR,PDCM,PCID,DCA,SSE4.1,SSE4.2,POPCNT>
Apr 21 12:24:00 SERVER AMD Features=0x2c100800<SYSCALL,NX,Page1GB,RDTSCP,LM>
Apr 21 12:24:00 SERVER AMD Features2=0x1<LAHF>
Apr 21 12:24:00 SERVER VT-x: PAT,HLT,MTF,PAUSE,EPT,UG,VPID
Apr 21 12:24:00 SERVER TSC: P-state invariant, performance statistics
Apr 21 12:24:00 SERVER real memory = 52613349376 (50176 MB)
Apr 21 12:24:00 SERVER avail memory = 49989259264 (47673 MB)
Apr 21 12:24:00 SERVER Event timer "LAPIC" quality 600
Apr 21 12:24:00 SERVER ACPI APIC Table: <031712 APIC1550>
Apr 21 12:24:00 SERVER FreeBSD/SMP: Multiprocessor System Detected: 8 CPUs
Apr 21 12:24:00 SERVER FreeBSD/SMP: 1 package(s) x 4 core(s) x 2 SMT threads
Apr 21 12:24:00 SERVER cpu0 (BSP): APIC ID: 0
Apr 21 12:24:00 SERVER cpu1 (AP): APIC ID: 1
Apr 21 12:24:00 SERVER cpu2 (AP): APIC ID: 2
Apr 21 12:24:00 SERVER cpu3 (AP): APIC ID: 3
Apr 21 12:24:00 SERVER cpu4 (AP): APIC ID: 18
Apr 21 12:24:00 SERVER cpu5 (AP): APIC ID: 19
Apr 21 12:24:00 SERVER cpu6 (AP): APIC ID: 20
Apr 21 12:24:00 SERVER cpu7 (AP): APIC ID: 21
Apr 21 12:24:00 SERVER random: <Software, Yarrow> initialized
Apr 21 12:24:00 SERVER WARNING: VIMAGE (virtualized network stack) is a highly experimental feature.
Apr 21 12:24:00 SERVER ACPI BIOS Warning (bug): 32/64X length mismatch in FADT/Gpe0Block: 128/64 (20150515/tbfadt-644)
Apr 21 12:24:00 SERVER ioapic0: Changing APIC ID to 6
Apr 21 12:24:00 SERVER ioapic1: Changing APIC ID to 7
Apr 21 12:24:00 SERVER ioapic0 <Version 2.0> irqs 0-23 on motherboard
Apr 21 12:24:00 SERVER ioapic1 <Version 2.0> irqs 24-47 on motherboard
Apr 21 12:24:00 SERVER kbd1 at kbdmux0
Apr 21 12:24:00 SERVER cryptosoft0: <software crypto> on motherboard
Apr 21 12:24:00 SERVER aesni0: No AESNI support.
Apr 21 12:24:00 SERVER padlock0: No ACE support.
Apr 21 12:24:00 SERVER acpi0: <NEC > on motherboard
Apr 21 12:24:00 SERVER acpi0: Power Button (fixed)
Apr 21 12:24:00 SERVER cpu0: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER cpu1: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER cpu2: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER cpu3: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER cpu4: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER cpu5: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER cpu6: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER cpu7: <ACPI CPU> on acpi0
Apr 21 12:24:00 SERVER attimer0: <AT timer> port 0x40-0x43 irq 0 on acpi0
Apr 21 12:24:00 SERVER Timecounter "i8254" frequency 1193182 Hz quality 0
Apr 21 12:24:00 SERVER Event timer "i8254" frequency 1193182 Hz quality 100
Apr 21 12:24:00 SERVER atrtc0: <AT realtime clock> port 0x70-0x71 irq 8 on acpi0
Apr 21 12:24:00 SERVER Event timer "RTC" frequency 32768 Hz quality 0
Apr 21 12:24:00 SERVER hpet0: <High Precision Event Timer> iomem 0xfed00000-0xfed003ff on acpi0
Apr 21 12:24:00 SERVER Timecounter "HPET" frequency 14318180 Hz quality 950
Apr 21 12:24:00 SERVER Event timer "HPET" frequency 14318180 Hz quality 350
Apr 21 12:24:00 SERVER Event timer "HPET1" frequency 14318180 Hz quality 340
Apr 21 12:24:00 SERVER Event timer "HPET2" frequency 14318180 Hz quality 340
Apr 21 12:24:00 SERVER Event timer "HPET3" frequency 14318180 Hz quality 340
Apr 21 12:24:00 SERVER Timecounter "ACPI-safe" frequency 3579545 Hz quality 850
Apr 21 12:24:00 SERVER acpi_timer0: <24-bit timer at 3.579545MHz> port 0x808-0x80b on acpi0
Apr 21 12:24:00 SERVER pcib0: <ACPI Host-PCI bridge> port 0xcf8-0xcff on acpi0
Apr 21 12:24:00 SERVER pci0: <ACPI PCI bus> on pcib0
Apr 21 12:24:00 SERVER pcib1: <ACPI PCI-PCI bridge> at device 1.0 on pci0
Apr 21 12:24:00 SERVER pci5: <ACPI PCI bus> on pcib1
Apr 21 12:24:00 SERVER igb0: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xef80-0xef9f mem 0xfafe0000-0xfaffffff,0xfafc0000-0xfafdffff,0xfaf9c000-0xfaf9ffff irq 28 at device 0.0 on pci5
Apr 21 12:24:00 SERVER igb0: Using MSIX interrupts with 9 vectors
Apr 21 12:24:00 SERVER igb0: Ethernet address: 00:30:48:f3:5b:fe
Apr 21 12:24:00 SERVER igb0: Bound queue 0 to cpu 0
Apr 21 12:24:00 SERVER igb0: Bound queue 1 to cpu 1
Apr 21 12:24:00 SERVER igb0: Bound queue 2 to cpu 2
Apr 21 12:24:00 SERVER igb0: Bound queue 3 to cpu 3
Apr 21 12:24:00 SERVER igb0: Bound queue 4 to cpu 4
Apr 21 12:24:00 SERVER igb0: Bound queue 5 to cpu 5
Apr 21 12:24:00 SERVER igb0: Bound queue 6 to cpu 6
Apr 21 12:24:00 SERVER igb0: Bound queue 7 to cpu 7
Apr 21 12:24:00 SERVER igb1: <Intel(R) PRO/1000 Network Connection, Version - 2.5.3-k> port 0xef40-0xef5f mem 0xfaf60000-0xfaf7ffff,0xfaf40000-0xfaf5ffff,0xfaf1c000-0xfaf1ffff irq 40 at device 0.1 on pci5
Apr 21 12:24:00 SERVER igb1: Using MSIX interrupts with 9 vectors
Apr 21 12:24:00 SERVER igb1: Ethernet address: 00:30:48:f3:5b:ff
Apr 21 12:24:00 SERVER igb1: Bound queue 0 to cpu 0
Apr 21 12:24:00 SERVER igb1: Bound queue 1 to cpu 1
Apr 21 12:24:00 SERVER igb1: Bound queue 2 to cpu 2
Apr 21 12:24:00 SERVER igb1: Bound queue 3 to cpu 3
Apr 21 12:24:00 SERVER igb1: Bound queue 4 to cpu 4
Apr 21 12:24:00 SERVER igb1: Bound queue 5 to cpu 5
Apr 21 12:24:00 SERVER igb1: Bound queue 6 to cpu 6
Apr 21 12:24:00 SERVER igb1: Bound queue 7 to cpu 7
Apr 21 12:24:00 SERVER pcib2: <ACPI PCI-PCI bridge> at device 3.0 on pci0
Apr 21 12:24:00 SERVER pci4: <ACPI PCI bus> on pcib2
Apr 21 12:24:00 SERVER cxgbc0: <Chelsio T310, 1 port> mem 0xf9f7e000-0xf9f7efff,0xfa000000-0xfa7fffff,0xf9f7f000-0xf9f7ffff irq 24 at device 0.0 on pci4
Apr 21 12:24:00 SERVER cxgbc0: using MSI-X interrupts (9 vectors)
Apr 21 12:24:00 SERVER kernel: found newer FW version(7.12), driver compiled for version 7.11
Apr 21 12:24:00 SERVER kernel: found newer FW version(7.12), driver compiled for version 7.11
Apr 21 12:24:00 SERVER cxgb0: <Port 0 10GBASE-R> on cxgbc0
Apr 21 12:24:00 SERVER cxgb0: Using defaults for TSO: 65518/35/2048
Apr 21 12:24:00 SERVER cxgb0: Ethernet address: 00:07:43:05:ce:ce
Apr 21 12:24:00 SERVER cxgbc0: Firmware Version 7.12.0
Apr 21 12:24:00 SERVER pcib3: <ACPI PCI-PCI bridge> at device 5.0 on pci0
Apr 21 12:24:00 SERVER pci3: <ACPI PCI bus> on pcib3
Apr 21 12:24:00 SERVER mps0: <Avago Technologies (LSI) SAS2008> port 0xd000-0xd0ff mem 0xfadb0000-0xfadbffff,0xfadc0000-0xfadfffff irq 26 at device 0.0 on pci3
Apr 21 12:24:00 SERVER mps0: Firmware: 20.00.04.00, Driver: 20.00.00.00-fbsd
Apr 21 12:24:00 SERVER mps0: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Apr 21 12:24:00 SERVER pcib4: <ACPI PCI-PCI bridge> at device 7.0 on pci0
Apr 21 12:24:00 SERVER pci2: <ACPI PCI bus> on pcib4
Apr 21 12:24:00 SERVER mps1: <Avago Technologies (LSI) SAS2008> port 0xc000-0xc0ff mem 0xfabb0000-0xfabbffff,0xfabc0000-0xfabfffff irq 30 at device 0.0 on pci2
Apr 21 12:24:00 SERVER mps1: Firmware: 20.00.04.00, Driver: 20.00.00.00-fbsd
Apr 21 12:24:00 SERVER mps1: IOCCapabilities: 1285c<ScsiTaskFull,DiagTrace,SnapBuf,EEDP,TransRetry,EventReplay,HostDisc>
Apr 21 12:24:00 SERVER pci0: <base peripheral, interrupt controller> at device 20.0 (no driver attached)
Apr 21 12:24:00 SERVER pci0: <base peripheral, interrupt controller> at device 20.1 (no driver attached)
Apr 21 12:24:00 SERVER pci0: <base peripheral, interrupt controller> at device 20.2 (no driver attached)
Apr 21 12:24:00 SERVER pci0: <base peripheral, interrupt controller> at device 20.3 (no driver attached)
Apr 21 12:24:00 SERVER uhci0: <Intel 82801JI (ICH10) USB controller USB-D> port 0xbea0-0xbebf irq 16 at device 26.0 on pci0
Apr 21 12:24:00 SERVER uhci0: LegSup = 0x2f00
Apr 21 12:24:00 SERVER usbus0 on uhci0
Apr 21 12:24:00 SERVER uhci1: <Intel 82801JI (ICH10) USB controller USB-E> port 0xbe80-0xbe9f irq 21 at device 26.1 on pci0
Apr 21 12:24:00 SERVER uhci1: LegSup = 0x2f00
Apr 21 12:24:00 SERVER usbus1 on uhci1
Apr 21 12:24:00 SERVER uhci2: <Intel 82801JI (ICH10) USB controller USB-F> port 0xbe20-0xbe3f irq 19 at device 26.2 on pci0
Apr 21 12:24:00 SERVER uhci2: LegSup = 0x2f00
Apr 21 12:24:00 SERVER usbus2 on uhci2
Apr 21 12:24:00 SERVER ehci0: <Intel 82801JI (ICH10) USB 2.0 controller USB-B> mem 0xfaaf4000-0xfaaf43ff irq 18 at device 26.7 on pci0
Apr 21 12:24:00 SERVER usbus3: EHCI version 1.0
Apr 21 12:24:00 SERVER usbus3 on ehci0
Apr 21 12:24:00 SERVER uhci3: <Intel 82801JI (ICH10) USB controller USB-A> port 0xbf20-0xbf3f irq 23 at device 29.0 on pci0
Apr 21 12:24:00 SERVER uhci3: LegSup = 0x2f00
Apr 21 12:24:00 SERVER usbus4 on uhci3
Apr 21 12:24:00 SERVER uhci4: <Intel 82801JI (ICH10) USB controller USB-B> port 0xbf00-0xbf1f irq 19 at device 29.1 on pci0
Apr 21 12:24:00 SERVER uhci4: LegSup = 0x2f00
Apr 21 12:24:00 SERVER usbus5 on uhci4
Apr 21 12:24:00 SERVER uhci5: <Intel 82801JI (ICH10) USB controller USB-C> port 0xbec0-0xbedf irq 18 at device 29.2 on pci0
Apr 21 12:24:00 SERVER uhci5: LegSup = 0x2f00
Apr 21 12:24:00 SERVER usbus6 on uhci5
Apr 21 12:24:00 SERVER ehci1: <Intel 82801JI (ICH10) USB 2.0 controller USB-A> mem 0xfaaf6000-0xfaaf63ff irq 23 at device 29.7 on pci0
Apr 21 12:24:00 SERVER usbus7: EHCI version 1.0
Apr 21 12:24:00 SERVER usbus7 on ehci1
Apr 21 12:24:00 SERVER pcib5: <ACPI PCI-PCI bridge> at device 30.0 on pci0
Apr 21 12:24:00 SERVER pci1: <ACPI PCI bus> on pcib5
Apr 21 12:24:00 SERVER vgapci0: <VGA-compatible display> mem 0xf7000000-0xf7ffffff,0xf8ffc000-0xf8ffffff,0xf9000000-0xf97fffff irq 16 at device 3.0 on pci1
Apr 21 12:24:00 SERVER vgapci0: Boot video device
Apr 21 12:24:00 SERVER isab0: <PCI-ISA bridge> at device 31.0 on pci0
Apr 21 12:24:00 SERVER isa0: <ISA bus> on isab0
Apr 21 12:24:00 SERVER ahci0: <Intel ICH10 AHCI SATA controller> port 0xbff0-0xbff7,0xbfac-0xbfaf,0xbfe0-0xbfe7,0xbfa8-0xbfab,0xbf80-0xbf9f mem 0xfaafa000-0xfaafa7ff irq 19 at device 31.2 on pci0
Apr 21 12:24:00 SERVER ahci0: AHCI v1.20 with 6 3Gbps ports, Port Multiplier not supported
Apr 21 12:24:00 SERVER ahcich0: <AHCI channel> at channel 0 on ahci0
Apr 21 12:24:00 SERVER ahcich1: <AHCI channel> at channel 1 on ahci0
Apr 21 12:24:00 SERVER ahcich2: <AHCI channel> at channel 2 on ahci0
Apr 21 12:24:00 SERVER ahcich3: <AHCI channel> at channel 3 on ahci0
Apr 21 12:24:00 SERVER ahcich4: <AHCI channel> at channel 4 on ahci0
Apr 21 12:24:00 SERVER ahcich5: <AHCI channel> at channel 5 on ahci0
Apr 21 12:24:00 SERVER acpi_button0: <Power Button> on acpi0
Apr 21 12:24:00 SERVER qpi0: <QPI system bus> on motherboard
Apr 21 12:24:00 SERVER pcib6: <QPI Host-PCI bridge> pcibus 255 on qpi0
Apr 21 12:24:00 SERVER pci255: <PCI bus> on pcib6
Apr 21 12:24:00 SERVER ichwd0: <Intel ICH10R watchdog timer> on isa0
Apr 21 12:24:00 SERVER wbwd0: <Winbond 83627HF/F/HG/G (0x52/0x41) Watchdog Timer> at port 0x2e-0x2f on isa0
Apr 21 12:24:00 SERVER orm0: <ISA Option ROMs> at iomem 0xc0000-0xc7fff,0xc8000-0xc8fff,0xc9000-0xc9fff on isa0
Apr 21 12:24:00 SERVER sc0: <System console> at flags 0x100 on isa0
Apr 21 12:24:00 SERVER sc0: VGA <16 virtual consoles, flags=0x300>
Apr 21 12:24:00 SERVER vga0: <Generic ISA VGA> at port 0x3c0-0x3df iomem 0xa0000-0xbffff on isa0
Apr 21 12:24:00 SERVER atkbdc0: <Keyboard controller (i8042)> at port 0x60,0x64 on isa0
Apr 21 12:24:00 SERVER atkbd0: <AT Keyboard> irq 1 on atkbdc0
Apr 21 12:24:00 SERVER kbd0 at atkbd0
Apr 21 12:24:00 SERVER atkbd0: [GIANT-LOCKED]
Apr 21 12:24:00 SERVER coretemp0: <CPU On-Die Thermal Sensors> on cpu0
Apr 21 12:24:00 SERVER est0: <Enhanced SpeedStep Frequency Control> on cpu0
Apr 21 12:24:00 SERVER coretemp1: <CPU On-Die Thermal Sensors> on cpu1
Apr 21 12:24:00 SERVER est1: <Enhanced SpeedStep Frequency Control> on cpu1
Apr 21 12:24:00 SERVER coretemp2: <CPU On-Die Thermal Sensors> on cpu2
Apr 21 12:24:00 SERVER est2: <Enhanced SpeedStep Frequency Control> on cpu2
Apr 21 12:24:00 SERVER coretemp3: <CPU On-Die Thermal Sensors> on cpu3
Apr 21 12:24:00 SERVER est3: <Enhanced SpeedStep Frequency Control> on cpu3
Apr 21 12:24:00 SERVER coretemp4: <CPU On-Die Thermal Sensors> on cpu4
Apr 21 12:24:00 SERVER est4: <Enhanced SpeedStep Frequency Control> on cpu4
Apr 21 12:24:00 SERVER coretemp5: <CPU On-Die Thermal Sensors> on cpu5
Apr 21 12:24:00 SERVER est5: <Enhanced SpeedStep Frequency Control> on cpu5
Apr 21 12:24:00 SERVER coretemp6: <CPU On-Die Thermal Sensors> on cpu6
Apr 21 12:24:00 SERVER est6: <Enhanced SpeedStep Frequency Control> on cpu6
Apr 21 12:24:00 SERVER coretemp7: <CPU On-Die Thermal Sensors> on cpu7
Apr 21 12:24:00 SERVER est7: <Enhanced SpeedStep Frequency Control> on cpu7
Apr 21 12:24:00 SERVER ZFS filesystem version: 5
Apr 21 12:24:00 SERVER ZFS storage pool version: features support (5000)
Apr 21 12:24:00 SERVER Timecounters tick every 1.000 msec
Apr 21 12:24:00 SERVER ipfw2 (+ipv6) initialized, divert enabled, nat enabled, default to accept, logging disabled
Apr 21 12:24:00 SERVER random: unblocking device.
Apr 21 12:24:00 SERVER usbus0: 12Mbps Full Speed USB v1.0
Apr 21 12:24:00 SERVER usbus1: 12Mbps Full Speed USB v1.0
Apr 21 12:24:00 SERVER usbus2: 12Mbps Full Speed USB v1.0
Apr 21 12:24:00 SERVER usbus3: 480Mbps High Speed USB v2.0
Apr 21 12:24:00 SERVER usbus4: 12Mbps Full Speed USB v1.0
Apr 21 12:24:00 SERVER usbus5: 12Mbps Full Speed USB v1.0
Apr 21 12:24:00 SERVER usbus6: 12Mbps Full Speed USB v1.0
Apr 21 12:24:00 SERVER usbus7: 480Mbps High Speed USB v2.0
Apr 21 12:24:00 SERVER ugen0.1: <Intel> at usbus0
Apr 21 12:24:00 SERVER uhub0: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus0
Apr 21 12:24:00 SERVER ugen2.1: <Intel> at usbus2
Apr 21 12:24:00 SERVER uhub1: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus2
Apr 21 12:24:00 SERVER ugen1.1: <Intel> at usbus1
Apr 21 12:24:00 SERVER uhub2: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus1
Apr 21 12:24:00 SERVER ugen5.1: <Intel> at usbus5
Apr 21 12:24:00 SERVER uhub3: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus5
Apr 21 12:24:00 SERVER ugen4.1: <Intel> at usbus4
Apr 21 12:24:00 SERVER uhub4: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus4
Apr 21 12:24:00 SERVER ugen3.1: <Intel> at usbus3
Apr 21 12:24:00 SERVER uhub5: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus3
Apr 21 12:24:00 SERVER ugen7.1: <Intel> at usbus7
Apr 21 12:24:00 SERVER uhub6: <Intel EHCI root HUB, class 9/0, rev 2.00/1.00, addr 1> on usbus7
Apr 21 12:24:00 SERVER ugen6.1: <Intel> at usbus6
Apr 21 12:24:00 SERVER uhub7: <Intel UHCI root HUB, class 9/0, rev 1.00/1.00, addr 1> on usbus6
Apr 21 12:24:00 SERVER uhub0: 2 ports with 2 removable, self powered
Apr 21 12:24:00 SERVER uhub2: 2 ports with 2 removable, self powered
Apr 21 12:24:00 SERVER uhub1: 2 ports with 2 removable, self powered
Apr 21 12:24:00 SERVER uhub4: 2 ports with 2 removable, self powered
Apr 21 12:24:00 SERVER uhub3: 2 ports with 2 removable, self powered
Apr 21 12:24:00 SERVER uhub7: 2 ports with 2 removable, self powered
Apr 21 12:24:00 SERVER mps0: SAS Address for SATA device = 84789f379ad200ad
Apr 21 12:24:00 SERVER mps0: SAS Address from SATA device = 84789f379ad200ad
Apr 21 12:24:00 SERVER mps0: SAS Address for SATA device = 84789f3797d4ffac
Apr 21 12:24:00 SERVER mps0: SAS Address from SATA device = 84789f3797d4ffac
Apr 21 12:24:00 SERVER mps1: SAS Address for SATA device = 84789e559cd5f3ac
Apr 21 12:24:00 SERVER mps1: SAS Address from SATA device = 84789e559cd5f3ac
Apr 21 12:24:00 SERVER mps1: SAS Address for SATA device = 84789e59a0ccd7ac
Apr 21 12:24:00 SERVER mps1: SAS Address from SATA device = 84789e59a0ccd7ac
Apr 21 12:24:00 SERVER mps1: SAS Address for SATA device = 84789f3589b4f3ac
Apr 21 12:24:00 SERVER mps1: SAS Address from SATA device = 84789f3589b4f3ac
Apr 21 12:24:00 SERVER mps1: SAS Address for SATA device = 84789f3780c4fbac
Apr 21 12:24:00 SERVER mps1: SAS Address from SATA device = 84789f3780c4fbac
Apr 21 12:24:00 SERVER uhub6: 6 ports with 6 removable, self powered
Apr 21 12:24:00 SERVER uhub5: 6 ports with 6 removable, self powered
Apr 21 12:24:00 SERVER ugen4.2: <Mitsumi Electric> at usbus4
Apr 21 12:24:00 SERVER uhub8: <Mitsumi Electric Hub in Apple Extended USB Keyboard, class 9/0, rev 1.10/4.20, addr 2> on usbus4
Apr 21 12:24:00 SERVER ugen2.2: <American Megatrends Inc.> at usbus2
Apr 21 12:24:00 SERVER ukbd0: <Keyboard Interface> on usbus2
Apr 21 12:24:00 SERVER kbd2 at ukbd0
Apr 21 12:24:00 SERVER ugen7.2: <TDK LoR> at usbus7
Apr 21 12:24:00 SERVER umass0: <TDK LoR TF30 USB 3.0, class 0/0, rev 2.10/1.10, addr 2> on usbus7
Apr 21 12:24:00 SERVER umass0: SCSI over Bulk-Only; quirks = 0x8100
Apr 21 12:24:00 SERVER umass0:9:0:-1: Attached to scbus9
Apr 21 12:24:00 SERVER uhub8: 3 ports with 2 removable, bus powered
Apr 21 12:24:00 SERVER ugen4.3: <Mitsumi Electric> at usbus4
Apr 21 12:24:00 SERVER ukbd1: <Mitsumi Electric Apple Extended USB Keyboard, class 0/0, rev 1.10/4.20, addr 3> on usbus4
Apr 21 12:24:00 SERVER kbd3 at ukbd1
Apr 21 12:24:00 SERVER da6 at umass-sim0 bus 0 scbus9 target 0 lun 0
Apr 21 12:24:00 SERVER da6: <TDK LoR TF30 USB 3.0 PMAP> Removable Direct Access SPC-4 SCSI device
Apr 21 12:24:00 SERVER da6: Serial Number 07073C5D11D7D004
Apr 21 12:24:00 SERVER da6: 40.000MB/s transfers
Apr 21 12:24:00 SERVER da6: 7381MB (15116736 512 byte sectors)
Apr 21 12:24:00 SERVER da6: quirks=0x2<NO_6_BYTE>
Apr 21 12:24:00 SERVER da2 at mps1 bus 0 scbus1 target 2 lun 0
Apr 21 12:24:00 SERVER da2: <ATA Hitachi HDS72303 A5C0> Fixed Direct Access SPC-4 SCSI device
Apr 21 12:24:00 SERVER da2: Serial Number MK0311YHH692NA
Apr 21 12:24:00 SERVER da2: 600.000MB/s transfers
Apr 21 12:24:00 SERVER da2: Command Queueing enabled
Apr 21 12:24:00 SERVER da2: 2861588MB (5860533168 512 byte sectors)
Apr 21 12:24:00 SERVER da0 at mps0 bus 0 scbus0 target 5 lun 0
Apr 21 12:24:00 SERVER da0: <ATA Hitachi HDS72303 A5C0> Fixed Direct Access SPC-4 SCSI device
Apr 21 12:24:00 SERVER da0: Serial Number MK0331YHH8GRXA
Apr 21 12:24:00 SERVER da0: 600.000MB/s transfers
Apr 21 12:24:00 SERVER da0: Command Queueing enabled
Apr 21 12:24:00 SERVER da0: 2861588MB (5860533168 512 byte sectors)
Apr 21 12:24:00 SERVER da3 at mps1 bus 0 scbus1 target 4 lun 0
Apr 21 12:24:00 SERVER da3: <ATA Hitachi HDS72303 A5C0> Fixed Direct Access SPC-4 SCSI device
Apr 21 12:24:00 SERVER da3: Serial Number MK0311YHGVLSNA
Apr 21 12:24:00 SERVER da3: 600.000MB/s transfers
Apr 21 12:24:00 SERVER da3: Command Queueing enabled
Apr 21 12:24:00 SERVER da3: 2861588MB (5860533168 512 byte sectors)
Apr 21 12:24:00 SERVER da4 at mps1 bus 0 scbus1 target 6 lun 0
Apr 21 12:24:00 SERVER da4: <ATA Hitachi HDS72303 A5C0> Fixed Direct Access SPC-4 SCSI device
Apr 21 12:24:00 SERVER da4: Serial Number MK0311YHH80BVA
Apr 21 12:24:00 SERVER da4: 600.000MB/s transfers
Apr 21 12:24:00 SERVER da4: Command Queueing enabled
Apr 21 12:24:00 SERVER da4: 2861588MB (5860533168 512 byte sectors)
Apr 21 12:24:00 SERVER da1 at mps0 bus 0 scbus0 target 7 lun 0
Apr 21 12:24:00 SERVER da1: <ATA Hitachi HDS72303 A5C0> Fixed Direct Access SPC-4 SCSI device
Apr 21 12:24:00 SERVER da1: Serial Number MK0331YHH8JPYA
Apr 21 12:24:00 SERVER da1: 600.000MB/s transfers
Apr 21 12:24:00 SERVER da1: Command Queueing enabled
Apr 21 12:24:00 SERVER da1: 2861588MB (5860533168 512 byte sectors)
Apr 21 12:24:00 SERVER da5 at mps1 bus 0 scbus1 target 7 lun 0
Apr 21 12:24:00 SERVER da5: <ATA Hitachi HDS72303 A5C0> Fixed Direct Access SPC-4 SCSI device
Apr 21 12:24:00 SERVER da5: Serial Number MK0311YHGZPJ2A
Apr 21 12:24:00 SERVER da5: 600.000MB/s transfers
Apr 21 12:24:00 SERVER da5: Command Queueing enabled
Apr 21 12:24:00 SERVER da5: 2861588MB (5860533168 512 byte sectors)
Apr 21 12:24:00 SERVER SMP: AP CPU #1 Launched!
Apr 21 12:24:00 SERVER SMP: AP CPU #6 Launched!
Apr 21 12:24:00 SERVER SMP: AP CPU #3 Launched!
Apr 21 12:24:00 SERVER SMP: AP CPU #2 Launched!
Apr 21 12:24:00 SERVER SMP: AP CPU #7 Launched!
Apr 21 12:24:00 SERVER SMP: AP CPU #5 Launched!
Apr 21 12:24:00 SERVER SMP: AP CPU #4 Launched!
Apr 21 12:24:00 SERVER Timecounter "TSC" frequency 2128045176 Hz quality 1000
Apr 21 12:24:00 SERVER Trying to mount root from zfs:freenas-boot/ROOT/default []...
Apr 21 12:24:00 SERVER GEOM_RAID5: Module loaded, version 1.3.20140711.62 (rev f91e28e40bf7)
Apr 21 12:24:00 SERVER kernel: igb1: link state changed to UP
Apr 21 12:24:00 SERVER kernel: igb1: link state changed to UP
Apr 21 12:24:00 SERVER ipmi0: <IPMI System Interface> on isa0
Apr 21 12:24:00 SERVER ipmi0: KCS mode found at io 0xca2 alignment 0x1 on isa
Apr 21 12:24:00 SERVER ipmi0: IPMI device rev. 1, firmware rev. 3.03, version 2.0
Apr 21 12:24:00 SERVER ipmi0: Number of channels 2
Apr 21 12:24:00 SERVER ipmi0: Attached watchdog
Apr 21 12:24:00 SERVER hwpmc: SOFT/16/64/0x67<INT,USR,SYS,REA,WRI> TSC/1/64/0x20<REA> IAP/4/48/0x3ff<INT,USR,SYS,EDG,THR,REA,WRI,INV,QUA,PRC> IAF/3/48/0x67<INT,USR,SYS,REA,WRI> UCP/8/48/0x3f8<EDG,THR,REA,WRI,INV,QUA,PRC> UCF/1/48/0x60<REA,WRI>
Apr 21 12:24:00 SERVER kernel: igb1: link state changed to DOWN
Apr 21 12:24:00 SERVER kernel: igb1: link state changed to DOWN
Apr 21 12:24:00 SERVER kernel: cxgb0: link state changed to DOWN
Apr 21 12:24:00 SERVER kernel: cxgb0: link state changed to DOWN
Apr 21 12:24:00 SERVER kernel: vlan0: link state changed to DOWN
Apr 21 12:24:00 SERVER kernel: vlan0: link state changed to DOWN
Apr 21 12:24:00 SERVER kernel: igb1: link state changed to UP
Apr 21 12:24:00 SERVER kernel: igb1: link state changed to UP
Apr 21 12:24:00 SERVER kernel: cxgb0: link state changed to UP
Apr 21 12:24:00 SERVER kernel: cxgb0: link state changed to UP
Apr 21 12:24:00 SERVER kernel: vlan0: link state changed to UP
Apr 21 12:24:00 SERVER kernel: vlan0: link state changed to UP
Apr 21 12:24:00 SERVER ums0: <Mouse Interface> on usbus2
Apr 21 12:24:00 SERVER uhid0: <Mitsumi Electric Apple Extended USB Keyboard, class 0/0, rev 1.10/4.20, addr 3> on usbus4
Apr 21 12:24:00 SERVER vboxdrv: fAsync=0 offMin=0x252 offMax=0x10f8
Apr 21 12:25:15 SERVER (da6:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 0c c1 37 00 00 44 00
Apr 21 12:25:15 SERVER (da6:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Apr 21 12:25:15 SERVER (da6:umass-sim0:0:0:0): Retrying command
Apr 21 12:25:18 SERVER GEOM_ELI: Device da0p1.eli created.
Apr 21 12:25:18 SERVER GEOM_ELI: Encryption: AES-XTS 128
Apr 21 12:25:18 SERVER GEOM_ELI: Crypto: software
Apr 21 12:25:18 SERVER GEOM_ELI: Device da1p1.eli created.
Apr 21 12:25:18 SERVER GEOM_ELI: Encryption: AES-XTS 128
Apr 21 12:25:18 SERVER GEOM_ELI: Crypto: software
Apr 21 12:25:18 SERVER GEOM_ELI: Device da4p1.eli created.
Apr 21 12:25:18 SERVER GEOM_ELI: Encryption: AES-XTS 128
Apr 21 12:25:18 SERVER GEOM_ELI: Crypto: software
Apr 21 12:25:18 SERVER GEOM_ELI: Device da2p1.eli created.
Apr 21 12:25:18 SERVER GEOM_ELI: Encryption: AES-XTS 128
Apr 21 12:25:18 SERVER GEOM_ELI: Crypto: software
Apr 21 12:25:18 SERVER GEOM_ELI: Device da3p1.eli created.
Apr 21 12:25:18 SERVER GEOM_ELI: Encryption: AES-XTS 128
Apr 21 12:25:18 SERVER GEOM_ELI: Crypto: software
Apr 21 12:25:18 SERVER GEOM_ELI: Device da5p1.eli created.
Apr 21 12:25:18 SERVER GEOM_ELI: Encryption: AES-XTS 128
Apr 21 12:25:18 SERVER GEOM_ELI: Crypto: software
Apr 21 12:25:18 SERVER ntpd[2068]: ntpd 4.2.8p6-a (1): Starting
Apr 21 12:26:34 SERVER (da6:umass-sim0:0:0:0): READ(10). CDB: 28 00 00 0f 36 85 00 00 6b 00
Apr 21 12:26:34 SERVER (da6:umass-sim0:0:0:0): CAM status: CCB request completed with an error
Apr 21 12:26:34 SERVER (da6:umass-sim0:0:0:0): Retrying command
Apr 21 12:26:36 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: /usr/local/bin/net -d 0 getlocalsid
Apr 21 12:26:36 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: /usr/local/bin/net -d 0 setlocalsid S-1-5-21-3291793312-625427945-195985885
Apr 21 12:26:36 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: /sbin/sysctl -n 'kern.maxfilesperproc'
Apr 21 12:26:37 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: mount
Apr 21 12:26:37 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: /usr/local/bin/net -d 0 getlocalsid
Apr 21 12:26:37 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: /usr/bin/getent passwd ‘xxxxx’
Apr 21 12:26:37 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: /usr/bin/getent passwd ‘xxxxx’
Apr 21 12:26:37 SERVER generate_smb4_conf.py: [common.pipesubr:61] Popen()ing: /usr/bin/getent passwd ‘xxxxx’
Apr 21 12:26:37 SERVER smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 1
Apr 21 12:26:38 SERVER root: /etc/rc: WARNING: failed precmd routine for vmware_guestd
Apr 21 12:26:38 SERVER smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 2
Apr 21 12:26:39 SERVER smbd: dnssd_clientstub ConnectToServer: connect()-> No of tries: 3
Apr 21 12:26:41 SERVER smbd: dnssd_clientstub ConnectToServer: connect() failed path:/var/run/mdnsd Socket:35 Err:-1 Errno:2 No such file or directory
Apr 21 12:26:55 SERVER mDNSResponder: mDNSResponder (Engineering Build) (Apr 11 2016 18:06:30) starting
Apr 21 12:26:55 SERVER mDNSResponder: 10: Listening for incoming Unix Domain Socket client requests
Apr 21 12:26:55 SERVER mDNSResponder: mDNS_AddDNSServer: Lock not held! mDNS_busy (0) mDNS_reentrancy (0)
Apr 21 12:26:55 SERVER mDNSResponder: mDNSPlatformSourceAddrForDest: connect 1.1.1.1 failed errno 51 (Network is unreachable)
Apr 21 12:27:03 SERVER mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000801406D60 SERVER.local. (Addr) that's already in the list
Apr 21 12:27:03 SERVER mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000801407180 1.4.168.192.in-addr.arpa. (PTR) that's already in the list
Apr 21 12:27:03 SERVER mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 0000000801409D60 SERVER.local. (Addr) that's already in the list
Apr 21 12:27:03 SERVER mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 000000080140A180 1.6.168.192.in-addr.arpa. (PTR) that's already in the list
Apr 21 12:27:03 SERVER mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 000000080140BD60 SERVER.local. (Addr) that's already in the list
Apr 21 12:27:03 SERVER mDNSResponder: mDNS_Register_internal: ERROR!! Tried to register AuthRecord 000000080140C180 1.2.168.192.in-addr.arpa. (PTR) that's already in the list
Apr 21 12:27:05 SERVER alert.py: [common.system:210] Failed to send email: [Errno 8] hostname nor servname provided, or not known

 

[BigDave]

Does anyone know if specific firmware versions of Chelsio S310E-CR cards are better supported then others?

The log indicates the driver is compiled for FW version 7.11
You card is showing up as having FW version 7.12

When I installed my 10GbE hardware, I came to a quick conclusion that the older 310 cards were slower, problematic
and not worth effort. There is a reason they are priced so low. The T-420 cards have worked without issue for me
but I've had way to much trouble locating cards with full height brackets. I'm moving towards Intel X520 (just paid
$75 for another dual port this morning), the X520-DA1 currently installed in my server, is working very well.
If I was able to afford them the Chelsio T-520 cards are sweet, but are not within my budget range.

 

[helloha]

Thanks for pointing it out...

It is strange because they only seem to have released up until 7.11.

I think I pinpointed the issue to the card itself because it happens both on AFP and CIFS.

I kind of need to stick to Chelsio because I run OSX and intel cards don't have driver support on that platform.

I'll try to update the firmware to the correct version and report back.

 

[Mlovelace]

I heard about fiberstore from a colleague who has had a pair of 10Gbase-ER SFP+ transceivers in production going on 6 months now. They cost him $180 each when the vendor locked transceivers would have been $2200 each. He has had no issues and they are functionally the same as the non-clone modules. I can report a similar experience on a much shorter time line. So far with these transceivers I've had no issues with signal loss, noise, temperature increase, or packet loss. They have been functionally the same as the non-clone transceivers we have. I can't say if they will last year over year, but at $16-$20 each I don't care if I have to replace them more often.

As a followup to my previous post, I now have 68 of the fiberstore transceivers in production for the following: Chelsio T520-CR, Intel X520-DA2, HP 560SFP+, NetApp X1160A-R6 Mezzanine and Extreme Networks X670v-48X. There have been zero issues, all voltages and temps are normal, no rx/tx errors or dropped packets. Overall positive experience.

 

[jgreco]

68 of the fiberstore transceivers in production

*says a bad word*

 

[FatBoyNotSoSlim]

Just get this:

Lot 5 Chelsio S310E-CR 10Gbps SFP+ Adapter PCI-e x8 10GB 110-1107-30 10 Gigabit
http://www.benl.ebay.be/itm/361426967654?_trksid=p2060353.m1438.l2649&ssPageName=STRK:MEBIDX:IT

Add twinax cable and you're done!

Heh, I actually bought this, and whilst they do work fine in Freenas for my needs, they were hard crashing with BSOD when used in Windows, and aren't supported in ESXi 5.0+ (My fault for not checking the HCL closely enough).

But this raises a good point. I'm now on the lookout for an X520 card for my ESXi box*, can I use that with the Chelsio cards with the generic direct attach cable I already have? I got this one from FS.com http://www.fs.com/products/35194.html and it was working during my testing, so it'd be great to be able to keep it if that means it'll still work between the Chelsio and Intel cards.

*I was also looking for a T420 card for my esxi box, however it's no longer on the HCL after 5.1, and I don't want to have the same issue again.

 

[jgreco]

Probably. The Intel cards seem to be fine with generic direct attach from what I've seen, but most of what I do involves actual switches.

 

[CookiesLikeWhoa]

Maybe this isn't the right place since it deals with 40g but I'm trying to wrap my head around this a bit since I've had my eyes opened to the networking world. So, sorry in advance!

I understand a 40g to 40g link (QSFP+ to QSFP+), but I've been seeing these QSFP+ to 4 SFP+ 10g breakout cables. What's the point of these cables? I thinking you connect the 4 SFP+ transceivers to a switch and trunk the port to get 40g of bandwith but 10g per connection. The only thing about that is don't switches require all the connections to be the same speed if you are using link aggregation? If so, how would a breakout like that work?

Google only got me so far :(

 

[depasseg]

From my extremely limited experience with 40G, the QSFP+ is exactly like it sound - 4 10G SFP+ connections. The switch can be configured to recognize it as a single 40G link, or you you configure the port to be split into 4 different 10G ports (look at the QSFP ports and notice the numbering counts by 4). Then you configure each 10G port as you normally would.

https://www.force10networks.com/CSP...s/S-Series/S4810_Installation_26_Nov-2012.pdf

 

[CookiesLikeWhoa]

Thank you for the response!

So if you're running a Chelsio t580 out of a server and have a QSFP+ to SFP+ breakout cable to your switch, you could configure it to have 4 10g links or a single 40g link? That's pretty wild. Though in this day and age of SSD's I can understand it.

 

[depasseg]

So if you're running a Chelsio t580 out of a server and have a QSFP+ to SFP+ breakout cable to your switch, you could configure it to have 4 10g links or a single 40g link?

Yes, exactly!

 

[CookiesLikeWhoa]

Yes, exactly!

Thank you for the help! It makes sense, but it just seemed as if it made too much sense.

 

[jgreco]

Thank you for the help! It makes sense, but it just seemed as if it made too much sense.

Yeah, we've reached the point at which it's gotten more difficult to get to that next bump from 10G to a new speed, so the intermediate 40G tier is similar to a pre-built LACP hardware configuration. There's also a 10-lane 100Gbps product. There's a not too horrible read about it here:

http://www.information-age.com/tech...ethernet-market-could-mean-network-management

Anyways the short form is that a "10G with 40G uplink" switch is basically just a switch with 10G switching silicon; four 10G lanes are bonded together to make that 40G. That implies that you could potentially peel them apart if the switch software allows for it.

In the data center, density is king, so if you look at something like the Mellanox SX1012 (12 x QSFP+) which is a half width 1U switch, you might note that if you were to put two of these side by side at the top of rack, you could use 11 ports with breakout cables such as these and get two 10G connections per rack unit (redundancy) for 44U of space, and then have two 40G uplinks from the rack, all with a minimal amount of fuss.

 

[CookiesLikeWhoa]

That was an interesting read. Didn't realize 40G wasn't "standard", though it makes sense now that I've read that.

Now with that all be said, do you have to have a specific switch to handle the 4x10G bonding or can you just use any managed switch, bond the ports and run it to your selected 40G transceiver?

 

[TheDubiousDubber]

I ran out of ports on my current switch, so it came time to upgrade. Managed to pick up a 48 port with 2x 10GB SFP+ ports. I am planning to utilize the 10GB to connect a single workstation and my FreeNAS server. I'm looking at picking up a couple of the Intel X520s, but came up with a few questions. The -DA seems to be, obviously, cheaper than the -SR. Roughly $40 cheaper. I looked at getting optics from Fiberstore, but that would mostly make up the difference when you factor in shipping. ($16 + $21 s&h). I'm wondering if anyone has had any luck with other generics or something that can be had cheaply on eBay with these cards? Also, I'm curious if there is any reason to go with a dual port card over a single port? I have no need for link aggregation and not sure what else a second port would offer a home user. I'm still a networking noob, so thought I'd ask, just in case.

Edit: Just reread an earlier post by JGreco saying Intel usually only work with Intel optics. Though other questions still apply.

 

[jgreco]

I ran out of ports on my current switch, so it came time to upgrade. Managed to pick up a 48 port with 2x 10GB SFP+ ports. I am planning to utilize the 10GB to connect a single workstation and my FreeNAS server. I'm looking at picking up a couple of the Intel X520s, but came up with a few questions. The -DA seems to be, obviously, cheaper than the -SR. Roughly $40 cheaper. I looked at getting optics from Fiberstore, but that would mostly make up the difference when you factor in shipping. ($16 + $21 s&h). I'm wondering if anyone has had any luck with other generics or something that can be had cheaply on eBay with these cards? Also, I'm curious if there is any reason to go with a dual port card over a single port? I have no need for link aggregation and not sure what else a second port would offer a home user. I'm still a networking noob, so thought I'd ask, just in case.

Edit: Just reread an earlier post by JGreco saying Intel usually only work with Intel optics. Though other questions still apply.

I believe Fiberstore will sell you Intel compatible optics. It's just a matter of how they're coded.

If you're going with fiber and you can get -SR at a reasonable price, it may make sense to do so, rather than dealing with trying to source an optic separately.

You can absolutely do a single port card if that's all you need. The duals are sometimes more readily found because they tend to be used in servers that way, but looks like the singles are on eBay for around $50 right now. No endorsement of seller intended. You might someday regret the single port, but at that point, just buy another or buy a dual and sell the single, or whatever. Don't overthink it.

It does look like the price premium for a -SR (with optic) card is ~$40.

 

[TheDubiousDubber]

Managed to pick up two x520 cards with optics for both, all for around $150. Considering the switch was around the same, after you add in cables and switch side transceivers, I built a 10gig back bone all for under $400. Now I just have to hook it all up and make sure it works properly. *crosses fingers*

 

[Stryf]

Is the TOE (tcp/ip Offload Engine) recommended or should I avoid it?

I'm trying to decide on either the CC2-N320E-SR or the TOE feature in CC2-S320E-SR. The S320E-SR looks like it has more stuff on it and i'm not worried about the full height size. My other choice is the Intel X520-DA2 which is about $30 more on ebay.

My Spec: Supermicro X10SL7-F w/ Xeon E3-1220 v3 @ 3.10GHz / 16GB RAM

 

[jgreco]

The best answer that I can give you is that the cxgb driver claims to support transmit/receive checksum offload, TCP segmentation offload, and large receive offload, and I'd imagine that largely requires TOE hardware.

I believe that there've been some unhappiness stories with the 320's, and I can't say I've paid a lot of attention to them. The T420-CR is an absolutely awesome card with FreeNAS, though, and the T520-LL-CR makes my ESXi hosts #include <random obscene comparison here>

 

[Stryf]

I went and bought the Intel X520-DA2 and DA1 cards and a 10 meter 50/125 LC-LC Aqua colored fiber cable. One purchase came with four Unix branded modules.

Transferring from Windows 10 desktop to Freenas server with a raidz1 4x4TB using CIFS.
1) Default adapter settings got me a solid 144MB/s transfer on a 9GB MP4 file.
2) Setting the adapters with Jumbo Frame to 9014 and transfer buffers maxed out i got up to 600MB/s on the same file.

I still need to fine tune the adapters on both ends so I'll post back later on results.