10Gb NICs and iperf is showing 1Gb speeds

Status
Not open for further replies.

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
I tried putting in the tunables but I'm a bit confused on

Code:
hw.cxgbe.config_file="uwire"


Where do those "Tunables" go. I tried Loader but it doesn't like quotes.

what chelsio card are you exactly using, which bus type and so on?

Chesio T420 CR on slot 7 of my X8DTH-iF motherboard which according to the manual is labeled...
Code:
Slot7 PCI-E 2.0 X8
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
Here is what I have for tunables...
 

Attachments

  • Tunables.JPG
    Tunables.JPG
    155.3 KB · Views: 549

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
and here is my pciconf -lc...

Code:
none2@pci0:6:0:0:       class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 128(2048) FLR link x8(x8)
                 speed 2.5(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0008[138] = MFVC 1
ecap 0003[1b8] = Serial 1 0000000000000000
ecap 0010[1c4] = SRIOV 1
ecap 000e[204] = ARI 1
none3@pci0:6:0:1:       class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 128(2048) FLR link x8(x8)
                 speed 2.5(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 0010[144] = SRIOV 1
ecap 000e[184] = ARI 1
none4@pci0:6:0:2:       class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 128(2048) FLR link x8(x8)
                 speed 2.5(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 0010[144] = SRIOV 1
ecap 000e[184] = ARI 1
none5@pci0:6:0:3:       class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 128(2048) FLR link x8(x8)
                 speed 2.5(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 0010[144] = SRIOV 1
ecap 000e[184] = ARI 1
t4nex0@pci0:6:0:4:      class=0x020000 card=0x00001425 chip=0x44011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 32 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 128 messages in map 0x20 enabled
    cap 10[6c] = PCI-Express 2 endpoint max data 128(2048) FLR link x8(x8)
                 speed 2.5(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 000e[144] = ARI 1


lspci...

Code:
06:00.0 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller
06:00.1 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller
06:00.2 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller
06:00.3 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller
06:00.4 Ethernet controller: Chelsio Communications Inc T420-CR Unified Wire Ethernet Controller
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
Code:
ix0@pci0:2:0:0: class=0x020000 card=0x00068086 chip=0x10fb8086 rev=0x01 hdr=0x00
  cap 01[40] = powerspec 3  supports D0 D3  current D0
  cap 05[50] = MSI supports 1 message, 64 bit, vector masks
  cap 11[70] = MSI-X supports 64 messages in map 0x20 enabled
  cap 10[a0] = PCI-Express 2 endpoint max data 128(512) FLR link x8(x8)
  speed 5.0(5.0) ASPM disabled(L0s)
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0003[140] = Serial 1 90e2baffff0045fc
ecap 000e[150] = ARI 1
ecap 0010[160] = SRIOV 1

This is my Intel 10Gb card in my FreeNAS box (iperf tested at over 930Gbps) notice the
connection speed (I've highlighted in red) and I also see no serial number???
I'm wondering if the bargain card/cards are faulty.
  1. t4nex0@pci0:6:0:4: class=0x020000 card=0x00001425 chip=0x44011425 rev=0x00 hdr=0x00
  2. cap 01[40] = powerspec 3 supports D0 D3 current D0
  3. cap 05[48] = MSI supports 32 messages, 64 bit, vector masks
  4. cap 11[60] = MSI-X supports 128 messages in map 0x20 enabled
  5. cap 10[6c] = PCI-Express 2 endpoint max data 128(2048) FLR link x8(x8)
  6. speed 2.5(5.0) ASPM disabled(L0s/L1)
  7. cap 03[a8] = VPD
  8. ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
  9. ecap 0003[138] = Serial 1 0000000000000000
  10. ecap 000e[144] = ARI 1
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
now that you point that out, it does seem a bit strange. That is on my older SuperMicro I'm trying to migrate away from so that might, though doubtfully explain the speed, the serial number on the other hand does have me scratching my head.
I ran lspci on my newer system and I don't even see the Ethernet devices listed...

I tried to | grep ethernet but got nothing so I'm not exactly sure which devices are the Ethernet controllers...

Code:
00:00.0 Host bridge: Intel Corporation 5520 I/O Hub to ESI Port (rev 22)
00:01.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 1 (rev 22)
00:03.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 3 (rev 22)
00:05.0 PCI bridge: Intel Corporation 5520/X58 I/O Hub PCI Express Root Port 5 (rev 22)
00:07.0 PCI bridge: Intel Corporation 5520/5500/X58 I/O Hub PCI Express Root Port 7 (rev 22)
00:09.0 PCI bridge: Intel Corporation 7500/5520/5500/X58 I/O Hub PCI Express Root Port 9 (rev 22)
00:13.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub I/OxAPIC Interrupt Controller (rev 22)
00:14.0 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub System Management Registers (rev 22)
00:14.1 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub GPIO and Scratch Pad Registers (rev 22)
00:14.2 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Control Status and RAS Registers (rev 22)
00:14.3 PIC: Intel Corporation 7500/5520/5500/X58 I/O Hub Throttle Registers (rev 22)
00:16.0 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.1 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.2 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.3 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.4 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.5 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.6 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:16.7 System peripheral: Intel Corporation 5520/5500/X58 Chipset QuickData Technology Device (rev 22)
00:1a.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #4
00:1a.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #5
00:1a.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #6
00:1a.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #2
00:1d.0 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #1
00:1d.1 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #2
00:1d.2 USB controller: Intel Corporation 82801JI (ICH10 Family) USB UHCI Controller #3
00:1d.7 USB controller: Intel Corporation 82801JI (ICH10 Family) USB2 EHCI Controller #1
00:1e.0 PCI bridge: Intel Corporation 82801 PCI Bridge (rev 90)
00:1f.0 ISA bridge: Intel Corporation 82801JIR (ICH10R) LPC Interface Controller
00:1f.2 SATA controller: Intel Corporation 82801JI (ICH10 Family) SATA AHCI Controller
00:1f.3 SMBus: Intel Corporation 82801JI (ICH10 Family) SMBus Controller
02:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
03:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
04:00.0 Serial Attached SCSI controller: LSI Logic / Symbios Logic SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
06:04.0 VGA compatible controller: Matrox Electronics Systems Ltd. MGA G200eW WPCM450 (rev 0a)


You'll notice there are no serial numbers listed on this one either....

Code:
hostb0@pci0:0:0:0:    class=0x060000 card=0x040015d9 chip=0x34068086 rev=0x22 hdr=0x00
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port max data 128(128) link x4(x4)
                 speed 2.5(2.5) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib1@pci0:0:1:0:    class=0x060400 card=0x040015d9 chip=0x34088086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 128(256) link x0(x4)
                 speed 0.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib2@pci0:0:3:0:    class=0x060400 card=0x040015d9 chip=0x340a8086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 256(256) link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib3@pci0:0:5:0:    class=0x060400 card=0x040015d9 chip=0x340c8086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 256(256) link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
pcib4@pci0:0:7:0:    class=0x060400 card=0x040015d9 chip=0x340e8086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 256(256) link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib5@pci0:0:9:0:    class=0x060400 card=0x040015d9 chip=0x34108086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 128(256) link x0(x8)
                 speed 0.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ioapic0@pci0:0:19:0:    class=0x080020 card=0x00000000 chip=0x342d8086 rev=0x22 hdr=0x00
    cap 01[6c] = powerspec 3  supports D0 D3  current D0
none0@pci0:0:20:0:    class=0x080000 card=0x00000000 chip=0x342e8086 rev=0x22 hdr=0x00
    cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
                 ASPM disabled(L0s)
none1@pci0:0:20:1:    class=0x080000 card=0x00000000 chip=0x34228086 rev=0x22 hdr=0x00
    cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
                 ASPM disabled(L0s)
none2@pci0:0:20:2:    class=0x080000 card=0x00000000 chip=0x34238086 rev=0x22 hdr=0x00
    cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
                 ASPM disabled(L0s)
none3@pci0:0:20:3:    class=0x080000 card=0x00000000 chip=0x34388086 rev=0x22 hdr=0x00
none4@pci0:0:22:0:    class=0x088000 card=0x040015d9 chip=0x34308086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none5@pci0:0:22:1:    class=0x088000 card=0x040015d9 chip=0x34318086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none6@pci0:0:22:2:    class=0x088000 card=0x040015d9 chip=0x34328086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none7@pci0:0:22:3:    class=0x088000 card=0x040015d9 chip=0x34338086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none8@pci0:0:22:4:    class=0x088000 card=0x040015d9 chip=0x34298086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none9@pci0:0:22:5:    class=0x088000 card=0x040015d9 chip=0x342a8086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none10@pci0:0:22:6:    class=0x088000 card=0x040015d9 chip=0x342b8086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none11@pci0:0:22:7:    class=0x088000 card=0x040015d9 chip=0x342c8086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
uhci0@pci0:0:26:0:    class=0x0c0300 card=0x040015d9 chip=0x3a378086 rev=0x00 hdr=0x00
    cap 13[50] = PCI Advanced Features: FLR TP
uhci1@pci0:0:26:1:    class=0x0c0300 card=0x040015d9 chip=0x3a388086 rev=0x00 hdr=0x00
    cap 13[50] = PCI Advanced Features: FLR TP
uhci2@pci0:0:26:2:    class=0x0c0300 card=0x040015d9 chip=0x3a398086 rev=0x00 hdr=0x00
    cap 13[50] = PCI Advanced Features: FLR TP
ehci0@pci0:0:26:7:    class=0x0c0320 card=0x040015d9 chip=0x3a3c8086 rev=0x00 hdr=0x00
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP
uhci3@pci0:0:29:0:    class=0x0c0300 card=0x040015d9 chip=0x3a348086 rev=0x00 hdr=0x00
    cap 13[50] = PCI Advanced Features: FLR TP
uhci4@pci0:0:29:1:    class=0x0c0300 card=0x040015d9 chip=0x3a358086 rev=0x00 hdr=0x00
    cap 13[50] = PCI Advanced Features: FLR TP
uhci5@pci0:0:29:2:    class=0x0c0300 card=0x040015d9 chip=0x3a368086 rev=0x00 hdr=0x00
    cap 13[50] = PCI Advanced Features: FLR TP
ehci1@pci0:0:29:7:    class=0x0c0320 card=0x040015d9 chip=0x3a3a8086 rev=0x00 hdr=0x00
    cap 01[50] = powerspec 2  supports D0 D3  current D0
    cap 0a[58] = EHCI Debug Port at offset 0xa0 in map 0x14
    cap 13[98] = PCI Advanced Features: FLR TP
pcib6@pci0:0:30:0:    class=0x060401 card=0x040015d9 chip=0x244e8086 rev=0x90 hdr=0x01
    cap 0d[50] = PCI Bridge card=0x040015d9
isab0@pci0:0:31:0:    class=0x060100 card=0x040015d9 chip=0x3a168086 rev=0x00 hdr=0x00
    cap 09[e0] = vendor (length 12) Intel cap 1 version 0
         features: SATA RAID-5, 4 PCI-e x1 slots
ahci0@pci0:0:31:2:    class=0x010601 card=0x040015d9 chip=0x3a228086 rev=0x00 hdr=0x00
    cap 05[80] = MSI supports 16 messages enabled with 1 message
    cap 01[70] = powerspec 3  supports D0 D3  current D0
    cap 12[a8] = SATA Index-Data Pair
    cap 13[b0] = PCI Advanced Features: FLR TP
none12@pci0:0:31:3:    class=0x0c0500 card=0x040015d9 chip=0x3a308086 rev=0x00 hdr=0x00
mps0@pci0:2:0:0:    class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit
    cap 11[c0] = MSI-X supports 15 messages in map 0x14 enabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SRIOV 1
ecap 000e[190] = ARI 1
mps1@pci0:3:0:0:    class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit
    cap 11[c0] = MSI-X supports 15 messages in map 0x14 enabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SRIOV 1
ecap 000e[190] = ARI 1
mps2@pci0:4:0:0:    class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit
    cap 11[c0] = MSI-X supports 15 messages in map 0x14 enabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 1 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SRIOV 1
ecap 000e[190] = ARI 1
vgapci0@pci0:6:4:0:    class=0x030000 card=0x040015d9 chip=0x0532102b rev=0x0a hdr=0x00
    cap 01[dc] = powerspec 1  supports D0 D3  current D0
pcib8@pci0:128:0:0:    class=0x060400 card=0x040015d9 chip=0x34208086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port max data 128(256) link x0(x4)
                 speed 0.0(2.5) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib9@pci0:128:1:0:    class=0x060400 card=0x040015d9 chip=0x34088086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 128(256) link x0(x4)
                 speed 0.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib10@pci0:128:3:0:    class=0x060400 card=0x040015d9 chip=0x340a8086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 256(256) link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib11@pci0:128:5:0:    class=0x060400 card=0x040015d9 chip=0x340c8086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 128(256) link x0(x8)
                 speed 0.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
pcib12@pci0:128:7:0:    class=0x060400 card=0x040015d9 chip=0x340e8086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 256(256) link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ecap 000b[160] = Vendor 0
pcib13@pci0:128:9:0:    class=0x060400 card=0x040015d9 chip=0x34108086 rev=0x22 hdr=0x01
    cap 0d[40] = PCI Bridge card=0x040015d9
    cap 05[60] = MSI supports 2 messages, vector masks
    cap 10[90] = PCI-Express 2 root port slot max data 256(256) link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 000d[150] = ACS 1
ioapic1@pci0:128:19:0:    class=0x080020 card=0x00000000 chip=0x342d8086 rev=0x22 hdr=0x00
    cap 01[6c] = powerspec 3  supports D0 D3  current D0
none13@pci0:128:20:0:    class=0x080000 card=0x00000000 chip=0x342e8086 rev=0x22 hdr=0x00
    cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
                 ASPM disabled(L0s)
none14@pci0:128:20:1:    class=0x080000 card=0x00000000 chip=0x34228086 rev=0x22 hdr=0x00
    cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
                 ASPM disabled(L0s)
none15@pci0:128:20:2:    class=0x080000 card=0x00000000 chip=0x34238086 rev=0x22 hdr=0x00
    cap 10[40] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
                 ASPM disabled(L0s)
none16@pci0:128:20:3:    class=0x080000 card=0x00000000 chip=0x34388086 rev=0x22 hdr=0x00
none17@pci0:128:22:0:    class=0x088000 card=0x040015d9 chip=0x34308086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none18@pci0:128:22:1:    class=0x088000 card=0x040015d9 chip=0x34318086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none19@pci0:128:22:2:    class=0x088000 card=0x040015d9 chip=0x34328086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none20@pci0:128:22:3:    class=0x088000 card=0x040015d9 chip=0x34338086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none21@pci0:128:22:4:    class=0x088000 card=0x040015d9 chip=0x34298086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none22@pci0:128:22:5:    class=0x088000 card=0x040015d9 chip=0x342a8086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none23@pci0:128:22:6:    class=0x088000 card=0x040015d9 chip=0x342b8086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none24@pci0:128:22:7:    class=0x088000 card=0x040015d9 chip=0x342c8086 rev=0x22 hdr=0x00
    cap 11[80] = MSI-X supports 1 message in map 0x10
    cap 10[90] = PCI-Express 2 root endpoint max data 128(128) link x0(x0)
    cap 01[e0] = powerspec 3  supports D0 D3  current D0
none25@pci0:131:0:0:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0008[138] = MFVC 1
ecap 0003[1b8] = Serial 1 0000000000000000
ecap 0010[1c4] = SRIOV 1
ecap 000e[204] = ARI 1
none26@pci0:131:0:1:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 0010[144] = SRIOV 1
ecap 000e[184] = ARI 1
none27@pci0:131:0:2:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 0010[144] = SRIOV 1
ecap 000e[184] = ARI 1
none28@pci0:131:0:3:    class=0x020000 card=0x00001425 chip=0x40011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 8 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 8 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 0010[144] = SRIOV 1
ecap 000e[184] = ARI 1
t4nex0@pci0:131:0:4:    class=0x020000 card=0x00001425 chip=0x44011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 32 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 128 messages in map 0x20 enabled
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 000e[144] = ARI 1
none29@pci0:131:0:5:    class=0x010000 card=0x00001425 chip=0x45011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 32 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 40 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 000e[144] = ARI 1
none30@pci0:131:0:6:    class=0x0c0400 card=0x00001425 chip=0x46011425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 32 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 40 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 000e[144] = ARI 1
none31@pci0:131:0:7:    class=0x020000 card=0x00001425 chip=0x00001425 rev=0x00 hdr=0x00
    cap 01[40] = powerspec 3  supports D0 D3  current D0
    cap 05[48] = MSI supports 32 messages, 64 bit, vector masks
    cap 11[60] = MSI-X supports 32 messages in map 0x20
    cap 10[6c] = PCI-Express 2 endpoint max data 256(2048) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s/L1)
    cap 03[a8] = VPD
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0003[138] = Serial 1 0000000000000000
ecap 000e[144] = ARI 1
mps3@pci0:133:0:0:    class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit
    cap 11[c0] = MSI-X supports 15 messages in map 0x14 enabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SRIOV 1
ecap 000e[190] = ARI 1
mps4@pci0:134:0:0:    class=0x010700 card=0x30201000 chip=0x00721000 rev=0x03 hdr=0x00
    cap 01[50] = powerspec 3  supports D0 D1 D2 D3  current D0
    cap 10[68] = PCI-Express 2 endpoint max data 256(4096) FLR link x8(x8)
                 speed 5.0(5.0) ASPM disabled(L0s)
    cap 03[d0] = VPD
    cap 05[a8] = MSI supports 1 message, 64 bit
    cap 11[c0] = MSI-X supports 15 messages in map 0x14 enabled
ecap 0001[100] = AER 1 0 fatal 0 non-fatal 0 corrected
ecap 0004[138] = Power Budgeting 1
ecap 0010[150] = SRIOV 1
ecap 000e[190] = ARI 1
hostb1@pci0:255:0:0:    class=0x060000 card=0x80868086 chip=0x2c708086 rev=0x02 hdr=0x00
hostb2@pci0:255:0:1:    class=0x060000 card=0x80868086 chip=0x2d818086 rev=0x02 hdr=0x00
hostb3@pci0:255:2:0:    class=0x060000 card=0x80868086 chip=0x2d908086 rev=0x02 hdr=0x00
hostb4@pci0:255:2:1:    class=0x060000 card=0x80868086 chip=0x2d918086 rev=0x02 hdr=0x00
hostb5@pci0:255:2:2:    class=0x060000 card=0x80868086 chip=0x2d928086 rev=0x02 hdr=0x00
hostb6@pci0:255:2:3:    class=0x060000 card=0x80868086 chip=0x2d938086 rev=0x02 hdr=0x00
hostb7@pci0:255:2:4:    class=0x060000 card=0x80868086 chip=0x2d948086 rev=0x02 hdr=0x00
hostb8@pci0:255:2:5:    class=0x060000 card=0x80868086 chip=0x2d958086 rev=0x02 hdr=0x00
hostb9@pci0:255:3:0:    class=0x060000 card=0x80868086 chip=0x2d988086 rev=0x02 hdr=0x00
hostb10@pci0:255:3:1:    class=0x060000 card=0x80868086 chip=0x2d998086 rev=0x02 hdr=0x00
hostb11@pci0:255:3:2:    class=0x060000 card=0x80868086 chip=0x2d9a8086 rev=0x02 hdr=0x00
hostb12@pci0:255:3:4:    class=0x060000 card=0x80868086 chip=0x2d9c8086 rev=0x02 hdr=0x00
hostb13@pci0:255:4:0:    class=0x060000 card=0x80868086 chip=0x2da08086 rev=0x02 hdr=0x00
hostb14@pci0:255:4:1:    class=0x060000 card=0x80868086 chip=0x2da18086 rev=0x02 hdr=0x00
hostb15@pci0:255:4:2:    class=0x060000 card=0x80868086 chip=0x2da28086 rev=0x02 hdr=0x00
hostb16@pci0:255:4:3:    class=0x060000 card=0x80868086 chip=0x2da38086 rev=0x02 hdr=0x00
hostb17@pci0:255:5:0:    class=0x060000 card=0x80868086 chip=0x2da88086 rev=0x02 hdr=0x00
hostb18@pci0:255:5:1:    class=0x060000 card=0x80868086 chip=0x2da98086 rev=0x02 hdr=0x00
hostb19@pci0:255:5:2:    class=0x060000 card=0x80868086 chip=0x2daa8086 rev=0x02 hdr=0x00
hostb20@pci0:255:5:3:    class=0x060000 card=0x80868086 chip=0x2dab8086 rev=0x02 hdr=0x00
hostb21@pci0:255:6:0:    class=0x060000 card=0x80868086 chip=0x2db08086 rev=0x02 hdr=0x00
hostb22@pci0:255:6:1:    class=0x060000 card=0x80868086 chip=0x2db18086 rev=0x02 hdr=0x00
hostb23@pci0:255:6:2:    class=0x060000 card=0x80868086 chip=0x2db28086 rev=0x02 hdr=0x00
hostb24@pci0:255:6:3:    class=0x060000 card=0x80868086 chip=0x2db38086 rev=0x02 hdr=0x00



Do you think it might be worth moving them to another PCI port to see if there is any change?
I also have a couple old Intel 10GbE XF SR EXPX9501AFXSR eithernet cards. One is in my VMWare server atm. I could put them in place of these cards and see if the results change. If they do I'll send these Chelsio's back and replace them with some other 10Gb cards as I really don't want to run those power hungry Intel Cards I have all the time. I'm burning enough eletrons as it is.
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
BTW, good looking out, I didn't see that until you pointed it out ;) Thank you!
 

BigDave

FreeNAS Enthusiast
Joined
Oct 6, 2013
Messages
2,479
Do you think it might be worth moving them to another PCI port to see if there is any change?
I also have a couple old Intel 10GbE XF SR EXPX9501AFXSR eithernet cards. One is in my VMWare server atm. I could put them in place of these cards and see if the results change. If they do I'll send these Chelsio's back and replace them with some other 10Gb cards as I really don't want to run those power hungry Intel Cards I have all the time. I'm burning enough eletrons as it is.
I think it's worth a shot to change them out and re-test.
Place the 9501s in the same PCIe slots, so as not to change two things at once.

As I have just recently gotten involved with 10GbE hardware, I must admit,
trouble shooting can be a bit more complicated with the optical transceivers
in the mix. I made the investment in a SFP+ copper DAC twinax cable, as I
expecting issues like this when I started.
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
LOL, nice! Sure you do the copper and I do the optics. Heck I'd do optics to everything if I could afford it, lol, lower latency, light speed :D, ect. I just like the idea of running a laser light networking, but all in due time, maybe one of these days I'll run 4Gb to all the workstations, but I bet in the years to come everyone will be running 10Gb copper and the big thing by then will be 100Gb backbones in the home and small business, lol.

Yep, sounds like a plan. I'm going to try iperf from a live CD distro first just for poops and giggles then if I get the same results I'll put the Intel 10Gb NICs in and test again with FreeNAS, if the speeds are still slow try the live CD again, if still slow I'll move slots and test again.
So, it'll be a bit before all this is done but I'll post back my results and I'll be sure to keep logs of the results and changes. Chances are I won't do it today. Going to chill a bit today and change the oil on the wife's van, so likely later this week.
If I find those Chelsio cards look to be the problem then they are definitely heading back to where I bought them and I'll find some other module cards to replace them. I'd really like to keep the 10Gb fiber and do more fiber stuff in the months to come which means I'll also need more USB 3 and 3.1 devices to try and use the 10Gb as much as possible, lol :P
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Heck I'd do optics to everything if I could afford it, lol, lower latency, light speed :D, ect. I just like the idea of running a laser light networking,

FQTMNV9I2KBQQ2E.LARGE.jpg


but all in due time, maybe one of these days I'll run 4Gb to all the workstations, but I bet in the years to come everyone will be running 10Gb copper and the big thing by then will be 100Gb backbones in the home and small business, lol.

I don't think that's true. What we've seen so far is that in the period of 1993, 1996, 1999, 2003, that ethernet went from 10M to 100M to 1G to 10G, with commodity hardware availability trailing out a few years. Except for the 10G jump. Where it's 2016 and we're still not seeing cheap 10G. I think this is basically because we hit a point where networking was generally "fast enough" and that next jump to 10G might very well be *it* for consumer networking for at least another decade.
 

Mirfster

Doesn't know what he's talking about
Joined
Oct 2, 2015
Messages
3,215
Could it possibly be needing a firmware update? While doubtful, I figured I would throw that out there.

I have Intel 10GB (Mezzanine), Chelsio 10GB 2-Port PCI-e (110-1088-30 Standard) and QLogic QLE8152 cards. If needed can grab some comparison info. Well maybe not for the QLogic QLE8152, I don't think the drivers are there for FreeNas; I use them for my ESXi systems...
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
Yea, if you would, that would be great!

I've been running "iperf -s" (default) and "iperf -s -w 512k" and sometimes with a -P 2 or 4

So far the fastest I'm seeing is about 3.75 Gb/s one direction and about 2Gb/s the other which got me thinking today, maybe I have dirty lenses? So I've been looking into ways to properly clean the modules as the cables are pretty easy to clean with a lint free cloth and some isopropyl, basically wet, then dry GENTLY! lol and did some digging on cleaning modules but not much as I got busy with some house hold duties.

I still haven't had a chance to run tests with a live CD. So still have much on my list to do.


Sorry for the delayed reply. On the road. I'll check in on your other thread.
Thanks bud, I appreciate it and definitely appreciate you and everyone else here being so patient with me. Fiber and 10Gb anything is new to me and I'd believe my skills are spread so thin over such a wide area that I'm just unable to master any one area.
I know at times I may come across ignorant and at times it likely is the case but at least I know it :D

Could it possibly be needing a firmware update? While doubtful, I figured I would throw that out there.
That is possible. I haven't really looked into that either, nor given it a thought. Something else I may need to consider. Thanks for throwing that out there!

I don't think that's true. What we've seen so far is that in the period.....
Very true! I think 10Gb will come to homes since we are now seeing more and more machines that are able to handle 10Gb and with 1Gb internet connections coming to the home, but it is still likely a ways out, but then following that it will likely be much longer before we see further upgrades unless quantum computing starts coming to the homes too, but I don't see that happening in our lifetimes for reasons I care not to debate :D
I mean heck, look how long we were on 10/100 until we started seeing 1Gb as a standard. Heck, I'm still swapping switches out here and there that are 10/100, lol!

I think running fiber is fun in it's own way and a GREAT learning experience as frustrating as it is at times!

Anyhow, I'm going to get back at it, I have some other stuff I need to get done, a network monitoring application to troubleshoot, ect.

As soon as I know more I'll be posting it up. I'd really like to see the connection capable of 8, 9, preferably 10+Gb/s, even if I don't fully use it, it'll give me further knowledge, experience, bragging rights, : D ect.
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
maybe I have dirty lenses? So I've been looking into ways to properly clean the modules as the cables are pretty easy to clean with a lint free cloth and some isopropyl, basically wet, then dry GENTLY! lol and did some digging on cleaning modules but not much as I got busy with some house hold duties.

You can always check the ethernet interface for errors. The usual non-fatal communications issues should probably show up in the "errs" columns of "netstat 10" ... run it on both sides at the same time. No errors == no good reason to clean. Also unless you're dragging the ends through peanut butter you're not likely to see any dirt problems on a short range cable. The signal's usually just too strong for that.

Fiber and 10Gb anything is new to me and I'd believe my skills are spread so thin over such a wide area that I'm just unable to master any one area.
I know at times I may come across ignorant and at times it likely is the case but at least I know it :D

The ability to develop basic competence in the areas of your choosing is an excellent life skill. Look around you at the people who don't have it. And then be happy.

I mean heck, look how long we were on 10/100 until we started seeing 1Gb as a standard.

Only about three to four years. Gig was cheap and very standard by ~2004-2005. And, unlike 10/100, it had much better interoperability because vendors weren't busy stupiding it to death.

Heck, I'm still swapping switches out here and there that are 10/100, lol!

Well, sure, you can still buy a 10/100 switch, and people actually do. I have no flippin' clue why they do, though.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
It's going to take a lot more people jumping into 10Gbe to bring the prices down. I just pulled 400M of 12 strand singlemode to our data center across the street and the 10Gbe sfp+ LR modules run about $1400 a piece. The upgrade will be worth it though since we will be able to share our Nimble array with the other vmware cluster.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
It's going to take a lot more people jumping into 10Gbe to bring the prices down. I just pulled 400M of 12 strand singlemode to our data center across the street and the 10Gbe sfp+ LR modules run about $1400 a piece. The upgrade will be worth it though since we will be able to share our Nimble array with the other vmware cluster.

That's part of why I was disappointed Apple didn't include 10G on the Mac Pro Trashcan.

You couldn't have whacked 100M off that and gone with OM3 SR? I suppose the LR4 option for 40/100 was the decider there, but good lord, $1400 optics... hopefully they came in gold and diamond casings.
 

Mlovelace

Guru
Joined
Aug 19, 2014
Messages
1,111
That's part of why I was disappointed Apple didn't include 10G on the Mac Pro Trashcan.

You couldn't have whacked 100M off that and gone with OM3 SR? I suppose the LR4 option for 40/100 was the decider there, but good lord, $1400 optics... hopefully they came in gold and diamond casings.
Unfortunately I'm stuck buy Extreme Networks branded transceivers because my manager is afraid of invalidating our service contract on the X670v-48X switches by using "unsupported" modules. I can get a "compatible" transceiver for about $300 but it was a non-starter... I've learned to pick my battles.
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
non-fatal communications issues should probably show up in the "errs" columns of "netstat 10"
Good to know! See didn't know that either!!! Learning lots here. I'm seriously thinking I need to dig into some BSD books. Probably wouldn't be a bad idea since I'm going to stop running as much MS-Crap here as I can.

Look around you at the people who don't have it. And then be happy.
Oh! I couldn't agree more! My skills are VERY handy, I can repair cars, repair the house, repair computers, networks, I'm a ham radio operator that wires up antennas, connects things together, ect. I have NO programming skills, but if it can talk I can RTFM and dig around the web to figure out how to make it talk or fly by the seat of my pants and figure it out. I was raised on welfare and REFUSE to go back to it. I'm self employed, granted I don't make a lot but I do better than I would flipping burgers! by far! So I agree and am happy and at the same time not, I wish I could know and learn more faster.

Well, sure, you can still buy a 10/100 switch, and people actually do. I have no flippin' clue why they do, though.
HA! so true! Then they call people like me and say, "It's so slow, why?!?!" Well you're running crap, get some decent equipment and WAMMIE! LOL. I did that for a client recently, replaced the 10/100 cisco switch with a 1Gb switch, swapped out the server with a server with some SSDs and more RAM, swapped out some workstation drives with SSDs, yea, they are happy peeps now, lol! Not to mention replaced their Sonic Wall with a PfSense box, lol!!! :D

Unfortunately I'm stuck buy Extreme Networks branded transceivers because my manager is afraid of invalidating our service contract on the X670v-48X switches by using "unsupported" modules. I can get a "compatible" transceiver for about $300 but it was a non-starter... I've learned to pick my battles.
Well better safe than sorry I guess and if you have the money why not? Right?


No errors == no good reason to clean
Almost forgot to reply to this. I fired up the netstat and yea, no errors, the switch even showed no errors on the interfaces and I only have 3ft (1 Meter) jumpers, all is good but for some stupid reason I still am not able to get more than 3.75Gb one direction. The other direction I'm seeing 2Gb and it's driving me crazy!!!! I've done iperf against the machine itself and it shows a potential of something like 48 or 49Gb/s. I've even dd'd the drives and they are capable of 29 to 39Gb/s with encryption enabled. I've tried iperf to a windows virtual machine,... yea that was funny, 1.2Gb/s TOP speed, lol!!

So I don't know what's going on. I still haven't shut down the machines and loaded a live CD yet and tried something like a direct connect. I should do that right now while I'm thinking about it now that I'm done transferring data on the server.

I even tried turning STP off on the switch, that made no difference.
 

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546

Visseroth

Guru
Joined
Nov 4, 2011
Messages
546
Code:
 iperf -s -w 512k
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size:  512 KByte
------------------------------------------------------------
[  4] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 47955
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  2.39 GBytes  2.06 Gbits/sec
[  5] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 48137
[  5]  0.0-10.0 sec  2.34 GBytes  2.01 Gbits/sec
[  4] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 48660
[  4]  0.0-10.0 sec  2.38 GBytes  2.04 Gbits/sec
[  5] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 48666
[  5]  0.0-10.0 sec  2.53 GBytes  2.17 Gbits/sec
[  4] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 48673
[  4]  0.0-10.0 sec  2.46 GBytes  2.11 Gbits/sec
[  5] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 48682
[  5]  0.0-10.0 sec  2.11 GBytes  1.81 Gbits/sec
[  4] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 48688
[  4]  0.0-10.0 sec  2.52 GBytes  2.16 Gbits/sec
[  5] local 10.10.10.252 port 5001 connected with 10.10.10.242 port 48694
[  5]  0.0-10.0 sec  2.46 GBytes  2.11 Gbits/sec
^C[patrick@storage] ~% iperf -c 10.10.10.242 -w 512k
------------------------------------------------------------
Client connecting to 10.10.10.242, TCP port 5001
TCP window size:  513 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 10.10.10.252 port 34563 connected with 10.10.10.242 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.55 GBytes  1.33 Gbits/sec
[patrick@storage] ~% iperf -c 10.10.10.242 -w 512k
------------------------------------------------------------
Client connecting to 10.10.10.242, TCP port 5001
TCP window size:  513 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 10.10.10.252 port 45233 connected with 10.10.10.242 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.67 GBytes  2.29 Gbits/sec
[patrick@storage] ~% iperf -c 10.10.10.242 -w 512k
------------------------------------------------------------
Client connecting to 10.10.10.242, TCP port 5001
TCP window size:  513 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 10.10.10.252 port 12522 connected with 10.10.10.242 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.81 GBytes  2.41 Gbits/sec
[patrick@storage] ~% iperf -c 10.10.10.242 -w 512k
------------------------------------------------------------
Client connecting to 10.10.10.242, TCP port 5001
TCP window size:  513 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 10.10.10.252 port 32862 connected with 10.10.10.242 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.73 GBytes  2.34 Gbits/sec
[patrick@storage] ~% iperf -c 10.10.10.242 -w 512k
------------------------------------------------------------
Client connecting to 10.10.10.242, TCP port 5001
TCP window size:  513 KByte (WARNING: requested  512 KByte)
------------------------------------------------------------
[  3] local 10.10.10.252 port 16466 connected with 10.10.10.242 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  2.90 GBytes  2.49 Gbits/sec



Code:
          input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
      2102     0     0    2187008       5227     0     462058     0
      2048     0     0    2228616       5056     0     444516     0
      2355     0     0    2360035       5492     0     549816     0
      2029     0     0    2251171       5243     0     454112     0
      2055     0     0    2197448       5270     0     486256     0
      2092     0     0    2262751       5192     0     453736     0
      2177     0     0    2267013       5353     0     461054     0
      3128     0     0    2776384       6804     0    1908736     0
    918907     0     0 1468817502    1889823     0  128488945     0
   1854449     0     0 2812706877    3712691     0  252547040     0
   1462510     0     0 2205285477    2919236     0  198836961     0
   1707247     0     0 2586128201    3415587     0  232354116     0
   1404497     0     0 2124122053    2809390     0  193991948     0
   1733668     0     0 2623438049    3466992     0  236126973     0
   1680264     0     0 2472415667    3314011     0  225505310     0
      2017     0     0    2257640       4942     0     573177     0
      2021     0     0    2296165       4972     0     411236     0
      2115     0     0    2241965       5328     0     465548     0
      2076     0     0    2233707       5204     0     446396     0
      2063     0     0    2248003       5249     0     465614     0
      2066     0     0    2249874       5377     0     449858     0
            input        (Total)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
      1998     0     0    2099542       5098     0     429360     0
      1949     0     0    2158867       4926     0     422246     0
     55749     0     0    5884818    1257157     0 1898322156     0
     64876     0     0    6204883    2306732     0 3489941006     0
     55338     0     0    5849477    3333666     0 5047063190     0
     60244     0     0    6205127    3588115     0 5432757994     0
     67531     0     0    6607482    3991234     0 6044028238     0
     60697     0     0    6403260    3593152     0 5440247010     0
     13612     0     0    2821564     715338     0 1077775318     0
      1911     0     0    2039294       4643     0     393750     0
      1896     0     0    2101366       4741     0     413018     0
      1852     0     0    2066172       4674     0     387768     0
      1890     0     0    2087927       4781     0     399100     0
      1825     0     0    2037523       4516     0     378190     0
      1884     0     0    2007203       4640     0     399552     0
      1914     0     0    2058175       4691     0     389192     0


See... Slow... No errors, WTF mate? LOL
MTU on my whole network is set to 1518, I guess I could enable jumbo frames but not all the machines are configured for jumbo frames.
 
Status
Not open for further replies.
Top