SOLVED Heck of a time with 10GB - Autoneg at 100MBits?

[Shifty One]

As the title says... I have having a heck of a time getting ix0/ix1 10GBit working here. I had it working, forgot what I did, upgraded ram, rebooted... and lost my changes. I "think" that I used ifconfig to force some NIC settings.... I did not document this... After I upgraded from 32 to 64GB, I am back to 100MBits.

I have tried hard setting the interface to 10GB on both Switch and interface. Doesnt help. `history` seems to be lost between sessions so I cant see what I did.

ifconfig output:
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: connected to 10GB-Agg (Port 3) options=e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether d0:50:99:dd:59:d7 inet 10.1.150.22 netmask 0xffffff00 broadcast 10.1.150.255 media: Ethernet autoselect (100baseTX <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED>

I have a Ubiquity Aggregate 8-port 10GB switch. I am using (or tried) both Copper 10GB SFP+ devices, Models QSFPTEK QT-SFP-10G-T and 10Gtek ASF-10G-T. The QSFPEK claim to be compatible with Ubiquity but the 10Gtek are listed as compatible with Ubiquity (on their page). I have had both SFPs workign with other Intel NICs and Free/TrueNAS.

Switch currently thinks the port is in 10GB mode:

Status 10,000 FDX
Part: SFP-10G-SR
Compliance: 10G Base-SR

TrueNAS thinks otherwise... Media Subtype: 100baseTX

In the past, I "think" i used ifconfig to force it to 10GB... Obviously this did not persist past a reboot.

NIC Details:

ix0@pci0:65:0:0: class=0x020000 card=0x15631849 chip=0x15638086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Controller 10G X550T' class = network subclass = ethernet ix1@pci0:65:0:1: class=0x020000 card=0x15631849 chip=0x15638086 rev=0x01 hdr=0x00 vendor = 'Intel Corporation' device = 'Ethernet Controller 10G X550T' class = network subclass = ethernet

Motherboard/CPU: (Memory: 4 16GB sticks, all the same part number)

Base Board Information Manufacturer: ASRockRack Product Name: EPYCD8-2T Processor Information Socket Designation: CPU1 Type: Central Processor Family: Zen Manufacturer: Advanced Micro Devices, Inc. Version: AMD EPYC 7252 8-Core Processor Voltage: 1.1 V External Clock: 100 MHz Max Speed: 3200 MHz Current Speed: 3100 MHz Status: Populated, Enabled Upgrade: Socket SP3 Memory Device Array Handle: 0x0023 Error Information Handle: 0x002D Total Width: 72 bits Data Width: 64 bits Size: 16384 MB Form Factor: DIMM Set: None Locator: DIMM 0 Bank Locator: P0 CHANNEL B Type: DDR4 Type Detail: Synchronous Registered (Buffered) Speed: 2666 MT/s Manufacturer: SK Hynix Serial Number: REDACTED Asset Tag: Not Specified Part Number: HMA82GR7AFR8N-VK Rank: 2 Configured Memory Speed: 2666 MT/s Minimum Voltage: 1.2 V Maximum Voltage: 1.2 V Configured Voltage: 1.2 V

 

[Shifty One]

Sorry, running TrueNAS-12.0-U2.1. About to update to U8.1... Maybe even just Scale just to move from BSD to Debian....

 

[Shifty One]

Went ahead and moved to Scale from Core... Same result.

enp65s0f0​

• LINK STATE UP
• 
  
• • In: 0.39 KiB/s
  • Out:0.40 KiB/s
• Media Type: Ethernet
• Media Subtype: 100Mb/s Twisted Pair
• IP Address: 10.1.150.22/24 (+1)

 

[Ericloewe]

Have you done actual tests? It could be just a reporting bug, especially with SFP+ to twisted-pair adapters in the mix.

 

[Shifty One]

Oh, it is 100MBits for sure... I have a few VMs running in a SSD pool.. They are not happy at all. I dont think I have seen over 9 MBytes/sec on this interface since the reboot.

Here is ethtool output after moving to Core from Scale.

root@Shift-NAS[~]# ethtool enp65s0f0
Settings for enp65s0f0:
Supported ports: [ TP ]
Supported link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
2500baseT/Full
5000baseT/Full
Supported pause frame use: Symmetric
Supports auto-negotiation: Yes
Supported FEC modes: Not reported
Advertised link modes: 100baseT/Full
1000baseT/Full
10000baseT/Full
Advertised pause frame use: Symmetric
Advertised auto-negotiation: Yes
Advertised FEC modes: Not reported
Speed: 100Mb/s
Duplex: Full
Auto-negotiation: on
Port: Twisted Pair
PHYAD: 0
Transceiver: internal
MDI-X: Unknown
Supports Wake-on: umbg
Wake-on: g
Current message level: 0x00000007 (7)
drv probe link
Link detected: yes

EDIT: I do not want to seem off here. I have linux experience, mostly in the RHEL world for work. Home Desktop runs Kubuntu. I am open to any and all ideas/testing.

post upgrade to Scale... here is a simple `dd` test. Desktop is an Asus something with an Intel X540-AT2 card. Both switch and Desktop report 10GB connectivity. I am working to test dd from desktop to NAS, but now I am getting error "521" when mounting the NFS after upgrading to Scale...

 

[Shifty One]

Just made a 1GB test file using dd on my plex box.

dd if=/dev/urandom of=./test_file_1gb bs=1M count=1000

SCPed that over to the NAS (Plex is a HP Eleite Desk at 1GBit).

It is transfering at the max of ~9.3MBytes/sec. It took a total of 1m46s to copy the 1gb test file.

 

[Morris]

Have you tried different jumper cables? What is the distance and type of cable?

 

[joeschmuck]

What if you manually force the NIC to 10000baseT/Full using ifconfig?

 

[Shifty One]

Have you tried different jumper cables? What is the distance and type of cable?

But Did, but need more testing... I broke out my TDR and tested a few of my spare CableMatters 5-10ft CAT6 patch cables. At least one tested bad across the board (attenuation, pin out, etc... all bad)

What if you manually force the NIC to 10000baseT/Full using ifconfig?

I "think" this is what I did in the past. However when I forced it to 10000Base-T (before I upgraded to Scale), I no longer even connected. I want to test this again now that I am on Debian based Scale.

 

[Shifty One]

Just going to copy/paste my comment from my reddit post... Cabling strikes again!

So, like so many other 10GB posts..... I did fall into the same category... After running my TDR on a few spare cables, I did have a few bad cabled. I just swapped with a PM-rated CAT6A cable that I premade a bit ago... That cable tested OK. Plugged it in... Voila. 10GB!
I dont know exactly how cables just go bad in a controlled environment. Maybe if this was in the garage, corrosion on connectors, etc....
Now, the cable that was in place, CableMatters CAT6 CM-rated patch cable. This did pass the test. Wiremap, Attentuation, Length and NEXT (not sure what those tests are, there are 4 in total) all passed. Return Loss was "!" suspect but still passed. My pre-made also was suspect in the Return Loss.
So.... Ranting aside... Layer 1 strikes again!

 

[joeschmuck]

I dont know exactly how cables just go bad in a controlled environment.

All those tests you listed above, while great, you never mentioned a high frequency test which could show connection issues or wire damage.

So while a cable may work fine in a 1GB connection fine, it may not work in a 10GB connection just to to not having the proper frequency tolerance. Or it could be just a bad end connector.

 

[Arwen]

One note, 1Gbps & 10Gbps require all 4 pairs to be properly passed through. Miss one of the extra pairs, and the cable will cause a drop to 100Mbps, which only needs 2 pair, (like 10Mbps).

This bit me once. I had a wall box with a bad modular cable install. That link would only do 100Mbps. Since that box had 3 others that did 1Gbps, it was not a problem. (And I actually used one up-link at 10Mbps, so 2 working pairs was fine, on 1 out of the 4 cable runs.)

Of course, if your cable or connector fails on one of the 2 pairs used for 10/100Mbps, then the cable won't work at all.

 

[Shifty One]

you never mentioned a high frequency test

I did not. The batteries in my OmniScanner (now called Fluke) are dead so I didnt save the results. All checks passed what it claimed.

10Gbps require all 4 pairs to be properly passed through.

Oh for sure. And thanks for the info here! I did know about the pairs, but not how the interact at 10GB.

And thanks too all that helped in this thread! If at any time I was sounding argumentative, that was not the goal. I mean, I still dont get how a cable just goes bad... But it clearly was! Even my TDR said that.

To any one else that finds this thread in the future... Layer 1, Layer 1, Layer 1!!! Always check your cabling. If you dont have a TDR, go make/buy some cables. Heck, make 4! It is easier than buying new hardware, for sure.

 

[Arwen]

Copper cables can get kinked, breaking the copper internally. Or be pulled back from the 8p8c connector, (aka RJ-45).


I've had cables in production fail because the cable's conductor(s) got pulled back from the head, (8p8c).

In one case it was an 8 lane 1Gps trunk, so not too much impact. But, it was remote, a thousand miles from where I was. I worried about getting a new cable made, sent and installed. Got lucky, the on-site staff were skilled enough to chop off the old head and install a new one. Back in service within 1 hour after trouble shooting. Definitely high quality smart hands at that site .