Intel X540-T2 10Gb unsupported for FreeNAS 9.3

[jgreco]

Can someone with a Chelsio card confirm that these things come up as media: full-duplex instead of autoselect (full-duplex)?

 

[Darren Myers]

I can confirm the T420-CR i have is running at full duplex

 

[paulatmig]

Those are the optics I'm using (the actual Chelsio one). The Dell one you'd mentioned first I don't have, but if the Chelsio ones work I'm not going to go out of my way to try Dell optics. The cable I have is http://belkinbusiness.com/ca/products/f2f402ll-02m-om3-duplex-fiber-optic-cable-lclc50125-2-meters; that's the same specifications as the other cable. Also had some 62.5/125 cable as well, no dice their either.

 

[Mlovelace]

Can someone with a Chelsio card confirm that these things come up as media: full-duplex instead of autoselect (full-duplex)?

Code:

cxl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=6c07bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:07:43:2c:72:e0
        inet 10.97.68.58 netmask 0xffffff00 broadcast 10.97.68.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-Twinax <full-duplex>
        status: active

 

[Mlovelace]

Is it missing the gateway for that subnet? The no carrier leads me to think the gateway is missing...

 

[Darren Myers]

Are the cards recognized in FreeNAS? Have you tried changing what slot the card is in?

 

[jgreco]

Is it missing the gateway for that subnet? The no carrier leads me to think the gateway is missing...

What would that have to do with anything? An IP network does not require a gateway.

 

[Mlovelace]

What do you get back if you run

Code:

netstat -rn

 

[jgreco]

Code:

        media: Ethernet 10Gbase-Twinax <full-duplex>

That seems like a presentation anomaly but if that's what it does, that's what it does. I'm wondering what happens if the poster reporting the problem tries toggling off autonegotiation on the switch side... this kind of reeks of some sort of hardware layer issue.

 

[paulatmig]

Yes, confirmed that the card is recognized in FreeNAS - they just show Media Status: No Carrier. I did swap PCI-E bays with the card to a known-working bay (the one I originally had the Intel x520 in), and put it in a completely unused bay as well, no luck. Netstat -rn:

Internet:
Destination Gateway Flags Refs Use Netif Expire
default 172.16.0.58 UGS 0 1276 igb2
10.10.20.0/24 link#11 U 0 0 cxgbe0
10.10.20.25 link#11 UHS 0 41 lo0
10.10.25.0/24 link#12 U 0 0 cxgbe1
10.10.25.30 link#12 UHS 0 41 lo0
127.0.0.1 link#10 UH 0 14470 lo0
172.16.0.0/23 link#3 U 0 3211 igb2
172.16.0.230 link#3 UHS 0 41 lo0

Internet6:
Destination Gateway Flags Netif
Expire
::/96 ::1 UGRS lo0
::1 link#10 UH lo0
::ffff:0.0.0.0/96 ::1 UGRS lo0
fe80::/10 ::1 UGRS lo0
fe80::%lo0/64 link#10 U lo0
fe80::1%lo0 link#10 UHS lo0
ff01::%lo0/32 ::1 U lo0
ff02::/16 ::1 UGRS lo0
ff02::%lo0/32 ::1 U lo0

 

[Darren Myers]

That seems like a presentation anomaly but if that's what it does, that's what it does. I'm wondering what happens if the poster reporting the problem tries toggling off autonegotiation on the switch side... this kind of reeks of some sort of hardware layer issue.

That's what my next question was, are these connected to a switch? Mine is direct connections between cards (daisy chained if you will)

 

[paulatmig]

Will give the switch port a toggle and see what it do.

 

[jgreco]

That's what my next question was, are these connected to a switch? Mine is direct connections between cards (daisy chained if you will)

Daisy chained, with bridging configured at each node, let's re-invent 10base2 for the 10G era ... or better yet token ring ;-)

 

[Mlovelace]

In my experience the switch ports have to be configured to speed 10000 duplex full. 10gbe doesn't usually auto-negotiate speed/duplex/flow control.

Code:

enable flow-control tx-pause port 48
configure ports 48 auto off speed 10000 duplex full

 

[Mlovelace]

I was just curious about the gateway. There was an old bug where it could drop the gateway after a reboot and loose the connection.

 

[paulatmig]

Ah hah! The switch's firmware (Brocade ICX6610) was flashed a month back and the configuration was wiped - the 1/10gb optical ports were set back to their default... 1Gb. Apparently the Intel NIC is fine toggling between the two, but the Chelsio might not like operating at 1Gb? Anyway, thanks for the switch recommendation and all your guys' help!

 

[Darren Myers]

Daisy chained, with bridging configured at each node, let's re-invent 10base2 for the 10G era ... or better yet token ring ;-)

well it was originally for just 2 devices, but since my current raid array is failing i figured i'd make the migration to ESXi. So the plan now is 2 ESXi boxes connected to the FreeNAS server, and then the other 2 ports on each ESXi box are connected to a SAN. I'm looking at 10GB switches now as we speak actually because it is a pain in the butt lol

 

[jgreco]

In my experience the switch ports have to be configured to speed 10000 duplex full. 10gbe doesn't usually auto-negotiate speed/duplex/flow control.

Code:

enable flow-control tx-pause port 48
configure ports 48 auto off speed 10000 duplex full

It's pretty sad that 20 years after the introduction of autonegotiate that we haven't been able to make it work reliably.

10GbE is definitely supposed to do autonegotiation reliably and it is the first speed tier where I've seen a significant number of switches that refuse to allow manual configuration of its status. Since everything's full duplex these days, anyways, ... geez.

 

[jgreco]

well it was originally for just 2 devices, but since my current raid array is failing i figured i'd make the migration to ESXi. So the plan now is 2 ESXi boxes connected to the FreeNAS server, and then the other 2 ports on each ESXi box are connected to a SAN. I'm looking at 10GB switches now as we speak actually because it is a pain in the butt lol

There's a sticky on 10G stuff at the top of the forum if that's helpful.

 

[Jimm Chen]

Add some info here. In this post, I said I have two DELL 13G servers interconnected(direct CAT-6 cable, no switch) with 10Gbps Intel X540-T2 . And sometime is strange.

The good side: Installing FreeNAS 9.3 (20150604) on physical server A, X540 works very well. The ESXi 6.0.0(B) acts as iSCSI client with 10Gbps speed.

The bad side: If I install ESXi 6 on server A and run the same FreeNAS version as a VM inside, PCI passthrough Intel X540 to the VM(a total of two passthrough devices, one for each ethernet port on X540), then, freenas VM recognize the two devices as ix0 and ix1, but the link status is "no carrier" and I cannot ping the other server.

Sigh, don't know how to cope with it. So my only choice now is to have FreeNAS VM use vmxnet3 virtual NIC. The vmxnet3 way exhibits only half IOPS with 64KB block(from iSCSI client's perspective), so its suboptimal.


# ifconfig
em0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=9b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM>
ether 00:0c:29:04:df:5a
inet 172.27.222.74 netmask 0xffff0000 broadcast 172.27.255.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
vmx3f0: flags=8c02<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=403bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,VLAN_HWTSO>
ether 00:0c:29:04:df:64
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet 10Gbase-T
status: active
ix0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
ether a0:36:9f:75:1d:98
inet 10.172.27.74 netmask 0xffffff00 broadcast 10.172.27.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
ix1: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=407bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO>
ether a0:36:9f:75:1d:9a
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
ipfw0: flags=8800<SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x6
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>