10GBit problem

[spaxxilein]

Hello,

after using the HP NC523SFP card for some time and regularly experiencing network drops (system is still up, but cannot be reach or reach any other PC in the same subnet) i switched to Intel X520 with RJ45 10GBe optic. Before i used 10GB SFP+ SR optic. The optic dosnt seem to be the problem. Also we changed the other end from a HP523SFP card to a X710-T4 card.

But still the same thing happens:

after a random period of time the Link just stops working. There is no indication that it has to do with the amount of load. The optic shows no carrier in ifconfig. Removing the optic and reinserting dosnt help. Ifupdown does not help. Only a server restart on the Freenas Server fixes the issue.

Does anybody have any idea why that is?

TrueNAS-12.0-U2

Best Regards,

spaxxilein

 

[Hellione]

Hi! You changed both ends from sfp+ cards with sfp+ modules to copper connected rj45 cards without modules.
And you geht the same failure? So the cards, modules and cables are unlikely to be the problem.
What Switch is between those machines? What is your system, what your TrueNAS version, ..?

 

[spaxxilein]

Hi! You changed both ends from sfp+ cards with sfp+ modules to copper connected rj45 cards without modules.
And you geht the same failure? So the cards, modules and cables are unlikely to be the problem.
What Switch is between those machines? What is your system, what your TrueNAS version, ..?

Hello Hellione,

i changed the network card in the Freenas Server to X520 and added SFP+ Copper 10GB module. On the other end i changed to X710-T4 10GB RJ45 card. There is no router in between, the X710 is in our OPNSense Router and the Freenas is directly connected with RJ45 cable.

So yes, i changed both sides and still have the same error over time.

Best Regards,

spaxxilein

 

[Hellione]

Hmm, There is no router in between, the X710 is in our OPNSense Router, so the opensense is used as a router or switch?!
The system is unknown, the TrueNAS is unknown. With only two known network cards in unknown environment, i think no one can
help. Check opensense, or try without it by directly connecting some of the other unknown stuff :)

 

[spaxxilein]

Hmm, There is no router in between, the X710 is in our OPNSense Router, so the opensense is used as a router or switch?!
The system is unknown, the TrueNAS is unknown. With only two known network cards in unknown environment, i think no one can
help. Check opensense, or try without it by directly connecting some of the other unknown stuff :)

The OPNSense is used as a router. As i already mentioned we had the same problem before with different networking cards and without OPNSense. So i think its an issue with FreeBSD / Truenas. Thats why i ask if anybody has an idea how to fix this problem / where to look for solution.

 

[Patrick M. Hausen]

Possibly related or possibly not - there seems to be a problem with 10G Intel network interfaces on certain Supermicro systems:

SYS-E300-9D-4CN8TP

SYS-E300-9D-4CN8TP

forum.opnsense.org

The only 10G interfaces I run with TrueNAS are bnxt(4) - "Broadcom Net Extreme" or some such. They don't show this problem but have different ones

 

[spaxxilein]

Possibly related or possibly not - there seems to be a problem with 10G Intel network interfaces on certain Supermicro systems:

SYS-E300-9D-4CN8TP

SYS-E300-9D-4CN8TP

forum.opnsense.org

The only 10G interfaces I run with TrueNAS are bnxt(4) - "Broadcom Net Extreme" or some such. They don't show this problem but have different ones

Our Freenas and our OPNSense both dont run on Supermicro Hardware.

 

[Patrick M. Hausen]

Our Freenas and our OPNSense both dont run on Supermicro Hardware.

That's why I wrote "possibly related or possibly not". Some users on some Supermicro systems are experiencing reproducible failures of Intel 10G interfaces. The symptoms are precisely like yours. Complete loss of connectivity, only reboot helps.
I did not say the problem is limited to Supermicro only ...

HTH,
Patrick

 

[SweetAndLow]

What's the distance of the run? Those rj45 transceivers have a limit of 30m usually

 

[spaxxilein]

What's the distance of the run? Those rj45 transceivers have a limit of 30m usually

Cable is CAT7 ~5m long and the error occured before while we used fiber-cable as well.

Thats why i think it is Truenas/FreeBSD related issue.

 

[SweetAndLow]

Cable is CAT7 ~5m long and the error occured before while we used fiber-cable as well.

Thats why i think it is Truenas/FreeBSD related issue.

Are the transceivers coded for the manufacturer they are plugged into?

 

[spaxxilein]

Are the transceivers coded for the manufacturer they are plugged into?

Yes we are using Flexoptic transceiver for the X520 and it is coded for Intel.

 

[SweetAndLow]

Yes we are using Flexoptic transceiver for the X520 and it is coded for Intel.

What do the disconnect logs look like from the router?

 

[spaxxilein]

What do the disconnect logs look like from the router?

Code:

[Wed Feb 10 12:04:47 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Down
[Wed Feb 10 12:04:47 2021] vmbr0: port 4(enp2s0f3) entered disabled state
[Wed Feb 10 12:04:52 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[Wed Feb 10 12:04:52 2021] vmbr0: port 4(enp2s0f3) entered blocking state
[Wed Feb 10 12:04:52 2021] vmbr0: port 4(enp2s0f3) entered forwarding state
[Wed Feb 10 12:04:53 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Down
[Wed Feb 10 12:04:53 2021] vmbr0: port 4(enp2s0f3) entered disabled state
[Wed Feb 10 12:04:54 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[Wed Feb 10 12:04:54 2021] vmbr0: port 4(enp2s0f3) entered blocking state
[Wed Feb 10 12:04:54 2021] vmbr0: port 4(enp2s0f3) entered forwarding state
[Wed Feb 10 12:05:02 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Down
[Wed Feb 10 12:05:02 2021] vmbr0: port 4(enp2s0f3) entered disabled state
[Wed Feb 10 12:05:08 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[Wed Feb 10 12:05:08 2021] vmbr0: port 4(enp2s0f3) entered blocking state
[Wed Feb 10 12:05:08 2021] vmbr0: port 4(enp2s0f3) entered forwarding state
[Wed Feb 10 12:05:37 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Down
[Wed Feb 10 12:05:37 2021] vmbr0: port 4(enp2s0f3) entered disabled state
[Wed Feb 10 12:05:37 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Up, 10 Gbps Full Duplex, Flow Control: None
[Wed Feb 10 12:05:37 2021] vmbr0: port 4(enp2s0f3) entered blocking state
[Wed Feb 10 12:05:37 2021] vmbr0: port 4(enp2s0f3) entered forwarding state
[Wed Feb 10 12:06:02 2021] i40e 0000:02:00.3 enp2s0f3: NIC Link is Down
[Wed Feb 10 12:06:02 2021] vmbr0: port 4(enp2s0f3) entered disabled state

And after that it stayed down and didnt came back up again.

 

[SweetAndLow]

Sure looks like a bad cable or transceiver to me.

 

[spaxxilein]

Sure looks like a bad cable or transceiver to me.

Both transceivers fail at the same time. Tonight again, both network connection (1 to our PFSense, 1 directly to another host) failed. I dont think its an issue with the transceiver. Very strange issue - i dont really know how to get this fixed.