10Gbe woes - and upgrade time!

[Matthias]

Hi everyone!

Several months have passed with my 'new' build, getting accustomed to Truenas Core, and it's already time for some upgrades!

(hardware before all the upgrades)

I bought 3 additional Exos X16's, a Xeon-e 2244G, an X520 NIC and upgraded to 128GB ECC RAM.

The first to optimize the usable space in the pool and I added a hot spare.
(So i went from 3 data disks + 2 parity disks to 5 data, 2 parity disks and a hot spare. Transitioned using the hot spare flawlessly with the replication function!)

The second purely for Plex/transcoding purposes. (damn those prices are still high for these Xeons.. )

The third, 1GB recommended per TB storage, so bit the bullet there too

The last but not least on my 'nice to have list', is the X520 NIC.. as recommended here all over Truenas forum, which is giving me a headache at the moment..
I just can't get it to work.

So, both X520's (i have the exact same in my desktop running fine) were tested, interchanged, the wires checked, swapped trancievers, etc.. still couldn't get it to work.

I'm not quite sure where exactly the culprit lies, but i think it is related to Truenas or my config at least..

So, I am aware that running a copper Gbe and Fiber 10Gbe simultanious is not supported.

My switch shows me that the uplink is there (Mikrotik CRS326):

If i swap the leads of the fiber, no lights blink, and 'no link' is displayed, so i'm confident that's not it.

However, my stats page suggests the NAS side isn't transmitting anything:

Normally, in Truenas, 'link state' should be up even if there is no IP, if i'm correct?

But i thought, that should maybe be normal becaus no IP is set?

So i went and deleted the IP form the Gbe interface, applied and copied over on the IX3's interface page. (also tried all other IX interfaces..)
Hit apply changes and test for 60s, but no success.

Also tried pulling out the copper connection from the NAS to the switch, if for what ever reason that could interfere with assigning the address.

Then tried through the command line to get things running or changing the network config, but i'm failing to get results..

ifconfig results below:

Code:

Warning: the supported mechanisms for making configuration changes
are the TrueNAS WebUI and API exclusively. ALL OTHERS ARE
NOT SUPPORTED AND WILL RESULT IN UNDEFINED BEHAVIOR AND MAY
RESULT IN SYSTEM FAILURE.

root@truenas[~]# ifconfig
ix0: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 6c:b3:11:65:78:34
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
ix1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 6c:b3:11:65:78:36
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
ix2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether d0:50:99:fe:5b:3b
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
ix3: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether d0:50:99:fe:5b:3a
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb0: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4a120b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,WOL_MAGIC,VLAN_HWFILTER,RXCSUM_IPV6,NOMAP>
        ether d0:50:99:fe:5b:3c
        inet 192.168.1.33 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (1000baseT <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
igb1: flags=8822<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e527bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether d0:50:99:fe:5b:3d
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
        options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        inet6 ::1 prefixlen 128
        inet6 fe80::1%lo0 prefixlen 64 scopeid 0x7
        inet 127.0.0.1 netmask 0xff000000
        groups: lo
        nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>
pflog0: flags=0<> metric 0 mtu 33160
        groups: pflog
ue0: flags=8802<BROADCAST,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 6e:7f:8a:e9:cd:93
        nd6 options=9<PERFORMNUD,IFDISABLED>
bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        ether 58:9c:fc:10:39:17
        id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
        maxage 20 holdcnt 6 proto rstp maxaddr 2000 timeout 1200
        root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
        member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 12 priority 128 path cost 2000
        member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 11 priority 128 path cost 2000
        member: igb0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
                ifmaxaddr 0 port 5 priority 128 path cost 20000
        groups: bridge
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.1: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: PlexJail as nic: epair0b
        options=8<VLAN_MTU>
        ether d2:50:99:3d:c3:2d
        hwaddr 02:ad:79:de:36:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.2: flags=8963<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: adguardhomeJail as nic: epair0b
        options=8<VLAN_MTU>
        ether d2:50:99:76:4d:54
        hwaddr 02:38:e3:62:b7:0a
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
vnet0.3: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: associated with jail: syncthing as nic: epair0b
        options=8<VLAN_MTU>
        ether d2:50:99:5b:07:f8
        hwaddr 02:f3:99:cc:93:0a
        inet 172.16.0.1 netmask 0xfffffffc broadcast 172.16.0.3
        groups: epair
        media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>

I'f i'm not mistaken, "no carrier" means no link.. so either the cable is faulty (checked) or the tranciever is (also checked)..
Fiber is fiber, but trancievers are tricky.

As I read through the plethora of these questions and many suggestions, i bought FS.com trancievers and my desktop and NAS side are running these. The switch side works with either Mikrotik or the Uptimed trancievers.

Desktop runs fine.

How should I proceed to continue to investigate this problem?

Thanks in advance everyone!

 

[Matthias]

lspci output:

Code:

root@truenas[~]# lspci
00:00.0 Host bridge: Intel Corporation 8th Gen Core 4-core Workstation Processor Host Bridge/DRAM Registers [Coffee Lake S] (rev 07)
00:01.0 PCI bridge: Intel Corporation 6th-10th Gen Core Processor PCIe Controller (x16) (rev 07)
00:01.1 PCI bridge: Intel Corporation Xeon E3-1200 v5/E3-1500 v5/6th Gen Core Processor PCIe Controller (x8) (rev 07)
00:02.0 VGA compatible controller: Intel Corporation CoffeeLake-S GT2 [UHD Graphics P630]
00:08.0 System peripheral: Intel Corporation Xeon E3-1200 v5/v6 / E3-1500 v5 / 6th/7th/8th Gen Core Processor Gaussian Mixture Model
00:12.0 Signal processing controller: Intel Corporation Cannon Lake PCH ThermalController (rev 10)
00:14.0 USB controller: Intel Corporation Cannon Lake PCH USB 3.1 xHCI Host Controller (rev 10)
00:14.2 RAM memory: Intel Corporation Cannon Lake PCH Shared SRAM (rev 10)
00:15.0 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #0 (rev 10)
00:15.1 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH Serial IO I2C Controller #1 (rev 10)
00:16.0 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller (rev 10)
00:16.1 Communication controller: Intel Corporation Device a361 (rev 10)
00:16.4 Communication controller: Intel Corporation Cannon Lake PCH HECI Controller #2 (rev 10)
00:17.0 SATA controller: Intel Corporation Cannon Lake PCH SATA AHCI Controller(rev 10)
00:1b.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #17(rev f0)
00:1b.4 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #21(rev f0)
00:1c.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #1 (rev f0)
00:1c.2 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #3 (rev f0)
00:1c.3 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #4 (rev f0)
00:1d.0 PCI bridge: Intel Corporation Cannon Lake PCH PCI Express Root Port #9 (rev f0)
00:1e.0 Communication controller: Intel Corporation Cannon Lake PCH Serial IO UART Host Controller (rev 10)
00:1f.0 ISA bridge: Intel Corporation Cannon Point-LP LPC Controller (rev 10)
00:1f.4 SMBus: Intel Corporation Cannon Lake PCH SMBus Controller (rev 10)
00:1f.5 Serial bus controller [0c80]: Intel Corporation Cannon Lake PCH SPI Controller (rev 10)
01:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2008 PCI-Express Fusion-MPT SAS-2 [Falcon] (rev 03)
02:00.0 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection (rev 01)
02:00.1 Ethernet controller: Intel Corporation 82599 10 Gigabit Dual Port Network Connection (rev 01)
04:00.0 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
04:00.1 Ethernet controller: Intel Corporation Ethernet Controller 10G X550T (rev 01)
05:00.0 PCI bridge: ASPEED Technology, Inc. AST1150 PCI-to-PCI Bridge (rev 04)
06:00.0 VGA compatible controller: ASPEED Technology, Inc. ASPEED Graphics Family (rev 41)
07:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)
08:00.0 Ethernet controller: Intel Corporation I210 Gigabit Network Connection (rev 03)

 

[Matthias]

Tried jugglin' with the trancievers once again, no luck.
Enabled LAGG successfully in Truenas and the switch, still no luck.

"no carrier"

-> will try and swap new cables again, and hope for the best.

 

[Matthias]

Received 3 fiber cables, problem persists..

 

[jgreco]

Sorry, I missed this earlier. While I very much like the X520 as it is an inexpensive, readily available, generally decent card, it does have two quirks, one of which I suspect is biting you.

1) It's PCIe 2, so that's not fantastic, but still a good card despite this flaw

2) The genuine Intel cards are vendor locked to require Intel optics. You may try setting a sysctl tunable for "hw.ix.unsupported_sfp" to "1" which should cause common compatible SFP's to work regardless of vendor tagging. This will NOT force ALL possible SFP's to work, just the normal ones like FTLX8571D3BCV-xx where xx != IT. If you want to avoid this problem, pick up some used FTLX8571D3BCV-IT off eBay or buy an Intel vendor tagged module from a vendor such as fs.com. This is stupid, yes, and why we warn about it somewhere deep in the 10 Gig Networking Primer.

 

[Matthias]

Sorry, I missed this earlier. While I very much like the X520 as it is an inexpensive, readily available, generally decent card, it does have two quirks, one of which I suspect is biting you.

1) It's PCIe 2, so that's not fantastic, but still a good card despite this flaw

2) The genuine Intel cards are vendor locked to require Intel optics. You may try setting a sysctl tunable for "hw.ix.unsupported_sfp" to "1" which should cause common compatible SFP's to work regardless of vendor tagging. This will NOT force ALL possible SFP's to work, just the normal ones like FTLX8571D3BCV-xx where xx != IT. If you want to avoid this problem, pick up some used FTLX8571D3BCV-IT off eBay or buy an Intel vendor tagged module from a vendor such as fs.com. This is stupid, yes, and why we warn about it somewhere deep in the 10 Gig Networking Primer.

Hi @jgreco ! Thanks for the reply.

I will try to set up the sfp.
I did per your recommendation order fs optics already so I will give these a try. ;)

I managed to get my copper sfp+ x550 to work.. but they are so damned hot..

+ Optics and light speed makes me feel like a jedi.. so those midi chlorians better work before i use the force on that x520.. XD

PS: i couldn't find a yottamark sticker :(
However, their site mentions 'intel genuine chipset'.

X520's website

So i understand that this card could be fake.. however i hope not because i'm finding it hard to aquire X520's here in Belgium, at a decent price point.

Thank you!

 

[nKk]

When you try jgreco hint check if the NIC is UP because from your previous ifconfig list only ix2 is UP and RUNNING:

ix0: flags=8822<BROADCAST,SIMPLEX,MULTICAST>
ix1: flags=8822<BROADCAST,SIMPLEX,MULTICAST>
ix2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
^ ^
ix3: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST>

If the NIC is not UP and you plug the cable there is no link.
To be easy to detect ix0, ix1... and physical ports bring UP all "ix" interfaces from console and plug the cable.

ifconfig ix0 up
...
ifconfig ix3 up

And to check if the SFP module is recognize you can use:
ifconfig -v ix0

Should have something like this:
....
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FS PN: SFP-10GSR-85 SN: G1906102586 DATE: 2019-06-09
module temperature: 24.00 C Voltage: 3.29 Volts
RX: 0.00 mW (-40.00 dBm) TX: 0.0 mW (-40.00 dBm)

 

[Matthias]

When you try jgreco hint check if the NIC is UP because from your previous ifconfig list only ix2 is UP and RUNNING:

ix0: flags=8822<BROADCAST,SIMPLEX,MULTICAST>
ix1: flags=8822<BROADCAST,SIMPLEX,MULTICAST>
ix2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
^ ^
ix3: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST>

If the NIC is not UP and you plug the cable there is no link.
To be easy to detect ix0, ix1... and physical ports bring UP all "ix" interfaces from console and plug the cable.

ifconfig ix0 up
...
ifconfig ix3 up

And to check if the SFP module is recognize you can use:
ifconfig -v ix0

Should have something like this:
....
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FS PN: SFP-10GSR-85 SN: G1906102586 DATE: 2019-06-09
module temperature: 24.00 C Voltage: 3.29 Volts
RX: 0.00 mW (-40.00 dBm) TX: 0.0 mW (-40.00 dBm)

Thank you for the suggestion sir!

Will give it a try ;)

 

[Matthias]

When you try jgreco hint check if the NIC is UP because from your previous ifconfig list only ix2 is UP and RUNNING:

ix0: flags=8822<BROADCAST,SIMPLEX,MULTICAST>
ix1: flags=8822<BROADCAST,SIMPLEX,MULTICAST>
ix2: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST>
^ ^
ix3: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST>

If the NIC is not UP and you plug the cable there is no link.
To be easy to detect ix0, ix1... and physical ports bring UP all "ix" interfaces from console and plug the cable.

ifconfig ix0 up
...
ifconfig ix3 up

And to check if the SFP module is recognize you can use:
ifconfig -v ix0

Should have something like this:
....
plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
vendor: FS PN: SFP-10GSR-85 SN: G1906102586 DATE: 2019-06-09
module temperature: 24.00 C Voltage: 3.29 Volts
RX: 0.00 mW (-40.00 dBm) TX: 0.0 mW (-40.00 dBm)

Hi!

Appareantly it changed to ix1.. :)

Code:

root@truenas[~]# ifconfig -v ix1
ix1: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 6c:b3:11:0a:24:e2
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: FS PN: SFP-10GSR-85 SN: F2034169420 DATE: 2022-03-21
        module temperature: 21.90 C voltage: 3.34 Volts
        lane 1: RX power: 0.55 mW (-2.58 dBm) TX bias: 0.00 mA

no luck setting the ip fox ix1, so back to square one!

Tried all the SFP+ modules that i have:

Code:

root@truenas[~]# ifconfig -v ix1
ix1: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 6c:b3:11:0a:24:e2
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: Uptimed PN: UP-TR-SR-CI SN: UPC22SR080033 DATE: 2022-03-04
        module temperature: 27.00 C voltage: 3.33 Volts
        lane 1: RX power: 0.59 mW (-2.27 dBm) TX bias: 0.00 mA

And:

Code:

root@truenas[~]# ifconfig -v ix1
ix1: flags=8c22<BROADCAST,OACTIVE,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 6c:b3:11:0a:24:e2
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: MikroTik PN: S+85DLC03D SN: STST85SB4601070 DATE: 2021-11-11
        module temperature: 27.28 C voltage: 3.23 Volts
        lane 1: RX power: 0.52 mW (-2.88 dBm) TX bias: 0.00 mA

No luck setting the ip, link state down/no carrier.

I did notice, when i used the Uptimed SFP module, my link state went "link on" on my switch side.
So the card seems to like their modules better.. which probably means that that specific card or brand wont be "the one" for Truenas.

So it's firmware related?
If so, any chance that it could go like the m1015 that i flashed for my array?

Thank you

 

[Matthias]

Forgot to mention that I did set that tunable like @jgreco hinted.

 

[Matthias]

also.. re-read the 10 gig primer..
Found used Chelsio S320E-LP-CR's on Ebay Germany for 50€, used with 2 10Gbemodules, 3 left..

Will try to send back 1 Uptimed X520.. after I test that S320E.

 

[nKk]

Did you force ix1 UP with:
ifconfig ix1 up

From information about SFP module it seams that the card recognize the module but no transmit power, this can be because the ix1 is not UP.

 

[Matthias]

Did you force ix1 UP with:
ifconfig ix1 up

From information about SFP module it seams that the card recognize the module but no transmit power, this can be because the ix1 is not UP.

I tried indeed, et voila!

Code:

root@truenas[~]# ifconfig -v ix1
ix1: flags=8863<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=4e53fbb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,WOL_UCAST,WOL_MCAST,WOL_MAGIC,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6,NOMAP>
        ether 6c:b3:11:0a:24:e2
        inet 192.168.1.33 netmask 0xffffff00 broadcast 192.168.1.255
        media: Ethernet autoselect (10Gbase-SR <full-duplex,rxpause,txpause>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
        plugged: SFP/SFP+/SFP28 10G Base-SR (LC)
        vendor: Uptimed PN: UP-TR-SR-CI SN: UPC22SR080033 DATE: 2022-03-04
        module temperature: 33.71 C voltage: 3.33 Volts
        lane 1: RX power: 0.61 mW (-2.14 dBm) TX bias: 6.34 mA

and:

Thanks a million!! @nKk

+ found out that you have to use the "ifconfig (enter interface) up" every time you swap sfp modules. And if I didn't dreamt it.. even when only messing with the modules on the switch side

But anywayz, things up and running, filecopy at +/-400Mib/s so i'm good

 

[Matthias]

also.. re-read the 10 gig primer..
Found used Chelsio S320E-LP-CR's on Ebay Germany for 50€, used with 2 10Gbemodules, 3 left..

Will try to send back 1 Uptimed X520.. after I test that S320E.

I went on and got that canceled yesterday night, or this morning? , after i found out it's a pcie 1.0 card running at x8.
So i needed a full slot available, which i don't.

The seller was actually very helpful, if someone is still interested.

 

[Matthias]

Just out of curiousity, has anyone tried those 100$/€ region 'new china' Chelsio T520-CR's? I seem to be able to get them everywhere.

 

[Mlovelace]

Just out of curiousity, has anyone tried those 100$/€ region 'new china' Chelsio T520-CR's? I seem to be able to get them everywhere.

@Matthias you add the 'up' command to the option section of the interface configuration in the UI so that it continues to bring up the interface upon reboot.

 

[Matthias]

@Matthias you add the 'up' command to the option section of the interface configuration in the UI so that it continues to bring up the interface upon reboot.

View attachment 63859

Thank you, will check on that!

 

[Matthias]

Going to try and set lagg as failover, but can be tricky as i read here.

Also, maybe worth while mentioning here that i got it working using that fs module on the switch side and using the uptimed module on the x520 nas side. And only in that order. I mixed and checked with different combo's, it just wouldn't connect. (Including up command through cli after checking module through cli)

However the same scenario using my desktop has a different outcome, they all work flawlessly.

Don't quite understand why

So windows can do some stuff 'right'

 

[nKk]

I have similar problem with Mikrotik SFP switch - the link is not established independent of modules that I try on the switch.
I found that if I manually select network speed (not use auto negotiation) - for example 1GB in Mikrotik switch, the link is established and working without other issues.
If you have time for tests you can try to set manually 10GB for NAS SFP port on Mikrotik switch and try again different SFP modules combinations.

 

[sretalla]

The genuine Intel cards are vendor locked to require Intel optics. You may try setting a sysctl tunable for "hw.ix.unsupported_sfp" to "1" which should cause common compatible SFP's to work regardless of vendor tagging. This will NOT force ALL possible SFP's to work, just the normal ones like FTLX8571D3BCV-xx where xx != IT. If you want to avoid this problem, pick up some used FTLX8571D3BCV-IT off eBay or buy an Intel vendor tagged module from a vendor such as fs.com. This is stupid, yes, and why we warn about it somewhere deep in the 10 Gig Networking Primer.

I don't necessarily recommend planning on this being a good or permanent solution, but you can also run an unlocker on the card, like this one https://gist.github.com/ixs/dbaac42730dea9bd124f26cbd439c58e

I ran into a similar problem with an x710 that I bought and ended up unlocking that card while I was waiting for genuine Intel SFP+ modules. (https://github.com/sretalla/xl710-unlocker)

The interesting findings with that are that the FreeBSD driver is fine with it and I can use any brand/type of module, but the Linux and ESXi drivers both detect the EEPROM shenanigans and won't load the interface.