TrueNAS 12.0-U5 - Intel XL710-BM1 not linking to switch

P

philboske

Cadet
Joined
Dec 20, 2021
Messages
3

System Specs:​


I have a Gigabyte 4U appliance with the following specs:

AMD EPYC 7302P
256 GB ECC memory
2 x 256 GB mirrored OS SSD's
18 x 12 TB Seagate Exos HDD's (2 vdevs of 9 x RaidZ2, I think... to be upgraded to 4 vdevs later).
U.2. Optane SLOG
2 TB U.2. NVMe L2ARC
Intel XL710-BM1 4x10Gb SFP+ NIC
TrueNAS Core 12.0-U5

Specs may seem odd, but they were recommended by our vendor for our exact use case (high throughput research data generation and processing). Either that, or they just wanted a commission. Don't care; not my money :smile:


Network Configuration:​


I have updated the NVM on the NIC to version 7.30 as that is what Intel recommends against the 1.11.29 ixl driver that is packaged with FreeBSD 12.2, which 'uname -a' tells me TrueNAS Core 12.0-U5 is built on.

The physical network setup is: Mikrotik CRS326-24S+2Q+ Switch (tested alternative CRS312-4C+8XG) > 2m SM patch > ~200m SM trunk > 2m SM patch > ~100m SM trunk > 10m SM patch > NIC.

The configuration is quite simple: ixl0 has been statically assigned 192.168.42.20/24.

Code:
Name    Mtu Network       Address              Ipkts Ierrs Idrop    Opkts Oerrs  Coll
ixl0   1500 <Link#1>      64:9d:99:b1:8b:3c        0     0     0        0     0     0
ixl0      - 192.168.42.0/ 192.168.42.20            0     -     -       22     -     -


There are no Tunables related to networking.

My understanding is that auto-negotiation cannot be disabled with the ixl driver, however I have changed the advertise_speed sysctl value to 0x4 which represents 10Gb, and also ensured that Flow Control is disabled:

Code:
root@truenas[~]# sysctl -x dev.ixl.0.advertise_speed
dev.ixl.0.advertise_speed: 0x00000004
root@truenas[~]# sysctl -x dev.ixl.0.fc
dev.ixl.0.fc: 0000000000


These settings have been replicated on the Switch. Both the primary switch (CRS326) and secondary switch (CRS312) are running the latest RouterOS 7.1 Stable.


Problem(s):​

1) Unable to create a connection between the NIC and either Switch.
2) Unsure what end is causing the problem(s).
3) I'm not a SysAdmin o_O

Any assistance or guidance would be appreciated.

Tests Undertaken:​


Test #1 - Checking to see that the NIC works:​


Intel XL710-BM1 to Intel XL710-BM1 through full length with Switch disconnected and patched back onto itself (~630m of SM total, using 4 cores of the trunk).

As the following ifconfig outputs show, when patched back onto itself, it has both RX and TX power and identifies as a 10Gbase-LR module running full-duplex, which is correct.
Code:
root@truenas[~]# ifconfig -vvv ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: 10gb-01
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 64:9d:99:b1:8b:3c
        media: Ethernet autoselect (10Gbase-LR <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
        plugged: SFP/SFP+/SFP28 10G Base-LR (LC)
        vendor: FS PN: SFP-10GLR-31 SN: F2020636794 DATE: 2021-10-28
        module temperature: 32.26 C Voltage: 3.23 Volts
        RX: 0.17 mW (-7.57 dBm) TX: 0.61 mW (-2.12 dBm)

        SFF8472 DUMP (0xA0 0..127 range):
        03 04 07 20 00 00 02 00 00 00 00 06 67 02 0A 64
        00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20
        20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 4C
        52 2D 33 31 20 20 20 20 41 20 20 20 05 1E 00 D3
        00 3A 00 00 46 32 30 32 30 36 33 36 37 39 34 20
        20 20 20 20 32 31 31 30 32 38 20 20 68 FA 02 F9
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

root@truenas[~]# ifconfig -vvv ixl1
ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        description: 10gb-02
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 64:9d:99:b1:8b:3c
        hwaddr 64:9d:99:b1:8b:3d
        media: Ethernet autoselect (10Gbase-LR <full-duplex>)
        status: active
        nd6 options=9<PERFORMNUD,IFDISABLED>
        plugged: SFP/SFP+/SFP28 10G Base-LR (LC)
        vendor: FS PN: SFP-10GLR-31 SN: F2020636793 DATE: 2021-10-28
        module temperature: 31.46 C Voltage: 3.22 Volts
        RX: 0.28 mW (-5.40 dBm) TX: 0.55 mW (-2.54 dBm)

        SFF8472 DUMP (0xA0 0..127 range):
        03 04 07 20 00 00 02 00 00 00 00 06 67 02 0A 64
        00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20
        20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 4C
        52 2D 33 31 20 20 20 20 41 20 20 20 05 1E 00 D3
        00 3A 00 00 46 32 30 32 30 36 33 36 37 39 33 20
        20 20 20 20 32 31 31 30 32 38 20 20 68 FA 02 F8
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00


Test #2 - Checking to see that the switches work:​


MikroTik CRS326 to MikroTik CRS312 through full length with NIC disconnected and patched back onto itself (~630m of SM total, using 4 cores of the trunk).

To validate that the switches are both functioning and are happy with the fibre installation:
Code:
[admin@MikroTik] > /interface ethernet monitor sfp-sfpplus22
                        ;;; Big NAS Uplink #1
                      name: sfp-sfpplus22
                    status: link-ok
          auto-negotiation: done
                      rate: 10Gbps
               full-duplex: yes
           tx-flow-control: no
           rx-flow-control: no
               advertising: 
  link-partner-advertising: 
        sfp-module-present: yes
               sfp-rx-loss: no
              sfp-tx-fault: no
                  sfp-type: SFP-or-SFP+
        sfp-connector-type: LC
        sfp-link-length-sm: 10km
           sfp-vendor-name: FS
    sfp-vendor-part-number: SFP-10GLR-31
         sfp-vendor-serial: F2033674409
    sfp-manufacturing-date: 21-11-03
            sfp-wavelength: 1310nm
           sfp-temperature: 34C
        sfp-supply-voltage: 3.214V
       sfp-tx-bias-current: 34mA
              sfp-tx-power: -2.977dBm
              sfp-rx-power: -5.326dBm
           eeprom-checksum: good
                    eeprom: 0000: 03 04 07 20 00 00 00 22  00 01 00 06 67 00 0a 64  ... ..." ....g..d
                            0010: 00 00 00 00 46 53 20 20  20 20 20 20 20 20 20 20  ....FS           
                            0020: 20 20 20 20 00 00 00 00  53 46 50 2d 31 30 47 4c      .... SFP-10GL
                            0030: 52 2d 33 31 20 20 20 20  20 20 20 20 05 1e 00 95  R-31         ....
                            0040: 00 1a 0e 58 46 32 30 33  33 36 37 34 34 30 39 20  ...XF203 3674409 
                            0050: 20 20 20 20 32 31 31 31  30 33 20 20 68 f0 03 2f      2111 03  h../
                            0060: 00 00 08 05 3a 7a 2f 9f  43 c7 10 71 45 4c 9d 04  ....:z/. C..qEL..
                            0070: e5 ab 80 00 00 00 00 00  00 00 00 00 9b 03 d2 ec  ........ ........
                            0080: 50 00 f1 00 4b 00 f6 00  8c a0 75 30 8a ac 77 24  P...K... ..u0..w$
                            0090: b9 8c 05 dc af c8 09 c4  57 73 02 f7 45 77 03 bb  ........ Ws..Ew..
                            00a0: 57 73 00 b6 45 77 00 e5  00 00 00 00 00 00 00 00  Ws..Ew.. ........
                            00b0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
                            00c0: 00 00 00 00 3f 80 00 00  00 00 00 00 01 00 00 00  ....?... ........
                            00d0: 01 00 00 00 01 00 00 00  01 00 00 00 00 00 00 af  ........ ........
                            00e0: 22 26 7d 8c 43 98 13 ae  0b 75 00 00 00 00 00 00  "&}.C... .u......
                            00f0: 00 00 00 00 00 00 00 00  00 00 00 ff ff ff ff 00  ........ ........

[admin@MikroTik] > /interface ethernet monitor combo1
                      name: combo1
                    status: link-ok
          auto-negotiation: done
                      rate: 10Gbps
               full-duplex: yes
           tx-flow-control: no
           rx-flow-control: no
               advertising: 10M-half,10M-full,100M-half,100M-full,1000M-full,10000M-full,2500M-full,5000M-full
  link-partner-advertising: 
               combo-state: sfp
        sfp-module-present: yes
               sfp-rx-loss: no
              sfp-tx-fault: no
                  sfp-type: SFP-or-SFP+
        sfp-connector-type: LC
        sfp-link-length-sm: 10km
           sfp-vendor-name: FS
    sfp-vendor-part-number: SFP-10GLR-31
         sfp-vendor-serial: F2033674410
    sfp-manufacturing-date: 21-11-03
            sfp-wavelength: 1310nm
           sfp-temperature: 33C
        sfp-supply-voltage: 3.257V
       sfp-tx-bias-current: 34mA
              sfp-tx-power: -2.753dBm
              sfp-rx-power: -3.996dBm
           eeprom-checksum: good
                    eeprom: 0000: 03 04 07 20 00 00 00 22  00 01 00 06 67 00 0a 64  ... ..." ....g..d
                            0010: 00 00 00 00 46 53 20 20  20 20 20 20 20 20 20 20  ....FS           
                            0020: 20 20 20 20 00 00 00 00  53 46 50 2d 31 30 47 4c      .... SFP-10GL
                            0030: 52 2d 33 31 20 20 20 20  20 20 20 20 05 1e 00 95  R-31         ....
                            0040: 00 1a 0e 58 46 32 30 33  33 36 37 34 34 31 30 20  ...XF203 3674410 
                            0050: 20 20 20 20 32 31 31 31  30 33 20 20 68 f0 03 27      2111 03  h..'
                            0060: 00 00 08 7f 6c 3b 29 c7  9b 7a 69 50 5a b1 c0 20  ....l;). .ziPZ.. 
                            0070: a3 19 72 00 00 00 00 00  00 00 00 00 c1 93 36 4c  ..r..... ......6L
                            0080: 50 00 f1 00 4b 00 f6 00  8c a0 75 30 8a ac 77 24  P...K... ..u0..w$
                            0090: b9 8c 05 dc af c8 09 c4  57 73 02 f7 45 77 03 bb  ........ Ws..Ew..
                            00a0: 57 73 00 b6 45 77 00 e5  00 00 00 00 00 00 00 00  Ws..Ew.. ........
                            00b0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
                            00c0: 00 00 00 00 3f 80 00 00  00 00 00 00 01 00 00 00  ....?... ........
                            00d0: 01 00 00 00 01 00 00 00  01 00 00 00 00 00 00 af  ........ ........
                            00e0: 21 ff 7f 3c 43 59 14 b8  0f 90 00 00 00 00 00 00  !..<CY.. ........
                            00f0: 00 00 00 00 00 00 00 00  00 00 00 ff ff ff ff 00  ........ ........



Test #3 - Connecting NIC to CRS326 Switch:​

Code:
root@truenas[~]# ifconfig -vvv ixl0
ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
        options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 64:9d:99:b1:8b:3c
        inet 192.168.42.20 netmask 0xffffff00 broadcast 192.168.42.255
        media: Ethernet autoselect
        status: no carrier
        nd6 options=9<PERFORMNUD,IFDISABLED>
        plugged: SFP/SFP+/SFP28 10G Base-LR (LC)
        vendor: FS PN: SFP-10GLR-31 SN: F2020636795 DATE: 2021-11-23
        module temperature: 30.88 C Voltage: 3.24 Volts
        RX: 0.00 mW (-40.00 dBm) TX: 0.55 mW (-2.52 dBm)

        SFF8472 DUMP (0xA0 0..127 range):
        03 04 07 20 00 00 00 00 00 00 00 06 67 00 0A 64
        00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20
        20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 4C
        52 2D 33 31 20 20 20 20 41 20 20 20 05 1E 00 CF
        00 1A 00 00 46 32 30 32 30 36 33 36 37 39 35 20
        20 20 20 20 32 31 31 31 32 33 20 20 68 90 01 6B
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
        00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00

[admin@MikroTik] > /interface ethernet monitor sfp-sfpplus22
                        ;;; Big NAS Uplink #1
                      name: sfp-sfpplus22
                    status: no-link
          auto-negotiation: done
               advertising: 
  link-partner-advertising: 
        sfp-module-present: yes
               sfp-rx-loss: yes
              sfp-tx-fault: no
                  sfp-type: SFP-or-SFP+
        sfp-connector-type: LC
        sfp-link-length-sm: 10km
           sfp-vendor-name: FS
    sfp-vendor-part-number: SFP-10GLR-31
         sfp-vendor-serial: F2033674409
    sfp-manufacturing-date: 21-11-03
            sfp-wavelength: 1310nm
           sfp-temperature: 37C
        sfp-supply-voltage: 3.212V
       sfp-tx-bias-current: 35mA
              sfp-tx-power: -2.943dBm
              sfp-rx-power: -40dBm
           eeprom-checksum: good
                    eeprom: 0000: 03 04 07 20 00 00 00 22  00 01 00 06 67 00 0a 64  ... ..." ....g..d
                            0010: 00 00 00 00 46 53 20 20  20 20 20 20 20 20 20 20  ....FS           
                            0020: 20 20 20 20 00 00 00 00  53 46 50 2d 31 30 47 4c      .... SFP-10GL
                            0030: 52 2d 33 31 20 20 20 20  20 20 20 20 05 1e 00 95  R-31         ....
                            0040: 00 1a 0e 58 46 32 30 33  33 36 37 34 34 30 39 20  ...XF203 3674409 
                            0050: 20 20 20 20 32 31 31 31  30 33 20 20 68 f0 03 2f      2111 03  h../
                            0060: 00 00 08 05 3a 7a 2f 9f  43 c7 10 71 45 4c 9d 04  ....:z/. C..qEL..
                            0070: e5 ab 80 00 00 00 00 00  00 00 00 00 9b 03 d2 ec  ........ ........
                            0080: 50 00 f1 00 4b 00 f6 00  8c a0 75 30 8a ac 77 24  P...K... ..u0..w$
                            0090: b9 8c 05 dc af c8 09 c4  57 73 02 f7 45 77 03 bb  ........ Ws..Ew..
                            00a0: 57 73 00 b6 45 77 00 e5  00 00 00 00 00 00 00 00  Ws..Ew.. ........
                            00b0: 00 00 00 00 00 00 00 00  00 00 00 00 00 00 00 00  ........ ........
                            00c0: 00 00 00 00 3f 80 00 00  00 00 00 00 01 00 00 00  ....?... ........
                            00d0: 01 00 00 00 01 00 00 00  01 00 00 00 00 00 00 af  ........ ........
                            00e0: 25 fe 7d 7a 45 20 13 d6  00 01 00 00 00 00 02 00  %.}zE .. ........
                            00f0: 00 40 00 00 00 40 00 00  00 00 00 ff ff ff ff 00  .@...@.. ........
 
P

philboske

Cadet
Joined
Dec 20, 2021
Messages
3
Forgot the sysctl output showing fw/nvm versions:
Code:
dev.ixl.0.fw_version: fw 7.3.60988 api 1.10 nvm 7.30 etid 80008376 oem 1.263.0
dev.ixl.0.current_speed: Unknown
dev.ixl.0.supported_speeds: 6
dev.ixl.0.advertise_speed: 4
dev.ixl.0.fc: 0
 
P

philboske

Cadet
Joined
Dec 20, 2021
Messages
3
I had to take a processing workstation down to add a scratch drive because this NAS isn't operational, so while it was offline I decided to drop the XL710-BM1 into it.

OS: Xubuntu 20.04
Kernel: 5.4.0-90-generic
Driver: i40e 2.8.20-k
Code:
driver: i40e
version: 2.8.20-k
firmware-version: 7.30 0x80008376 1.1853.0
expansion-rom-version: 
bus-info: 0000:03:00.0
supports-statistics: yes
supports-test: yes
supports-eeprom-access: yes
supports-register-dump: yes
supports-priv-flags: yes

Cabling: The full ~630m loop through the infrastructure trunk and back.
Code:
    Identifier                                : 0x03 (SFP)
    Extended identifier                       : 0x04 (GBIC/SFP defined by 2-wire interface ID)
    Connector                                 : 0x07 (LC)
    Transceiver codes                         : 0x20 0x00 0x00 0x00 0x00 0x00 0x00 0x00 0x00
    Transceiver type                          : 10G Ethernet: 10G Base-LR
    Encoding                                  : 0x06 (64B/66B)
    BR, Nominal                               : 10300MBd
    Rate identifier                           : 0x00 (unspecified)
    Length (SMF,km)                           : 10km
    Length (SMF)                              : 10000m
    Length (50um)                             : 0m
    Length (62.5um)                           : 0m
    Length (Copper)                           : 0m
    Length (OM3)                              : 0m
    Laser wavelength                          : 1310nm
    Vendor name                               : FS
    Vendor OUI                                : 00:1b:21
    Vendor PN                                 : SFP-10GLR-31
    Vendor rev                                : A
    Option values                             : 0x00 0x1a
    Option                                    : RX_LOS implemented
    Option                                    : TX_FAULT implemented
    Option                                    : TX_DISABLE implemented
    BR margin, max                            : 0%
    BR margin, min                            : 0%
    Vendor SN                                 : F2020636795
    Date code                                 : 211123
    Optical diagnostics support               : Yes
    Laser bias current                        : 35.940 mA
    Laser output power                        : 0.5534 mW / -2.57 dBm
    Receiver signal average optical power     : 0.3613 mW / -4.42 dBm
    Module temperature                        : 40.00 degrees C / 104.00 degrees F
    Module voltage                            : 3.2338 V

Works perfectly fine out of the box once assigned a static IP.

So... What's the magical thing I have wrong in TrueNAS that makes it not want to play the game?
 
You must log in or register to reply here.

Similar threads

RandyJC
  • Locked
i can ping the jail, but the port is unreachable
Replies
6
Views
2K
RandyJC
RandyJC
J
  • Locked
Reboot causes router to crash and reboot
Replies
7
Views
2K
JamesWorts
J
W
  • Locked
SOLVED bridge from jails and iohyve guests to outside network fails
Replies
5
Views
3K
wurzeladministrator
W
N
Help getting 10 gig Peer to Peer working
Replies
16
Views
3K
Redcoat
Redcoat
A
Noticed these errors in the log, is this disk related?
Replies
3
Views
1K
sretalla
sretalla
Top