System Specs:
I have a Gigabyte 4U appliance with the following specs:
AMD EPYC 7302P
256 GB ECC memory
2 x 256 GB mirrored OS SSD's
18 x 12 TB Seagate Exos HDD's (2 vdevs of 9 x RaidZ2, I think... to be upgraded to 4 vdevs later).
U.2. Optane SLOG
2 TB U.2. NVMe L2ARC
Intel XL710-BM1 4x10Gb SFP+ NIC
TrueNAS Core 12.0-U5
Specs may seem odd, but they were recommended by our vendor for our exact use case (high throughput research data generation and processing). Either that, or they just wanted a commission. Don't care; not my money
Network Configuration:
I have updated the NVM on the NIC to version 7.30 as that is what Intel recommends against the 1.11.29 ixl driver that is packaged with FreeBSD 12.2, which 'uname -a' tells me TrueNAS Core 12.0-U5 is built on.
The physical network setup is: Mikrotik CRS326-24S+2Q+ Switch (tested alternative CRS312-4C+8XG) > 2m SM patch > ~200m SM trunk > 2m SM patch > ~100m SM trunk > 10m SM patch > NIC.
The configuration is quite simple: ixl0 has been statically assigned 192.168.42.20/24.
Code:
Name Mtu Network Address Ipkts Ierrs Idrop Opkts Oerrs Coll ixl0 1500 <Link#1> 64:9d:99:b1:8b:3c 0 0 0 0 0 0 ixl0 - 192.168.42.0/ 192.168.42.20 0 - - 22 - -
There are no Tunables related to networking.
My understanding is that auto-negotiation cannot be disabled with the ixl driver, however I have changed the advertise_speed sysctl value to 0x4 which represents 10Gb, and also ensured that Flow Control is disabled:
Code:
root@truenas[~]# sysctl -x dev.ixl.0.advertise_speed dev.ixl.0.advertise_speed: 0x00000004 root@truenas[~]# sysctl -x dev.ixl.0.fc dev.ixl.0.fc: 0000000000
These settings have been replicated on the Switch. Both the primary switch (CRS326) and secondary switch (CRS312) are running the latest RouterOS 7.1 Stable.
Problem(s):
1) Unable to create a connection between the NIC and either Switch.2) Unsure what end is causing the problem(s).
3) I'm not a SysAdmin
Any assistance or guidance would be appreciated.
Tests Undertaken:
Test #1 - Checking to see that the NIC works:
Intel XL710-BM1 to Intel XL710-BM1 through full length with Switch disconnected and patched back onto itself (~630m of SM total, using 4 cores of the trunk).
As the following ifconfig outputs show, when patched back onto itself, it has both RX and TX power and identifies as a 10Gbase-LR module running full-duplex, which is correct.
Code:
root@truenas[~]# ifconfig -vvv ixl0 ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: 10gb-01 options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 64:9d:99:b1:8b:3c media: Ethernet autoselect (10Gbase-LR <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> plugged: SFP/SFP+/SFP28 10G Base-LR (LC) vendor: FS PN: SFP-10GLR-31 SN: F2020636794 DATE: 2021-10-28 module temperature: 32.26 C Voltage: 3.23 Volts RX: 0.17 mW (-7.57 dBm) TX: 0.61 mW (-2.12 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 20 00 00 02 00 00 00 00 06 67 02 0A 64 00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 4C 52 2D 33 31 20 20 20 20 41 20 20 20 05 1E 00 D3 00 3A 00 00 46 32 30 32 30 36 33 36 37 39 34 20 20 20 20 20 32 31 31 30 32 38 20 20 68 FA 02 F9 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 root@truenas[~]# ifconfig -vvv ixl1 ixl1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 description: 10gb-02 options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 64:9d:99:b1:8b:3c hwaddr 64:9d:99:b1:8b:3d media: Ethernet autoselect (10Gbase-LR <full-duplex>) status: active nd6 options=9<PERFORMNUD,IFDISABLED> plugged: SFP/SFP+/SFP28 10G Base-LR (LC) vendor: FS PN: SFP-10GLR-31 SN: F2020636793 DATE: 2021-10-28 module temperature: 31.46 C Voltage: 3.22 Volts RX: 0.28 mW (-5.40 dBm) TX: 0.55 mW (-2.54 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 20 00 00 02 00 00 00 00 06 67 02 0A 64 00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 4C 52 2D 33 31 20 20 20 20 41 20 20 20 05 1E 00 D3 00 3A 00 00 46 32 30 32 30 36 33 36 37 39 33 20 20 20 20 20 32 31 31 30 32 38 20 20 68 FA 02 F8 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00
Test #2 - Checking to see that the switches work:
MikroTik CRS326 to MikroTik CRS312 through full length with NIC disconnected and patched back onto itself (~630m of SM total, using 4 cores of the trunk).
To validate that the switches are both functioning and are happy with the fibre installation:
Code:
[admin@MikroTik] > /interface ethernet monitor sfp-sfpplus22 ;;; Big NAS Uplink #1 name: sfp-sfpplus22 status: link-ok auto-negotiation: done rate: 10Gbps full-duplex: yes tx-flow-control: no rx-flow-control: no advertising: link-partner-advertising: sfp-module-present: yes sfp-rx-loss: no sfp-tx-fault: no sfp-type: SFP-or-SFP+ sfp-connector-type: LC sfp-link-length-sm: 10km sfp-vendor-name: FS sfp-vendor-part-number: SFP-10GLR-31 sfp-vendor-serial: F2033674409 sfp-manufacturing-date: 21-11-03 sfp-wavelength: 1310nm sfp-temperature: 34C sfp-supply-voltage: 3.214V sfp-tx-bias-current: 34mA sfp-tx-power: -2.977dBm sfp-rx-power: -5.326dBm eeprom-checksum: good eeprom: 0000: 03 04 07 20 00 00 00 22 00 01 00 06 67 00 0a 64 ... ..." ....g..d 0010: 00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 ....FS 0020: 20 20 20 20 00 00 00 00 53 46 50 2d 31 30 47 4c .... SFP-10GL 0030: 52 2d 33 31 20 20 20 20 20 20 20 20 05 1e 00 95 R-31 .... 0040: 00 1a 0e 58 46 32 30 33 33 36 37 34 34 30 39 20 ...XF203 3674409 0050: 20 20 20 20 32 31 31 31 30 33 20 20 68 f0 03 2f 2111 03 h../ 0060: 00 00 08 05 3a 7a 2f 9f 43 c7 10 71 45 4c 9d 04 ....:z/. C..qEL.. 0070: e5 ab 80 00 00 00 00 00 00 00 00 00 9b 03 d2 ec ........ ........ 0080: 50 00 f1 00 4b 00 f6 00 8c a0 75 30 8a ac 77 24 P...K... ..u0..w$ 0090: b9 8c 05 dc af c8 09 c4 57 73 02 f7 45 77 03 bb ........ Ws..Ew.. 00a0: 57 73 00 b6 45 77 00 e5 00 00 00 00 00 00 00 00 Ws..Ew.. ........ 00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 00c0: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....?... ........ 00d0: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 af ........ ........ 00e0: 22 26 7d 8c 43 98 13 ae 0b 75 00 00 00 00 00 00 "&}.C... .u...... 00f0: 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff 00 ........ ........ [admin@MikroTik] > /interface ethernet monitor combo1 name: combo1 status: link-ok auto-negotiation: done rate: 10Gbps full-duplex: yes tx-flow-control: no rx-flow-control: no advertising: 10M-half,10M-full,100M-half,100M-full,1000M-full,10000M-full,2500M-full,5000M-full link-partner-advertising: combo-state: sfp sfp-module-present: yes sfp-rx-loss: no sfp-tx-fault: no sfp-type: SFP-or-SFP+ sfp-connector-type: LC sfp-link-length-sm: 10km sfp-vendor-name: FS sfp-vendor-part-number: SFP-10GLR-31 sfp-vendor-serial: F2033674410 sfp-manufacturing-date: 21-11-03 sfp-wavelength: 1310nm sfp-temperature: 33C sfp-supply-voltage: 3.257V sfp-tx-bias-current: 34mA sfp-tx-power: -2.753dBm sfp-rx-power: -3.996dBm eeprom-checksum: good eeprom: 0000: 03 04 07 20 00 00 00 22 00 01 00 06 67 00 0a 64 ... ..." ....g..d 0010: 00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 ....FS 0020: 20 20 20 20 00 00 00 00 53 46 50 2d 31 30 47 4c .... SFP-10GL 0030: 52 2d 33 31 20 20 20 20 20 20 20 20 05 1e 00 95 R-31 .... 0040: 00 1a 0e 58 46 32 30 33 33 36 37 34 34 31 30 20 ...XF203 3674410 0050: 20 20 20 20 32 31 31 31 30 33 20 20 68 f0 03 27 2111 03 h..' 0060: 00 00 08 7f 6c 3b 29 c7 9b 7a 69 50 5a b1 c0 20 ....l;). .ziPZ.. 0070: a3 19 72 00 00 00 00 00 00 00 00 00 c1 93 36 4c ..r..... ......6L 0080: 50 00 f1 00 4b 00 f6 00 8c a0 75 30 8a ac 77 24 P...K... ..u0..w$ 0090: b9 8c 05 dc af c8 09 c4 57 73 02 f7 45 77 03 bb ........ Ws..Ew.. 00a0: 57 73 00 b6 45 77 00 e5 00 00 00 00 00 00 00 00 Ws..Ew.. ........ 00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 00c0: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....?... ........ 00d0: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 af ........ ........ 00e0: 21 ff 7f 3c 43 59 14 b8 0f 90 00 00 00 00 00 00 !..<CY.. ........ 00f0: 00 00 00 00 00 00 00 00 00 00 00 ff ff ff ff 00 ........ ........
Test #3 - Connecting NIC to CRS326 Switch:
Code:
root@truenas[~]# ifconfig -vvv ixl0 ixl0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500 options=e507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO,RXCSUM_IPV6,TXCSUM_IPV6> ether 64:9d:99:b1:8b:3c inet 192.168.42.20 netmask 0xffffff00 broadcast 192.168.42.255 media: Ethernet autoselect status: no carrier nd6 options=9<PERFORMNUD,IFDISABLED> plugged: SFP/SFP+/SFP28 10G Base-LR (LC) vendor: FS PN: SFP-10GLR-31 SN: F2020636795 DATE: 2021-11-23 module temperature: 30.88 C Voltage: 3.24 Volts RX: 0.00 mW (-40.00 dBm) TX: 0.55 mW (-2.52 dBm) SFF8472 DUMP (0xA0 0..127 range): 03 04 07 20 00 00 00 00 00 00 00 06 67 00 0A 64 00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 20 20 20 20 00 00 1B 21 53 46 50 2D 31 30 47 4C 52 2D 33 31 20 20 20 20 41 20 20 20 05 1E 00 CF 00 1A 00 00 46 32 30 32 30 36 33 36 37 39 35 20 20 20 20 20 32 31 31 31 32 33 20 20 68 90 01 6B 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 [admin@MikroTik] > /interface ethernet monitor sfp-sfpplus22 ;;; Big NAS Uplink #1 name: sfp-sfpplus22 status: no-link auto-negotiation: done advertising: link-partner-advertising: sfp-module-present: yes sfp-rx-loss: yes sfp-tx-fault: no sfp-type: SFP-or-SFP+ sfp-connector-type: LC sfp-link-length-sm: 10km sfp-vendor-name: FS sfp-vendor-part-number: SFP-10GLR-31 sfp-vendor-serial: F2033674409 sfp-manufacturing-date: 21-11-03 sfp-wavelength: 1310nm sfp-temperature: 37C sfp-supply-voltage: 3.212V sfp-tx-bias-current: 35mA sfp-tx-power: -2.943dBm sfp-rx-power: -40dBm eeprom-checksum: good eeprom: 0000: 03 04 07 20 00 00 00 22 00 01 00 06 67 00 0a 64 ... ..." ....g..d 0010: 00 00 00 00 46 53 20 20 20 20 20 20 20 20 20 20 ....FS 0020: 20 20 20 20 00 00 00 00 53 46 50 2d 31 30 47 4c .... SFP-10GL 0030: 52 2d 33 31 20 20 20 20 20 20 20 20 05 1e 00 95 R-31 .... 0040: 00 1a 0e 58 46 32 30 33 33 36 37 34 34 30 39 20 ...XF203 3674409 0050: 20 20 20 20 32 31 31 31 30 33 20 20 68 f0 03 2f 2111 03 h../ 0060: 00 00 08 05 3a 7a 2f 9f 43 c7 10 71 45 4c 9d 04 ....:z/. C..qEL.. 0070: e5 ab 80 00 00 00 00 00 00 00 00 00 9b 03 d2 ec ........ ........ 0080: 50 00 f1 00 4b 00 f6 00 8c a0 75 30 8a ac 77 24 P...K... ..u0..w$ 0090: b9 8c 05 dc af c8 09 c4 57 73 02 f7 45 77 03 bb ........ Ws..Ew.. 00a0: 57 73 00 b6 45 77 00 e5 00 00 00 00 00 00 00 00 Ws..Ew.. ........ 00b0: 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 00 ........ ........ 00c0: 00 00 00 00 3f 80 00 00 00 00 00 00 01 00 00 00 ....?... ........ 00d0: 01 00 00 00 01 00 00 00 01 00 00 00 00 00 00 af ........ ........ 00e0: 25 fe 7d 7a 45 20 13 d6 00 01 00 00 00 00 02 00 %.}zE .. ........ 00f0: 00 40 00 00 00 40 00 00 00 00 00 ff ff ff ff 00 .@...@.. ........