Intel x710 SFP speed throttle on after 12u4 upgrade to 12u6/7/8

Shimano

Dabbler
Joined
Jun 14, 2019
Messages
11
Hello Guys, please help me !

I've been using FreeNAS since V9, upgraded all the way to TrueNAS-12.0-U8.1, everythign was OK untill 12U4, after 12U4 all versions U6,U7 and now U8 are not usable for me, my network performance on Intel x710 SFP card connected via 10G SFP+ Intel modules to my Juniper switch is DROPING BAD like not even 1MB, the transfer speed is around 300-700Kb/s !!! even when i'm pinging my TrueNAS machine i get like 2000 ms and its fluctuating please help me, the workaround was to go to Boot and select U4 and boot and everything is working fine but it seems that something changed and i wanna update and fix so i can use future updates. Thank you for your time reading.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
What firmware is running on the X710? Is it an Intel branded X710?
 

Shimano

Dabbler
Joined
Jun 14, 2019
Messages
11
No sure how to check the firmware can you help me with a command ? yes its a Intel Branded X710

ixl0@pci0:19:0:0: class=0x020000 card=0x000015d9 chip=0x15728086 rev=0x02 hdr=0x00
vendor = 'Intel Corporation'
device = 'Ethernet Controller X710 for 10GbE SFP+'
class = network
subclass = ethernet
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I don't know, offhand. I don't use these directly on any TrueNAS hosts.

The manual for ixl discusses the issue and refers you to a useless FreeBSD port that cannot be used on TrueNAS. (bleh).

Since it is an Intel branded card, Intel does have utilities available. I don't know much about them. We have the Supermicro OEM cards here so that's where my familiarity primarily is.

The bit I can clue you in on is that these things are a little twitchy about the NVM firmware running on the card itself.

I guess I would try "sysctl -a | grep dev.ixl" to see if it outputted useful stuff, and you may be able to find useful clues about driver/firmware compatibility at


I am sorry for such relatively arcane and potentially useless answers.
 

Shimano

Dabbler
Joined
Jun 14, 2019
Messages
11
Well i bought my card from a supermiro vendor and it came in a supermicro white box but it has intel on the card itself and also TrueNAS sees it as a intel, there was no issues whatsoever with FreeNAS 11x and 12 up until 12u4, everything after 12u4 makes the network slow as a turtle, under 1MB and it flops a lot, if i have a ping on it it varies between 100-2000 all the time, if i select boot 12u4 everything goes to normal ping is under 1ms can it be a driver issue maybe ? here is the output:

root@truenas[~]# sysctl -a | grep dev.ixl
dev.ixl.0.wake: 0
dev.ixl.0.mac.xoff_recvd: 0
dev.ixl.0.mac.xoff_txd: 0
dev.ixl.0.mac.xon_recvd: 0
dev.ixl.0.mac.xon_txd: 0
dev.ixl.0.mac.tx_frames_big: 0
dev.ixl.0.mac.tx_frames_1024_1522: 934461
dev.ixl.0.mac.tx_frames_512_1023: 5956
dev.ixl.0.mac.tx_frames_256_511: 3337
dev.ixl.0.mac.tx_frames_128_255: 2881
dev.ixl.0.mac.tx_frames_65_127: 15062
dev.ixl.0.mac.tx_frames_64: 2015
dev.ixl.0.mac.checksum_errors: 0
dev.ixl.0.mac.rx_jabber: 0
dev.ixl.0.mac.rx_oversized: 0
dev.ixl.0.mac.rx_fragmented: 0
dev.ixl.0.mac.rx_undersize: 0
dev.ixl.0.mac.rx_frames_big: 0
dev.ixl.0.mac.rx_frames_1024_1522: 24079
dev.ixl.0.mac.rx_frames_512_1023: 1344
dev.ixl.0.mac.rx_frames_256_511: 4085
dev.ixl.0.mac.rx_frames_128_255: 3212
dev.ixl.0.mac.rx_frames_65_127: 10634
dev.ixl.0.mac.rx_frames_64: 39317
dev.ixl.0.mac.rx_length_errors: 0
dev.ixl.0.mac.remote_faults: 2
dev.ixl.0.mac.local_faults: 1
dev.ixl.0.mac.illegal_bytes: 0
dev.ixl.0.mac.crc_errors: 0
dev.ixl.0.mac.bcast_pkts_txd: 22
dev.ixl.0.mac.mcast_pkts_txd: 203
dev.ixl.0.mac.ucast_pkts_txd: 963487
dev.ixl.0.mac.good_octets_txd: 1425134927
dev.ixl.0.mac.rx_discards: 0
dev.ixl.0.mac.bcast_pkts_rcvd: 4313
dev.ixl.0.mac.mcast_pkts_rcvd: 3364
dev.ixl.0.mac.ucast_pkts_rcvd: 74994
dev.ixl.0.mac.good_octets_rcvd: 42990120
dev.ixl.0.pf.txq00.itr: 0
dev.ixl.0.pf.txq00.bytes: 1372008517
dev.ixl.0.pf.txq00.packets: 51621
dev.ixl.0.pf.txq00.mss_too_small: 0
dev.ixl.0.pf.txq00.tso: 25227
dev.ixl.0.pf.rxq00.itr: 0
dev.ixl.0.pf.rxq00.desc_err: 0
dev.ixl.0.pf.rxq00.bytes: 42572756
dev.ixl.0.pf.rxq00.packets: 81932
dev.ixl.0.pf.rxq00.irqs: 121789
dev.ixl.0.pf.rx_errors: 0
dev.ixl.0.pf.bcast_pkts_txd: 3560
dev.ixl.0.pf.mcast_pkts_txd: 281474976665599
dev.ixl.0.pf.ucast_pkts_txd: 10944130046
dev.ixl.0.pf.good_octets_txd: 15924706459974
dev.ixl.0.pf.rx_discards: 4293375796
dev.ixl.0.pf.bcast_pkts_rcvd: 1151786
dev.ixl.0.pf.mcast_pkts_rcvd: 23196
dev.ixl.0.pf.ucast_pkts_rcvd: 4151093237
dev.ixl.0.pf.good_octets_rcvd: 5417713012571
dev.ixl.0.admin_irq: 0
dev.ixl.0.link_active_on_if_down: 1
dev.ixl.0.eee.rx_lpi_count: 0
dev.ixl.0.eee.tx_lpi_count: 0
dev.ixl.0.eee.rx_lpi_status: 0
dev.ixl.0.eee.tx_lpi_status: 0
dev.ixl.0.eee.enable: 0
dev.ixl.0.fw_lldp: 1
dev.ixl.0.dynamic_tx_itr: 0
dev.ixl.0.dynamic_rx_itr: 0
dev.ixl.0.rx_itr: 62
dev.ixl.0.tx_itr: 122
dev.ixl.0.unallocated_queues: 383
dev.ixl.0.fw_version: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002a9b oem 1.262.0
dev.ixl.0.current_speed: 10 Gbps
dev.ixl.0.supported_speeds: 6
dev.ixl.0.advertise_speed: 6
dev.ixl.0.fc: 3
dev.ixl.0.iflib.rxq0.rxq_fl0.buf_size: 2048
dev.ixl.0.iflib.rxq0.rxq_fl0.credits: 1023
dev.ixl.0.iflib.rxq0.rxq_fl0.cidx: 20
dev.ixl.0.iflib.rxq0.rxq_fl0.pidx: 19
dev.ixl.0.iflib.rxq0.cpu: 3
dev.ixl.0.iflib.txq0.r_abdications: 0
dev.ixl.0.iflib.txq0.r_restarts: 0
dev.ixl.0.iflib.txq0.r_stalls: 0
dev.ixl.0.iflib.txq0.r_starts: 55598
dev.ixl.0.iflib.txq0.r_drops: 0
dev.ixl.0.iflib.txq0.r_enqueues: 56062
dev.ixl.0.iflib.txq0.ring_state: pidx_head: 0766 pidx_tail: 0766 cidx: 0766 state: IDLE
dev.ixl.0.iflib.txq0.txq_cleaned: 432345
dev.ixl.0.iflib.txq0.txq_processed: 432353
dev.ixl.0.iflib.txq0.txq_in_use: 9
dev.ixl.0.iflib.txq0.txq_cidx_processed: 226
dev.ixl.0.iflib.txq0.txq_cidx: 220
dev.ixl.0.iflib.txq0.txq_pidx: 230
dev.ixl.0.iflib.txq0.no_tx_dma_setup: 0
dev.ixl.0.iflib.txq0.txd_encap_efbig: 0
dev.ixl.0.iflib.txq0.tx_map_failed: 0
dev.ixl.0.iflib.txq0.no_desc_avail: 0
dev.ixl.0.iflib.txq0.mbuf_defrag_failed: 0
dev.ixl.0.iflib.txq0.m_pullups: 0
dev.ixl.0.iflib.txq0.mbuf_defrag: 0
dev.ixl.0.iflib.txq0.cpu: 2
dev.ixl.0.iflib.override_nrxds: 0
dev.ixl.0.iflib.override_ntxds: 0
dev.ixl.0.iflib.use_logical_cores: 0
dev.ixl.0.iflib.separate_txrx: 0
dev.ixl.0.iflib.core_offset: 0
dev.ixl.0.iflib.tx_abdicate: 0
dev.ixl.0.iflib.rx_budget: 0
dev.ixl.0.iflib.disable_msix: 0
dev.ixl.0.iflib.override_qs_enable: 0
dev.ixl.0.iflib.override_nrxqs: 0
dev.ixl.0.iflib.override_ntxqs: 0
dev.ixl.0.iflib.driver_version: 2.3.1-k
dev.ixl.0.%parent: pci5
dev.ixl.0.%pnpinfo: vendor=0x8086 device=0x1572 subvendor=0x15d9 subdevice=0x0000 class=0x020000
dev.ixl.0.%location: slot=0 function=0 dbsf=pci0:19:0:0 handle=\_SB_.PCI0.PE60.S1F0
dev.ixl.0.%driver: ixl
dev.ixl.0.%desc: Intel(R) Ethernet Controller X710 for 10GbE SFP+ - 2.3.1-k
dev.ixl.%parent:
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Well i bought my card from a supermiro vendor and it came in a supermicro white box but it has intel on the card itself

That sounds like it's likely to be a Supermicro branded part. This won't prevent itself from looking like an Intel part in many ways.

Around here, it became necessary to replace the old Solarflare cards we used with ESXi 6 because of the Linux driver deprecation, and I chose the X710 because it was possible to find the Supermicro AOC-STG-I4S pretty cheaply. It looks like


what's in that eBay listing. If this were an Intel part, I believe it'd probably be named an XL710-DA4. Intel does make an X710-DA4 that is a full-height card based on the X710 (not the 40G XL710). So I went down the obscure card rabbit hole anyways.

Despite the fact that Intel makes the chip that powers this thing, the firmware that configures it to be split up into 4 ports and makes all the "stuff" work right comes from Supermicro, who have tweaked the Intel firmware to do the right things on the Supermicro-built card. So you might want to be very careful about whether there's a Supermicro silkscreen on your card, in which case you have a Supermicro card, and you should have a specific part number for it as well. I don't know that there's an easy way to get this electronically. I suspect there may not be.

In any case, the bit we were looking for is this:

dev.ixl.0.fw_version: fw 5.0.40043 api 1.5 nvm 5.05 etid 80002a9b oem 1.262.0

You're on nvm 5.05, which is pretty old. For Supermicro, the upgrade train goes 5.05->6.01->7.2->8.3

So what may have happened to you is that something in the TrueNAS driver changed, like perhaps it was updated, and now doesn't work well with the old firmware.

This does not mean that it is good to upgrade to the latest firmware. You may need to do some research on the topic. I suggest that the pfSense forums might be a good resource as these cards are probably used in pfSense boxes. I've never recommended the X710 cards for TrueNAS, as the X520's are SUPER solid, and if you really need the last bit of performance, then usually the Chelsios are the go-to. So my insight here is somewhat limited.

I can tell you that if you have a Supermicro card and you need NVM updates, you have to e-mail their support address. Intel provides downloads for their retail branded products on the Intel download site.

It is also very difficult to downgrade NVM, or so I've been warned, so I do not recommend getting overly aggressive with the version upgrades. If you go all the way to Intel's latest 8.6 but find that FreeBSD's driver needs 7.2, you may be stuck with an unusable card until the FreeBSD driver is updated. So, research, and small incremental steps until your problem vanishes, that's the ticket in my opinion.
 

Shimano

Dabbler
Joined
Jun 14, 2019
Messages
11
This is exactly what i have written down on my NIC white box: AOC-STG-I4S. i'm not sure how exactly to upgrade the firmware of the card, the vendor i buy it from was like 5 years ago so it cannot offer me any support on it is there a tutorial somewhere ? i also have pfsense and works perfectly fine on pfsense
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
You'll need to reach out to Supermicro to acquire the firmware files. They will provide you with a download link where you will get some ZIP's containing UEFI files, so basically you put the extracted files onto a USB thumb drive, then use the UEFI "shell" at bootup to go and run the install script. So for example you would get these files from Supermicro

# ls *.zip
STGi4S2B_6.01.zip
STGi4S3B_NUP_7.2.zip
STGi4S3C_NUP_8.3.zip

and extract this zipfile to a USB key (these are the 6.01 update)

# ls -l STGi4S2B_UEFI
total 4038
-rw-rw-r-- 1 root wheel 4194304 Feb 1 2018 710MI2B.bin
-rw-rw-r-- 1 root wheel 2535163 Jan 16 2018 BootIMG.FLB
-rw-rw-r-- 1 root wheel 231 Feb 8 2018 Instruction.txt
-rw-rw-r-- 1 root wheel 305 Feb 8 2018 map.txt
-rw-rw-r-- 1 root wheel 674 Feb 9 2018 nvmupdate.cfg
-rw-rw-r-- 1 root wheel 3754752 Jan 16 2018 nvmupdate64e.efi
-rw-rw-r-- 1 root wheel 5535 Feb 9 2018 Release Note.txt
-rw-rw-r-- 1 root wheel 49 Feb 8 2018 STGi4S.nsh

Then UEFI boot and inside the UEFI shell, you do "fs1:" (may be some other number) to change to the thumb drive, and when you see these files, you then run STGi4S.nsh which is a script that should update the card to 6.01

You then have to power cycle the host to complete the update. I would definitely just increment one firmware level at a time, and research in advance which one is recommended for use with whatever driver is in current FreeBSD.

I briefly looked to see if there was any clear indication about what NVM to use with FreeBSD. FreeBSD 12.2 ships with an updater port that installs 8.0, so, optimistically, I am guessing that going all the way wouldn't ruin things. But, I don't know, it's not my card, and you are warned to be careful because I am definitely handing you magic bits that are deeper than the average user normally works with. Here's a cup of gasoline in your left hand, a lit match in your right hand... try not to start yourself on fire. Get my drift? Good luck.
 

Shimano

Dabbler
Joined
Jun 14, 2019
Messages
11
You'll need to reach out to Supermicro to acquire the firmware files. They will provide you with a download link where you will get some ZIP's containing UEFI files, so basically you put the extracted files onto a USB thumb drive, then use the UEFI "shell" at bootup to go and run the install script. So for example you would get these files from Supermicro

# ls *.zip
STGi4S2B_6.01.zip
STGi4S3B_NUP_7.2.zip
STGi4S3C_NUP_8.3.zip

and extract this zipfile to a USB key (these are the 6.01 update)

# ls -l STGi4S2B_UEFI
total 4038
-rw-rw-r-- 1 root wheel 4194304 Feb 1 2018 710MI2B.bin
-rw-rw-r-- 1 root wheel 2535163 Jan 16 2018 BootIMG.FLB
-rw-rw-r-- 1 root wheel 231 Feb 8 2018 Instruction.txt
-rw-rw-r-- 1 root wheel 305 Feb 8 2018 map.txt
-rw-rw-r-- 1 root wheel 674 Feb 9 2018 nvmupdate.cfg
-rw-rw-r-- 1 root wheel 3754752 Jan 16 2018 nvmupdate64e.efi
-rw-rw-r-- 1 root wheel 5535 Feb 9 2018 Release Note.txt
-rw-rw-r-- 1 root wheel 49 Feb 8 2018 STGi4S.nsh

Then UEFI boot and inside the UEFI shell, you do "fs1:" (may be some other number) to change to the thumb drive, and when you see these files, you then run STGi4S.nsh which is a script that should update the card to 6.01

You then have to power cycle the host to complete the update. I would definitely just increment one firmware level at a time, and research in advance which one is recommended for use with whatever driver is in current FreeBSD.

I briefly looked to see if there was any clear indication about what NVM to use with FreeBSD. FreeBSD 12.2 ships with an updater port that installs 8.0, so, optimistically, I am guessing that going all the way wouldn't ruin things. But, I don't know, it's not my card, and you are warned to be careful because I am definitely handing you magic bits that are deeper than the average user normally works with. Here's a cup of gasoline in your left hand, a lit match in your right hand... try not to start yourself on fire. Get my drift? Good luck.

Any chance you can help me with the update files ? i only found the b4s (witch is broadcom) i have the i4s, tried updateing with intel x710 from intel but didn't work
 

Shimano

Dabbler
Joined
Jun 14, 2019
Messages
11
All good, i managed to get the update form SM but i also updated TrueNAS to 13.0 and the issue is gone with current firmware ! so there is no reason to update anymore if by any chance this happens again i will use the firmware update, we can close this threat thanks for all the help <3
 
Top