SOLVED Locking Interface Media Subtype to 10000Mb/s

[serfnoob]

Hello,
I've had my TrueNAS up for about a week, it boots.

My build

Asrock Rack D1541D4U-2T8R
Intel Xeon D 1541
2 x 16GB Micron ECC RAM
pool1 is a single raidz2 vdev of 8 x 8TB Seagate Ironwolf Pro
pool2 is a mirror of 2 x Samsung SSD 980 500GB

Onboard LSI 3008, cross flashed to IT mode (I dont know how I did it, but it was a nice win for a noob)

Onboard Intel x540.
enp10s0f1 to my router
enp10s0f0 direct attached to a mac pro 5,1 via sonnet solo 10G Solo.

Primary use: Video editing
Secondary: Centralised share point for the house
Tertiary: Multi user Time machine share

Is it possible to set/define the "media Subtype" in TrueNAS Scale?
With MacPro powered OFF Dashboard for enp10s0f0 displays
"Media Subtype: Unknown Twisted Pair"

With MacPro powered ON Dashboard for enp10s0f0 displays
"Media Subtype: 100Mb/s Twisted Pair"
Speeds reflect the connection

If I reboot the NAS while the MacPro is on the Dashboard for enp10s0f0 displays
"Media Subtype: 10000Mb/s Twisted Pair"
and stays like that until I reboot the MacPro.

As it stands if I want to get 10GbE connection I need to reboot the NAS after I boot my edit suite.

I have very limited command line experience but willing to learn what I need to get this box fast & stable.
So far I stumbled my way into cross flashing the LSI3008, almost understand how to iperf3 and cant for the life of me get ozone to work
The CLI and I are not friends yet.

Thanks for the help,
Owen.


P.s I have more questions. I will try to ration them out and not overwhelm all you excellent people on here helping traffics like me.

 

[Arwen]

The program ethtool can do what you want, though it is not really supported to do such with TrueNAS SCALE. Meaning on reboot of your NAS, any changes would be reverted unless it was done through the GUI or the supported CLI.

Don't know why TrueNAS SCALE is not re-negotiating 10Gbps on boot of your Mac. Perhaps you have found a bug.

 

[Kris Moore]

Yea, that could very likely be a legitimate bug. Suggest you file a ticket on https://jira.ixsystems.com so we can investigate...

 

[serfnoob]

Yea, that could very likely be a legitimate bug. Suggest you file a ticket on https://jira.ixsystems.com so we can investigate...

Thanks Kris,
I have logged a bug report.
Looks like there are a few bug through the Interface section. I couldn’t change the default ip for that connection either. It would stay spinning wheel for the 60 seconds you need to confirm changes while in the “test” mode.
I managed to update the ip in the console and it’s been persistent, so it’s just a GUI issue.

Kind regards.

 

[serfnoob]

The program ethtool can do what you want, though it is not really supported to do such with TrueNAS SCALE. Meaning on reboot of your NAS, any changes would be reverted unless it was done through the GUI or the supported CLI.

Don't know why TrueNAS SCALE is not re-negotiating 10Gbps on boot of your Mac. Perhaps you have found a bug.

I can run ethtool in shell. I see parts that look like it would stop auto negotiate. How would I write the code and make persistent? Asking a lot from our new friendship… sorry and thanks.

 

[Arwen]

Okay, my output from my desktop Linux, (aka NOT TrueNAS SCALE), has the following. Note that my desktop is advertising link mode of 10/100 at either Half or Full duplex, or 1000 at Full duplex.

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

So, in this case I want to remove both 10 & 100Mbps, both Half & Full duplex and only use 1000Mbps, (aka 1Gbps);

root@arwen:~# ethtool -s enp3s0 advertise 0x020

Now the output is as below, I only advertise 1000baseT/Full;

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

The actual settings are in the manual page for "ethtool". Here are some relevant to you;

Code:

                  0x1000                      10000baseT Full
                  0x40000                     10000baseKX4 Full
                  0x80000                     10000baseKR Full
                  0x100000                    10000baseR_FEC
                  0x40000000000               10000baseCR Full
                  0x80000000000               10000baseSR Full
                  0x100000000000              10000baseLR Full
                  0x200000000000              10000baseLRM Full
                  0x400000000000              10000baseER Full

There is debate on whether disabling auto-negotiating is a good idea or not. But, today, >=10Gbps REQUIRES auto-negotiation. So, that is what I have shown, simply removing the lower speeds but still negotiating link mode. However, disabling auto-negotiation might work: as they say, your mileage may very.

 

[serfnoob]

Okay, my output from my desktop Linux, (aka NOT TrueNAS SCALE), has the following. Note that my desktop is advertising link mode of 10/100 at either Half or Full duplex, or 1000 at Full duplex.

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

So, in this case I want to remove both 10 & 100Mbps, both Half & Full duplex and only use 1000Mbps, (aka 1Gbps);

root@arwen:~# ethtool -s enp3s0 advertise 0x020

Now the output is as below, I only advertise 1000baseT/Full;

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

The actual settings are in the manual page for "ethtool". Here are some relevant to you;

Code:

                  0x1000                      10000baseT Full
                  0x40000                     10000baseKX4 Full
                  0x80000                     10000baseKR Full
                  0x100000                    10000baseR_FEC
                  0x40000000000               10000baseCR Full
                  0x80000000000               10000baseSR Full
                  0x100000000000              10000baseLR Full
                  0x200000000000              10000baseLRM Full
                  0x400000000000              10000baseER Full

There is debate on whether disabling auto-negotiating is a good idea or not. But, today, >=10Gbps REQUIRES auto-negotiation. So, that is what I have shown, simply removing the lower speeds but still negotiating link mode. However, disabling auto-negotiation might work: as they say, your mileage may very.

Amazing, I’ll need a day to wrap my noob brain around this but it looks very promising.

I was thinking ther may be some tom foolery possible with a VLAN and a VM somehow sitting in the 10GbE network just pinging to keep it alive.
But I don’t know how to do any of those things , or if even has a chance of working, also seems like a much steeper learning curve.

I thank you again.

 

[Bonnie Follweiler]

Thanks Kris,
I have logged a bug report.
Looks like there are a few bug through the Interface section. I couldn’t change the default ip for that connection either. It would stay spinning wheel for the 60 seconds you need to confirm changes while in the “test” mode.
I managed to update the ip in the console and it’s been persistent, so it’s just a GUI issue.

Kind regards.

As you run into GUI issues, like the one you described, would you file a ticket on https://jira.ixsystems.com with screenshots of the issue so we can address them as well please?

 

[serfnoob]

Okay, my output from my desktop Linux, (aka NOT TrueNAS SCALE), has the following. Note that my desktop is advertising link mode of 10/100 at either Half or Full duplex, or 1000 at Full duplex.

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

So, in this case I want to remove both 10 & 100Mbps, both Half & Full duplex and only use 1000Mbps, (aka 1Gbps);

root@arwen:~# ethtool -s enp3s0 advertise 0x020

Now the output is as below, I only advertise 1000baseT/Full;

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

The actual settings are in the manual page for "ethtool". Here are some relevant to you;

Code:

                  0x1000                      10000baseT Full
                  0x40000                     10000baseKX4 Full
                  0x80000                     10000baseKR Full
                  0x100000                    10000baseR_FEC
                  0x40000000000               10000baseCR Full
                  0x80000000000               10000baseSR Full
                  0x100000000000              10000baseLR Full
                  0x200000000000              10000baseLRM Full
                  0x400000000000              10000baseER Full

There is debate on whether disabling auto-negotiating is a good idea or not. But, today, >=10Gbps REQUIRES auto-negotiation. So, that is what I have shown, simply removing the lower speeds but still negotiating link mode. However, disabling auto-negotiation might work: as they say, your mileage may very.

Thanks Arwen.
Looks like you've solved it.
System Settings > Advanced > Init/Shutdown Scripts
ethtool -s enp10s0f0 advertise 0x1000

Another problem down... :)

 

[Arwen]

@serfnoob - Your welcome.

And thank you for following up & letting us know the results.

If you still see GUI issues with SCALE, be sure to report them in Jira. You won't be the only one to find them... but others may not have people here to help you create a bug report. (Like they said above, screen shots of the appropriate item and what is failing will be a good start in Jira.)

 

[duxnobis13]

Hi. Sorry to jump in here but I have a very similar issue with my QNAP 10GbE card. I have two separate systems both up to date. One case hosts a proxmox system and another case hosts a truenas sacele system. Both PCs host each one mentioned QNAP 10GbE card. The cards are wired with a Cat.6e cable and both systems are linked through a QNAP 10GbE switch. So all the wireing is 10GbE. Now when I setup the Proxmox I managed the card and network to run at desired speed. In Truenas I can only get 100Mb/s. I tried to set it via ethtool but it does not seem to write my requested amendment. Even the suggested script did not help.

I would like to ask you for help. Please let me know what to run in shell and what to provide in order to get dedicated help. Somehow I cant copy paste from shell. Any Idea how to?

 

[Arwen]

I listed the speed & duplex modes, which vary depending on interface type. You indicate 10Gbit/ps, E, but don't qualify if it's copper, or one of the various fiber connections. (Or even twin co-axe.) Using the wrong setting probably does nothing. I am guessing copper as that would explain 100Mbit/ps speed.

As for copying and pasting, you don't from the console or web GUI. You do so from a SSH connection using a SSH client program like PuTTY, MobaXTerm, SecureCRT, etc... Those allow copying and pasting.


Because you list the current speed as 100Mbit/ps, I would guess you have a cable, or port problem. 10/100Mbit/ps speeds only need 2 pairs of the 8p8c cable & connector. While >=1Gbit/ps requires all 4 pairs. So my guess is that the cable, server port or switch port is bad and not passing through 1 or more of the 4 extra wires needed for >=1Gbit/ps.

I once had a Sun SPARC Station 5 that would not join NIS, (this was a LONG time ago). Workstation would hang at boot, at NIS. A fellow Unix SysAdmin could not figure it out. A simple snoop on the SS5 showed it receiving packets fine. But, an examination of the SS5 Ethernet port showed a bent pin. When the pin was straightened out, that restored functionality.

 

[serfnoob]

Hi. Sorry to jump in here but I have a very similar issue with my QNAP 10GbE card. I have two separate systems both up to date. One case hosts a proxmox system and another case hosts a truenas sacele system. Both PCs host each one mentioned QNAP 10GbE card. The cards are wired with a Cat.6e cable and both systems are linked through a QNAP 10GbE switch. So all the wireing is 10GbE. Now when I setup the Proxmox I managed the card and network to run at desired speed. In Truenas I can only get 100Mb/s. I tried to set it via ethtool but it does not seem to write my requested amendment. Even the suggested script did not help.

I would like to ask you for help. Please let me know what to run in shell and what to provide in order to get dedicated help. Somehow I cant copy paste from shell. Any Idea how to?

Hi duxnobis,
There is a lot to unpack here, lets start with the shell. By design copy and paste was removed from the shell. Use ssh for long scripts or persistent monitoring you'll. https://youtu.be/qWKK_PNHnnA
You'll need to start SSH in your TrueNAS web GUI under Services>SSH

Beyond that more detail about your system might help. Are you running Core or Scale?
Have you tried connecting to the Proxmox box directly from a 10Gbe PC?

At the very bottom of this I think the problem might be that the QNAP cards might be made by Aquantia, and the driver support isn't as solid as with Intel or Chelsio.

Do you have the option of buying of second had 10gbe NICs from ebay. you can get them under 100.

I'm only learning as I go and way out of my depth here.

 

[silvioslim]

Okay, my output from my desktop Linux, (aka NOT TrueNAS SCALE), has the following. Note that my desktop is advertising link mode of 10/100 at either Half or Full duplex, or 1000 at Full duplex.

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Advertised pause frame use: Symmetric Receive-only
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

So, in this case I want to remove both 10 & 100Mbps, both Half & Full duplex and only use 1000Mbps, (aka 1Gbps);

root@arwen:~# ethtool -s enp3s0 advertise 0x020

Now the output is as below, I only advertise 1000baseT/Full;

Code:

root@arwen:~# ethtool enp3s0
Settings for enp3s0:
    Supported ports: [ TP MII ]
    Supported link modes:   10baseT/Half 10baseT/Full
                            100baseT/Half 100baseT/Full
                            1000baseT/Full
    Supported pause frame use: Symmetric Receive-only
    Supports auto-negotiation: Yes
    Supported FEC modes: Not reported
    Advertised link modes:  1000baseT/Full
    Advertised pause frame use: No
    Advertised auto-negotiation: Yes
    Advertised FEC modes: Not reported
    Link partner advertised link modes:  10baseT/Half 10baseT/Full
                                         100baseT/Half 100baseT/Full
                                         1000baseT/Half 1000baseT/Full
    Link partner advertised pause frame use: Symmetric Receive-only
    Link partner advertised auto-negotiation: Yes
    Link partner advertised FEC modes: Not reported
    Speed: 1000Mb/s
    Duplex: Full
    Port: MII
    PHYAD: 0
    Transceiver: external
    Auto-negotiation: on
    Supports Wake-on: pumbg
    Wake-on: d
    Current message level: 0x00000033 (51)
                   drv probe ifdown ifup
    Link detected: yes

The actual settings are in the manual page for "ethtool". Here are some relevant to you;

Code:

                  0x1000                      10000baseT Full
                  0x40000                     10000baseKX4 Full
                  0x80000                     10000baseKR Full
                  0x100000                    10000baseR_FEC
                  0x40000000000               10000baseCR Full
                  0x80000000000               10000baseSR Full
                  0x100000000000              10000baseLR Full
                  0x200000000000              10000baseLRM Full
                  0x400000000000              10000baseER Full

There is debate on whether disabling auto-negotiating is a good idea or not. But, today, >=10Gbps REQUIRES auto-negotiation. So, that is what I have shown, simply removing the lower speeds but still negotiating link mode. However, disabling auto-negotiation might work: as they say, your mileage may very.

Hi there, do you know if exist the same command "ethtool" for the Core version?

 

[Patrick M. Hausen]

Almost every parameter of a network interface is set with ifconfig in FreeBSD. See the man pages for ifconfig and the interface/driver in question to view all available options.

E.g. to lock the "em0" interface to 1000 full-duplex operation use:

Code:

ifconfig em0 mediatype 1000baseTX mediaopt full-duplex

To persist that setting use Network > Interfaces, edit the interface "em0" and put mediatype 1000baseTX mediaopt full-duplex in the "options" field.