Enable new network card with full 10 gbit bandwidth

T

turborierer

Cadet
Joined
Dec 2, 2022
Messages
3
Dear all,

the problem: TrueNAS Scale Sever with SuperMico MB was running good but the network speed was to low with 1 Gbit ethernet. So I put in a 10 Gbit PCI card, connected the server via fiber optic cable to the switch which is also indicating a 10 gibt connection. But if I run a iperf3 server on TrueNAS I just get 1 Gbit. Also the VM running on this server is showing that the "br0" where its attached only got 1 Gbit.

So I did some research and I think its related to the initial NIC which is somehow stick to the old network interface or includes it and holding the NIC back from using the greater bandwidth of the new card. Some redundancy setting? Anyway I'm lost to tell TrueNAS how to only use the new network card with full bandwidth.

Any help would be highly appreciated how to stick enp1s0f0 to br0 and tell them to use the full 10 Gbit.

Here come some prints about my setup:

Code:
lspci | egrep -i --color 'network|ethernet'
01:00.0 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
01:00.1 Ethernet controller: Intel Corporation 82599ES 10-Gigabit SFI/SFP+ Network Connection (rev 01)
06:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection
07:00.0 Ethernet controller: Intel Corporation 82574L Gigabit Network Connection




Code:
sudo lshw -class network -short
H/W path             Device         Class          Description
==============================================================
/0/100/1/0           enp1s0f0       network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/100/1/0.1         enp1s0f1       network        82599ES 10-Gigabit SFI/SFP+ Network Connection
/0/100/1c.4/0        enp6s0         network        82574L Gigabit Network Connection
/0/100/1c.5/0        enp7s0         network        82574L Gigabit Network Connection
/1                   br0            network        Ethernet interface
/2                   vnet0          network        Ethernet interface
/3                   kube-bridge    network        Ethernet interface
/4                   kube-dummy-if  network        Ethernet interface
/5                   vethade24a32   network        Ethernet interface
/6                   veth99e65218   network        Ethernet interface


Code:
ip link show
1: lo: <LOOPBACK,UP,LOWER_UP> mtu 65536 qdisc noqueue state UNKNOWN mode DEFAULT group default qlen 1000
    link/loopback 00:00:00:00:00:00 brd 00:00:00:00:00:00
2: enp6s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:30:50:14 brd ff:ff:ff:ff:ff:ff
3: enp7s0: <NO-CARRIER,BROADCAST,MULTICAST,UP> mtu 9000 qdisc pfifo_fast state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:25:90:30:50:15 brd ff:ff:ff:ff:ff:ff
4: enp1s0f0: <BROADCAST,MULTICAST> mtu 1500 qdisc mq state DOWN mode DEFAULT group default qlen 1000
    link/ether 00:11:0a:68:b5:64 brd ff:ff:ff:ff:ff:ff
5: enp1s0f1: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc mq master br0 state UP mode DEFAULT group default qlen 1000
    link/ether 00:11:0a:68:b5:65 brd ff:ff:ff:ff:ff:ff
6: br0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether 2a:5d:d5:66:f2:4c brd ff:ff:ff:ff:ff:ff
7: vnet0: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue master br0 state UNKNOWN mode DEFAULT group default qlen 1000
    link/ether fe:a0:98:3f:19:d9 brd ff:ff:ff:ff:ff:ff
8: kube-bridge: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 9000 qdisc noqueue state UP mode DEFAULT group default qlen 1000
    link/ether aa:14:80:2f:61:b9 brd ff:ff:ff:ff:ff:ff
9: kube-dummy-if: <BROADCAST,NOARP,UP,LOWER_UP> mtu 1500 qdisc noqueue state UNKNOWN mode DEFAULT group default
    link/ether 5e:c8:b7:c4:d5:c5 brd ff:ff:ff:ff:ff:ff
10: vethade24a32@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP mode DEFAULT group default
    link/ether ea:9f:bf:53:82:85 brd ff:ff:ff:ff:ff:ff link-netnsid 0
11: veth99e65218@if3: <BROADCAST,MULTICAST,UP,LOWER_UP> mtu 1500 qdisc noqueue master kube-bridge state UP mode DEFAULT group default
    link/ether 5e:4c:b2:a4:d6:b6 brd ff:ff:ff:ff:ff:ff link-netnsid 1


Code:
ip -br -c link show
lo               UNKNOWN        00:00:00:00:00:00 <LOOPBACK,UP,LOWER_UP>
enp6s0           DOWN           00:25:90:30:50:14 <NO-CARRIER,BROADCAST,MULTICAST,UP>
enp7s0           DOWN           00:25:90:30:50:15 <NO-CARRIER,BROADCAST,MULTICAST,UP>
enp1s0f0         DOWN           00:11:0a:68:b5:64 <BROADCAST,MULTICAST>
enp1s0f1         UP             00:11:0a:68:b5:65 <BROADCAST,MULTICAST,UP,LOWER_UP>
br0              UP             2a:5d:d5:66:f2:4c <BROADCAST,MULTICAST,UP,LOWER_UP>
vnet0            UNKNOWN        fe:a0:98:3f:19:d9 <BROADCAST,MULTICAST,UP,LOWER_UP>
kube-bridge      UP             aa:14:80:2f:61:b9 <BROADCAST,MULTICAST,UP,LOWER_UP>
kube-dummy-if    UNKNOWN        5e:c8:b7:c4:d5:c5 <BROADCAST,NOARP,UP,LOWER_UP>
vethade24a32@if3 UP             ea:9f:bf:53:82:85 <BROADCAST,MULTICAST,UP,LOWER_UP>
veth99e65218@if3 UP             5e:4c:b2:a4:d6:b6 <BROADCAST,MULTICAST,UP,LOWER_UP>


Code:
lshw -class network
*-network:0 DISABLED     
       description: Ethernet interface
       product: 82599ES 10-Gigabit SFI/SFP+ Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:01:00.0
       logical name: enp1s0f0
       version: 01
       serial: 00:11:0a:68:b5:64
       capacity: 10Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre 10000bt-fd
       configuration: autonegotiation=off broadcast=yes driver=ixgbe driverversion=5.10.131+truenas firmware=0x80000835, 1.2836.0 latency=0 link=no multicast=yes port=fibre
       resources: irq:33 memory:faa00000-faafffff ioport:ac00(size=32) memory:fa97c000-fa97ffff memory:fa980000-fa9fffff memory:c0000000-c00fffff memory:c0100000-c01fffff
  *-network:1
       description: Ethernet interface
       product: 82599ES 10-Gigabit SFI/SFP+ Network Connection
       vendor: Intel Corporation
       physical id: 0.1
       bus info: pci@0000:01:00.1
       logical name: enp1s0f1
       version: 01
       serial: 00:11:0a:68:b5:65
       size: 10Gbit/s
       capacity: 10Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi msix pciexpress vpd bus_master cap_list rom ethernet physical fibre 10000bt-fd
       configuration: autonegotiation=off broadcast=yes driver=ixgbe driverversion=5.10.131+truenas duplex=full firmware=0x80000835, 1.2836.0 latency=0 link=yes multicast=yes port=fibre speed=10Gbit/s
       resources: irq:79 memory:fa800000-fa8fffff ioport:a880(size=32) memory:fa77c000-fa77ffff memory:fa780000-fa7fffff memory:c0200000-c02fffff memory:c0300000-c03fffff
  *-network
       description: Ethernet interface
       product: 82574L Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:06:00.0
       logical name: enp6s0
       version: 00
       serial: 00:25:90:30:50:14
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=5.10.131+truenas firmware=1.8-0 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:16 memory:fbce0000-fbcfffff ioport:dc00(size=32) memory:fbcdc000-fbcdffff
  *-network
       description: Ethernet interface
       product: 82574L Gigabit Network Connection
       vendor: Intel Corporation
       physical id: 0
       bus info: pci@0000:07:00.0
       logical name: enp7s0
       version: 00
       serial: 00:25:90:30:50:15
       capacity: 1Gbit/s
       width: 32 bits
       clock: 33MHz
       capabilities: pm msi pciexpress msix bus_master cap_list ethernet physical tp 10bt 10bt-fd 100bt 100bt-fd 1000bt-fd autonegotiation
       configuration: autonegotiation=on broadcast=yes driver=e1000e driverversion=5.10.131+truenas firmware=1.8-0 latency=0 link=no multicast=yes port=twisted pair
       resources: irq:17 memory:fbde0000-fbdfffff ioport:ec00(size=32) memory:fbddc000-fbddffff
  *-network:0
       description: Ethernet interface
       physical id: 1
       logical name: br0
       serial: 2a:5d:d5:66:f2:4c
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=192.168.1.31 link=yes multicast=yes speed=10Gbit/s
  *-network:1
       description: Ethernet interface
       physical id: 2
       logical name: vnet0
       serial: fe:a0:98:3f:19:d9
       size: 10Mbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=tun driverversion=1.6 duplex=full link=yes multicast=yes port=twisted pair speed=10Mbit/s
  *-network:2
       description: Ethernet interface
       physical id: 3
       logical name: kube-bridge
       serial: aa:14:80:2f:61:b9
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=bridge driverversion=2.3 firmware=N/A ip=172.16.0.1 link=yes multicast=yes speed=10Gbit/s
  *-network:3
       description: Ethernet interface
       physical id: 4
       logical name: kube-dummy-if
       serial: 5e:c8:b7:c4:d5:c5
       capabilities: ethernet physical
       configuration: broadcast=yes driver=dummy driverversion=5.10.131+truenas ip=172.17.0.1
  *-network:4
       description: Ethernet interface
       physical id: 5
       logical name: vethade24a32
       serial: ea:9f:bf:53:82:85
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
  *-network:5
       description: Ethernet interface
       physical id: 6
       logical name: veth99e65218
       serial: 5e:4c:b2:a4:d6:b6
       size: 10Gbit/s
       capabilities: ethernet physical
       configuration: autonegotiation=off broadcast=yes driver=veth driverversion=1.0 duplex=full link=yes multicast=yes port=twisted pair speed=10Gbit/s
 
NugentS

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
Are both NICs active on the same logical network (e.g 192.168.0.x/24)
 
T

turborierer

Cadet
Joined
Dec 2, 2022
Messages
3
There is just one cable going from the new 10 Gbit card to the switch if its is what you mean. The old cable was removed.
The old network (ethernet form the motherboard) and the new PCI network card were connected to the same 192.168.1.x/24 network, yes.
 
Patrick M. Hausen

Patrick M. Hausen

Hall of Famer
Joined
Nov 25, 2013
Messages
7,776
turborierer said:
The old network (ethernet form the motherboard) and the new PCI network card were connected to the same 192.168.1.x/24 network, yes.
Click to expand...
That does not work. You cannot have more than one interface in the same network.
 
NugentS

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
@Patrick M. Hausen But the old interface will (should) be down and thus not in the routing tables as it doesn't have a cable attached.

@turborierer whats in the bridge br0?
 
T

turborierer

Cadet
Joined
Dec 2, 2022
Messages
3
br0 was created automatically and think this was done in the moment I created a VM on the true nas. So "br0" pops up as network interface in the VM.
Regarding old/new, actually just the new enp1s0f1 is working and the only interface in br0.
 
NugentS

NugentS

MVP
Joined
Apr 16, 2020
Messages
2,947
And what is the rest of your kit?
Please post a full hardware spec as per forum rules.
 
You must log in or register to reply here.

Important Announcement for the TrueNAS Community.

The TrueNAS Community has now been moved. This forum will now become READ-ONLY for historical purposes. Please feel free to join us on the new TrueNAS Community Forums.

Related topics on forums.truenas.com for thread: "Enable new network card with full 10 gbit bandwidth"

Similar threads

J
SOLVED Network device and interface.
Replies
6
Views
7K
jeremyoha450
J
M
Network failover trouble (active-backup)
Replies
14
Views
4K
MichaelDE
M
A
Trouble With Bridging
Replies
1
Views
3K
Im_JackJohnson
I
NickF
Weird Network Performance Problems?
Replies
6
Views
2K
NickF
NickF
kinroy
Bridge mode works in the vm to the internet and de smb share but freenas does not have internet anymore.
Replies
2
Views
2K
Samuel Tai
Samuel Tai
Top