PCIe 10Gb ethernet card setup issue

[FrancoisC]

Good morning all,

I've lost few hours trying to find the solution to my problem. I've come to the point where I will need external support.
I bought for my NAS an ASUS XG-C100C PCIe card. This is based on Aquantia AQC-107 controller.

Starting up the NAS, the card is detected and I can access the GUI both from the Motherboard based 1 Gb Ethernet controller (em0) and the Asus PCIe 10 Gb controller (aq0). They have different but same subnet (I read it's not the way to go...).

However, all the traffic from my jails are going throught the 1 Gb controller and not the 10 Gb one.
I tried to create a "bridge0" with the aq0 NIC (and not the em0) and configure the jails with vnet0:bridge0 but it's not working. I think I managed to make it work few minutes but then it stopped working.

With ifconfig, I could see aq0 and the jails all under the same bridge0.

I also tried completely disabling the em0 controller in the bios. I can again access the GUI using the aq0 controller (that is in the bridge0 by default) but the jails can't get their IP (via DHCP?). Although they are all under bridge0 with ifconfig.

To be honest, I'm a bit lost of what I'm doing wrong. Could someone guide me on how to use the aq0 as the primary NIC and completely discard/disconnected em0?

This is my ifconfig with my jails trying to start and get en IP. (em0 disabled in bios, only aq0 available)

aq0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=8100b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWFILTER>
ether 7c:10:c9:61:36:8c
inet 192.168.1.80 netmask 0xffffff00 broadcast 192.168.1.255
media: Ethernet autoselect <full-duplex,rxpause,txpause> (10Gbase-T <full-duplex,rxpause,txpause>)
status: active
nd6 options=9<PERFORMNUD,IFDISABLED>

lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=680003<RXCSUM,TXCSUM,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x2
inet 127.0.0.1 netmask 0xff000000
groups: lo
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

pflog0: flags=0<> metric 0 mtu 33160
groups: pflog

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
ether 02:0c:b1:a7:c1:00
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
member: aq0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000
groups: bridge
nd6 options=1<PERFORMNUD>

vnet0.3: flags=8942<BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: associated with jail: NextCloud_Marketing as nic: epair0b
options=8<VLAN_MTU>
ether 06:d4:c4:75:66:44
hwaddr 02:d7:5d:34:2d:0a
groups: epair
media: Ethernet 10Gbase-T (10Gbase-T <full-duplex>)
status: active
nd6 options=1<PERFORMNUD>

vnet0.3 is a jail with fixed IP address (but it's not accessible from my network). Jails with DHCP can't start.
Thanks all

Thank you.
Francois

 

[Patrick M. Hausen]

You must disable hardware acceleration for your aq0 interface.
You must remove the IP address from the aq0 interface and put it on the bridge interface.

Please report back if that fixes things. It's the mandatory documented way on how to configure bridging in FreeBSD. I don't know why TrueNAS still does not enforce it.

 

[FrancoisC]

Will check this right away.

Else, i've seen this: https://www.truenas.com/community/t...installation-failing-after-nic-upgrade.95788/

Trying your proposal right now.
Thanks @Patrick M. Hausen

 

[FrancoisC]

@Patrick M. Hausen.... It works... Thanks.

What I did: Turned off hardware offloading
Added tunables: if_atlantic_load YES LOADER
Deactivated previous Ethernet card in Bios.

I'm on 10gb!!!

 

[FrancoisC]

Sadly, it broked during the night (jails were not available but GUI yes).
Rebooting the NAS solved (temporarily?) the issue.

Monitoring it today... Else will switch to the more expensive X550...

 

[Patrick M. Hausen]

Did you statically create the bridge interface and assign the IP address to the bridge instead of the physical IF?

 

[FrancoisC]

No, I let the "automatic" bridge0 created by Freenas to handle the IP.
But the physical IF is handling the GUI, assigning the IP to the bridge will do the same?

so I should create a bridge0?

 

[Patrick M. Hausen]

Yes. Set all jails and VMs (if you have any) autostart to off and reboot the system. This way it will come up without a bridge interface.
Then manually create an interface named bridge0 with aq0 as the only member. Remove the IP address from aq0 and apply it to bridge0.
That's what the two step "test" and "save" method in the network interface settings is for. You will not lose connectivity and of course the NAS will communicate happily for shares and the UI with the IP address on the bridge.

After a test reboot you can re-enable your jails.

This is a necessary step. Bridge member interfaces MUST NOT HAVE AN IP ADDRESS. Period. Otherwise things will break. Documented, repeatedly confirmed by FreeBSD core developers ... I'm tired of re-telling this again and again and again. Sorry for shouting.

HTH,
Patrick

P.S. A link to the relevant FreeBSD discussion can be found here: https://www.truenas.com/community/threads/core-bridging-setup-still-in-need-of-improvement.92476/

 

[FrancoisC]

All done, it works. Testing it on the long run.

bridge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
description: Bridge 0
ether 02:0c:b1:a7:c1:00
inet 192.168.1.49 netmask 0xffffff00 broadcast 192.168.1.255
id 00:00:00:00:00:00 priority 32768 hellotime 2 fwddelay 15
maxage 20 holdcnt 6 proto stp-rstp maxaddr 2000 timeout 1200
root id 00:00:00:00:00:00 priority 32768 ifcost 0 port 0
member: vnet0.4 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 8 priority 128 path cost 2000
member: vnet0.3 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 7 priority 128 path cost 2000
member: vnet0.2 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 6 priority 128 path cost 2000
member: vnet0.1 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 5 priority 128 path cost 2000
member: aq0 flags=143<LEARNING,DISCOVER,AUTOEDGE,AUTOPTP>
ifmaxaddr 0 port 1 priority 128 path cost 2000
groups: bridge
nd6 options=9<PERFORMNUD,IFDISABLED>

Thank you again!

 

[FrancoisC]

Me again...
Not stable. Lost all connectivity (incl. GUI) after few hours (<2).

I get this on the screen continously:

 

[Patrick M. Hausen]

The aq driver is considered experimental. Possibly pick a better supported card. I always wonder why people buy these things at all? Isn't it obvious to look at hardware recommendations for a particular OS first?

 

[jgreco]

The aq driver is considered experimental. Possibly pick a better supported card. I always wonder why people buy these things at all? Isn't it obvious to look at hardware recommendations for a particular OS first?

Back when the forum started, it was real "wild west" full of crazy AMD APU setups, mid-2000 era desktops, etc., most of which had horrible problems. Many of those users left and never came back.

For awhile, we had this much more under control as I introduced people to Supermicro server boards, and Cyberjock heavily pushed people at my original "Hardware Suggestions" sticky. Unfortunately, this was a rough process, but the people who followed the advice tended to wind up with really good NAS units and had good outcomes.

Unfortunately, the kinder, gentler tone iX wants to see on the forum makes it less acceptable to call out users for the sort of "wish-buying" that ends up with the less desirable hardware setups, and with AMD eating into Intel's low-end Xeon market, people want to use the cheapest junk they can find again. An AMD board typically will NOT have the Intel ethernet parts on it, so you end up needing to add something on your own to supplement the Realtek garbage present by default, so ... well anyways we had a good run of "king Intel" for awhile, but I think stormier times are ahead.

It would be awesome if everything just worked all the time of course.

 

[FrancoisC]

The aq driver is considered experimental. Possibly pick a better supported card. I always wonder why people buy these things at all? Isn't it obvious to look at hardware recommendations for a particular OS first?

Well, I understand your point. Thing is that, especially since I'm not a NAS guru, I'm looking like probably several users here the best "bang for my bucks" options. Most OS nowadays do have extensive hardware support so that's also a bit new for me (well... I started to play with computers with a 386/486... and drivers were a mess, especially with Sound Blaster cards )

And as I'm not living from building and maintaining NAS, I'm doing my best to read/understand before buying/building things but sometimes all information is not that straightforward to find.

I'm thanking you for your time and I hope other people reading this will wait. And I'll buy the more expensive Intel card.

Cheers,

Francois

 

[flashdrive]

@FrancoisC

I have tried the ASUS XG-C100C PCIe card with Aquantia AQC-107 on TN Core myself and returned it due to "cost". No experience with stability.

In terms of "bang for your buck" 10 GBit ethernet copper based:

If you want "working with a tweak" for home usage this is the way I can confirm working in my setup:

2x 10 GBit Ports

HP 533FLR-T 2-Port 10GbE PCI-Express x8 Network Adapter doesn't show up

Hello, I have bought 2 x HP 533FLR-T 2-Port 10GbE PCI-Express x8 Network adapter cards Both of them fail to show up in TrueNAS. I have not connected a LAN cable to either of them. How may I install this card? What I have tried: 1. shell - ifconfig em0 is the mainboard onboard NIC...

www.truenas.com

and necessary DIY adapter:

HP 533FLR-T 2-Port 10GbE PCI-Express x8 Network Adapter doesn't show up

Hello, I have bought 2 x HP 533FLR-T 2-Port 10GbE PCI-Express x8 Network adapter cards Both of them fail to show up in TrueNAS. I have not connected a LAN cable to either of them. How may I install this card? What I have tried: 1. shell - ifconfig em0 is the mainboard onboard NIC...

www.truenas.com

And as always: listen to the Pro's - again thank you to @jgreco and @Patrick M. Hausen

Howto bridge:

SOLVED - Network - replace switch with internal NIC & routing - howto?

tldr: solution / proper howto: https://www.truenas.com/community/threads/network-replace-switch-with-internal-nic-routing-howto.96040/post-664230 Hello, I want to remove a LAN switch out of the setup. Situation: TrueNAS host: 1 onboard NIC with 1x LAN port 1 PCIe NIC with 2x LAN port ->...

www.truenas.com

 

[jgreco]

I'm thanking you for your time and I hope other people reading this will wait. And I'll buy the more expensive Intel card.

They shut the forum down for upgrade yesterday just as I started to write:

You could also just head on over to the 10 Gig Networking Primer and see if you can find a reasonable used card...

And I had something else I was going to say that I don't remember, alas.