Broadcom 57810s- 10GBASE T Network interface could not be accessed

[Apprisco]

Hello, I've heard from many people that since TrueNAS 12, 57810s was working fine.
This is a 10Gbps dual port ethernet card that I've had for a while and confirmed working.
I decided to ditch my complex network setup and instead directly go with a 10Gbps switch to simplify things a bit.
This means that I only connected the 57810s card to the switch instead of the built in *shudder* RealTek Gigabit port on my Asus X99-a II board.
Full system details:
Xeon E5 2696 V4 FCLGA2011-3
96GB 6x16GB 2133MHZ ECC UDIMM
GTX 1080
GT 730
LSI 16 drive SAS HBA
Broadcom 57810S Dell Dual 10GBASE-T card

Note that I first installed truenas having connected the realtek to the switch.
Is there any way to get TrueNAS to get an IP address using the ethernet card?

 

[jgreco]

The Broadcom cards are generally a problematic and bad choice. Please visit the 10 GIg Networking Primer for a discussion of cards known to work well. Additionally, please note that 10GBase-T is usually problematic because of its low adoption rate, pricey gear, and other related issues.

10 Gig Networking Primer

This is a discussion of high speed networking for newcomers, with specific emphasis on practical small- to medium-scale deployments for home labs or small office users. It originated with a forum thread located here that has many pages of...

www.truenas.com

 

[Apprisco]

Hello, I've heard from many users that do indeed use this card, so I'd like to get it working INSTEAD of being told to buy another compatible one.

 

[Apprisco]

Also, I prefer dual port 10GBASE-T than SFP+ as my switch is a 10GBASE-T switch.

 

[Ericloewe]

I've heard from many users that do indeed use this card

Many? Sure they're cheap, but even so they don't show up here often at all. I think we see Mellanox more frequently than we see BCM57810 NICs.

 

[Apprisco]

I've heard from five or so users online who got it working after TrueNAS 12. I'm just wondering what they did, since noone has a comprehensive guide.

 

[jgreco]

Hello, I've heard from many users

Name some. Provide links.

I get to be skeptical because I've been doing 10G support on these forums longer than ... maybe anyone.

I'd like to get it working INSTEAD of being told to buy another compatible one.

Then by all means, go ahead and do so. The problem is that when *I* spot a card as being known to be problematic, it's because I've probably seen dozens or even hundreds of mentions of it, and it has stuck in my mind as problematic. This is the same way that we originally spotted systems with less than 8GB of RAM as problematic, it was my informal collation of observations in my head.

The state of affairs at the time of authorship (almost ten years ago) was that 57810 with 10GBase-T had not been tested and wasn't known to work.

Support of Dell M620 broadcom BCM57810

David no longer works for Broadcom, and I have no idea what sort of shape this is currently in. This was originally QLogic silicon and I believe that driver bits originally came from there. Unfortunately, there's been some drift in the kernel support for ethernet, requiring driver updates to be made.

A common problem with older ethernet cards is that the driver developers move on to other jobs, taking with them the highly specialized knowledge needed to write that driver. Manufacturers have little incentive to hire someone new, usually at a higher salary, who is able to jump right in and pick up on what's going on. They mostly want to sell new cards, so support for older cards is often poor to nonexistent.

 

[Apprisco]

Name some. Provide links.

I get to be skeptical because I've been doing 10G support on these forums longer than ... maybe anyone.



Then by all means, go ahead and do so. The problem is that when *I* spot a card as being known to be problematic, it's because I've probably seen dozens or even hundreds of mentions of it, and it has stuck in my mind as problematic. This is the same way that we originally spotted systems with less than 8GB of RAM as problematic, it was my informal collation of observations in my head.

The state of affairs at the time of authorship (almost ten years ago) was that 57810 with 10GBase-T had not been tested and wasn't known to work.

Support of Dell M620 broadcom BCM57810

David no longer works for Broadcom, and I have no idea what sort of shape this is currently in. This was originally QLogic silicon and I believe that driver bits originally came from there. Unfortunately, there's been some drift in the kernel support for ethernet, requiring driver updates to be made.

A common problem with older ethernet cards is that the driver developers move on to other jobs, taking with them the highly specialized knowledge needed to write that driver. Manufacturers have little incentive to hire someone new, usually at a higher salary, who is able to jump right in and pick up on what's going on. They mostly want to sell new cards, so support for older cards is often poor to nonexistent.

Reddit - Dive into anything

www.reddit.com

Reddit - Dive into anything

www.reddit.com

I cba to go find more of the ones I found.

See, there's no guide at all to make a card work- nothing to go off of, even compiling drivers. I'd gladly help out coding one if needed even, but if there's nothing to go off of then how would I start?

 

[Apprisco]

Also, I haven't even been able to try the enable bxe option in the kernel as again, there's no guides on how to do this at all. The amount of information that's missing here is insane.

 

[jgreco]

https://www.reddit.com/r/truenas/co...?utm_source=reddit&utm_medium=web2x&context=3
https://www.reddit.com/r/freenas/comments/ldkwv5/psa_dell_w1gcr_broadcom_57810s_chipset_10gbaset/

Neither of these is "hearing from" a user, and one of them isn't even for TrueNAS CORE, because you did clearly say TrueNAS 12 in your opening post. You just saw some random posts on Reddit. Hardly a ringing endorsement, but if you think otherwise, then clearly it works great and we don't need to continue this thread... right?

See, there's no guide at all to make a card work- nothing to go off of, even compiling drivers. I'd gladly help out coding one if needed even, but if there's nothing to go off of then how would I start?

Probably start off in man ifnet(9) (this means to type "man 9 ifnet" at the shell prompt) and then in the kernel architecture handbook at https://docs.freebsd.org/en/books/arch-handbook/driverbasics/

I'd say it's also helpful to look to see if there's any device drivers available in OpenBSD, NetBSD, DragonflyBSD, or Linux, as these are often useful starting points.

Basically this boils down to whether or not there's a currently viable driver (observation of random user experiences suggests: not) or whether or not someone sufficiently talented and interested can fix the existing driver (long experience says: unlikely). If you are not this sufficiently interested and talented person, the best solution I'm aware of is to pick a card known to work. Which is, I believe, the advice I originally offered.

 

[Apprisco]

There is an option in the kernel to turn on bxe. How do I compile it with this? Also, your nickname checks out: Resident Grinch.
Not to mention, https://github.com/truenas/core-build/blob/master/build/profiles/freenas/kernel/TRUENAS.amd64 contains the BCM 578xx series cards, so I don't see why it shouldn't be supported.

 

[jgreco]

There is an option in the kernel to turn on bxe. How do I compile it with this?

TrueNAS is an appliance, you don't have an option to compile it. Developers need to build it on an external platform and then get a new appliance image.

I'm not seeing any sign that bxe is disabled. The OS as imported from upstream FreeBSD is clearly in possession of bxe, as observable in

os/sys/amd64/conf/GENERIC at truenas/13-stable · truenas/os

FreeBSD base for TrueNAS Core. Contribute to truenas/os development by creating an account on GitHub.

github.com

I do not have a development environment with the latest code set up, so I'm not sure what is in the TRUENAS kernel config itself. What makes you say that there is "an option in the kernel to turn on bxe"?

 

[Apprisco]

TrueNAS is an appliance, you don't have an option to compile it. Developers need to build it on an external platform and then get a new appliance image.

I'm not seeing any sign that bxe is disabled. The OS as imported from upstream FreeBSD is clearly in possession of bxe, as observable in

os/sys/amd64/conf/GENERIC at truenas/13-stable · truenas/os

FreeBSD base for TrueNAS Core. Contribute to truenas/os development by creating an account on GitHub.

github.com

I do not have a development environment with the latest code set up, so I'm not sure what is in the TRUENAS kernel config itself. What makes you say that there is "an option in the kernel to turn on bxe"?

My inexperience is clearly in play here, I just found an article that says that if bxe: LOAD or whatever is now in the kernel bootloader for certain.
Also, to be clear, I'm attempting to get this working on TrueNAS Scale/Core, either one that accepts this card will work.
I have tried both indeed.

 

[jgreco]

My inexperience is clearly in play here,

That's fine. My not having one of these cards to play with puts me at a similar disadvantage. But maybe we can fill in the gaps.

I just found an article that says that if bxe: LOAD or whatever is now in the kernel bootloader for certain.

That's weird, since it looks like it is in the GENERIC kernel and therefore ought to be compiled in. But let's experiment. The above sounds like a reference to creating a kernel tunable in /boot/loader.conf such as

if_bxe_load="YES"

This is NOT what you do in TrueNAS, but I include it in case it helps you figure this out. Inside TrueNAS, you go into the System->Tunables GUI and there is an option for adding a LOADER tunable, variable is "ix_bxe_load", value is "YES", and then reboot.

This doesn't make any real sense to me though. If it's not there, it is not there for a reason. If there's a reason, it probably shouldn't be used. If it shouldn't be used, then forcibly using it opens you up to unknown risk. And that's just not worth it.

What you might try first, to clarify the situation, is to try "ifconfig bxe0" from the console to see if there is such an interface already. If not, then "kldload if_bxe" might provide some useful hints.

 

[Apprisco]

Thank you for the advice! I'll let you know how it goes soon. Much more helpful than just telling me to get a new card :) Also, I've gotten the card to be detected in TrueNAS Scale, just couldn't get any traffic going on it, and had to use the built in RealTek LAN to even get the web interface on.
I assume even if I get TrueNAS itself working with the card, there won't be any hope of getting it working if I can't update the card's firmware. Is there a way to run upgrade packages for Red Hat Linux/ GnuPG on TrueNAS?

 

[Apprisco]

Update: seems like tunables only work on Core, and I'd prefer scale...

This is as far as I have gotten on the latest RC of truenas scale.

 

[Apprisco]

Another update: I've found the official linux drivers for this adapter including source code thanks to Dell. The specific card I'm using is the dell QLogic version of this broadcom chipset, so it may work. Is there a way to install drivers using this method? Also, so how would I go about upgrading the firmware?

 

[Apprisco]

That's fine. My not having one of these cards to play with puts me at a similar disadvantage. But maybe we can fill in the gaps.



That's weird, since it looks like it is in the GENERIC kernel and therefore ought to be compiled in. But let's experiment. The above sounds like a reference to creating a kernel tunable in /boot/loader.conf such as

if_bxe_load="YES"

This is NOT what you do in TrueNAS, but I include it in case it helps you figure this out. Inside TrueNAS, you go into the System->Tunables GUI and there is an option for adding a LOADER tunable, variable is "ix_bxe_load", value is "YES", and then reboot.

This doesn't make any real sense to me though. If it's not there, it is not there for a reason. If there's a reason, it probably shouldn't be used. If it shouldn't be used, then forcibly using it opens you up to unknown risk. And that's just not worth it.

What you might try first, to clarify the situation, is to try "ifconfig bxe0" from the console to see if there is such an interface already. If not, then "kldload if_bxe" might provide some useful hints.

This is even crazier. Setting a static ip for the network works, but still no traffic if I remove the realtek:

How in the world is it connecting via 50.84 but without a link?

 

[jgreco]

You set up a /32 netmask.

Also you have two IP addresses on the same IPv4 network, that's not allowed. You should place your interfaces on two different IP networks, and the /32 is likely breaking things in unusual ways. I can't predict the exact results of the /32 on Linux. But please refer to

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

 

[Apprisco]

You set up a /32 netmask. Also you have two IP addresses on the same IPv4 network, that's not allowed. You should place your interfaces on two different IP networks, and the /32 is likely breaking things in unusual ways. I can't predict the exact results of the /32 on Linux. But please refer to

Resource - Multiple network interfaces on a single subnet

We've had a number of people over the years bring up the topic of multiple network interfaces on a single subnet. Often done by Windows administrators or storage admins, where it works to varying degrees, this is sadly not proper IP networking...

www.truenas.com

I've also tried 24. I remove the original link's ip address, but I can't connect to the 84 ip. My last hope is the linux driver meant for red hat, as I was able to unlock apt-get and install rpm without breaking the current truenas install, I will be giving this a go.