Chelsio T520-CR + Ubiquity SFP+ Switch + FreeNAS 11.2 + Link Aggregation

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Ok, so let's start by saying I don't work with this stuff for a job and it's basically all brand new to me. After some time fighting things I have finally gotten everything set up and running(I think, it's only been a couple hours). I wanted to make a post documenting the struggles I had and the way things behaved.

My goal: I don't exactly need a 10gig network card but I thought it was going to be fun and was something new to play with. I already own the switch and was using lacp with my 1gig interfaces. Moving to 10gig would remove the network bottleneck and now my pool is the slow piece of the puzzle.

Hardware:
X10SRL, E5-1620v3, 128GB Samsung Memory, Ubiquity Unifi Switch 48, Supermicro 846 chassis, LSI 9211i(20.00.07.00 firmware), Chelsio T520-CR
The Chelsio T420-CR was what I initially bought but the ebay seller sent me the T420-SO-CR(no on-board memory and supports fewer connections) by mistake so I returned it. After that I noticed that the T520-CR is basically the same price now so that is what I eventually bought.

There are 3 parts to this adventure.
1. Broadcast storm
Ok so in my setup I have several jails that use vnet and vimage. When I first plugged in the new nic I just plugged in both interfaces and did not configure any link aggregation on the switch or in freenas. I figured I'll set it up as I go. Having 2 interfaces on the same subnet isn't necessary correct but I didn't think it was going to cause any problems, O boy was I wrong. In the process of setting everything up I needed to restart my Unifi5 jail so I could access the webui and setup the ports on the switch for aggregation. Well when you restart a jail and there are 2 interfaces on the same subnet this somehow creates a loop and broadcast storms the entire network and takes the whole thing down. And on top of this STP did not catch the loop because I guess it was in the jail networking stack or something(if anyone knows more please comment below). At one point my wifi access point even went offline from all the traffic, it was red in unifi.

Well after unplugging everything and rebooting things I got the network back up. To configure things I first setup aggregation on the switch ports, then plugged in the 10gig card and connected it to the switch. From here I used the console to configure lacp link aggregation and setup the interface, I used ipv4 and dhcp since I use mac address assigned ip's from the dhcp server on my network. So at this point I think I got things working.

2. Disable offloading
Taking a little side step, I noticed that in the console i was seeing
Code:
cxl0: tso4 disabled due to -txcsum.
cxl0: tso6 disabled due to -txcsum6.
cxl0: enable txcsum first

cxl1: tso4 disabled due to -txcsum.
cxl1: tso6 disabled due to -txcsum6.
cxl1: enable txcsum first

This really confused me but after some research I found out that it's most likely caused by my jails and how they use their own networking stack. So in the end this is no big deal other than some performance loss from no offloading.

3. Link State changed UP/DOWN
Now for the one that caused me the most problems. After I got the LAG setup and jails up and running I kepty seeing these log messages spamming /var/log/messages.

Logs from FreeNAS switch in /var/log/messages
Code:
Jan 14 10:45:09 tubby kernel: cxl0: link state changed to DOWN
Jan 14 10:45:09 tubby kernel: cxl0: link state changed to DOWN
Jan 14 10:45:10 tubby kernel: cxl0: link state changed to UP
Jan 14 10:45:10 tubby kernel: cxl0: link state changed to UP
Jan 14 10:45:10 tubby kernel: cxl0: link state changed to DOWN
Jan 14 10:45:10 tubby kernel: cxl0: link state changed to DOWN
Jan 14 10:45:11 tubby kernel: cxl0: link state changed to UP
Jan 14 10:45:11 tubby kernel: cxl0: link state changed to UP

Logs from Ubiquity switch in /var/log/messages
Code:
Jan 14 16:14:40 US-48 daemon.notice switch: TRAPMGR: Link Up: 0/50
Jan 14 16:14:40 US-48 daemon.notice switch: TRAPMGR: Link Down: 0/50
Jan 14 16:14:41 US-48 daemon.notice switch: TRAPMGR: Link Up: 0/50
Jan 14 16:14:41 US-48 daemon.notice switch: TRAPMGR: Link Down: 0/50
Jan 14 16:14:42 US-48 daemon.notice switch: TRAPMGR: Link Up: 0/50
Jan 14 16:14:43 US-48 daemon.notice switch: TRAPMGR: Link Down: 0/50

These would never stop. When I did a ping to the server I got about 25% packet loss so the interface was actually going down and affecting my network. When I unplugged cxl0 the messages went away and everything worked just fine using the cxl1 interface of the LAG. <= this part still confuses me why it was only the first interface. I took the card out and tried it in a linux server with a mikrotik switch and it worked just fine on both interfaces. So I knew the card and direct attach copper cables I was using worked and are not flawed. So I just started searching and reading, it took me about a week but I eventually found this thread:
https://community.ubnt.com/t5/UniFi...nt-2m-DAC-gt-Chelsio-T520-CR-link/m-p/2494470

And it was almost my exact problem. I bought some chelsio transceivers and 10gtek ubiquity transceivers and some multi mode fiber. Hooked everything up and instantly all my problems went away. There must be some kind of compatibility issue between Ubiquity switches and Chelsio. There is talk of vendor lock in with sfp+ stuff but the feeling I got when looking things up was that was a thing of the past, well turns out it is not a thing of the past.
Code:
Jan 15 17:04:45 tubby kernel: cxl0: link state changed to UP
Jan 15 17:04:45 tubby kernel: cxl0: link state changed to UP
Jan 15 17:06:26 tubby cxl1: transceiver unplugged.
Jan 15 17:06:26 tubby kernel: cxl1: link state changed to DOWN
Jan 15 17:06:26 tubby kernel: cxl1: link state changed to DOWN
Jan 15 17:06:31 tubby cxl1: 10Gbps SR transceiver inserted.
Jan 15 17:10:34 tubby kernel: cxl1: link state changed to UP
Jan 15 17:10:34 tubby kernel: cxl1: link state changed to UP

root@tubby:~ # ifconfig lagg0
lagg0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ac00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
    ether 00:07:43:31:44:00
    inet 192.168.1.183 netmask 0xffffff00 broadcast 192.168.1.255
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet autoselect
    status: active
    groups: lagg
    laggproto lacp lagghash l2,l3,l4
    laggport: cxl0 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
    laggport: cxl1 flags=1c<ACTIVE,COLLECTING,DISTRIBUTING>
root@tubby:~ # ifconfig cxl0
cxl0: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ac00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
    ether 00:07:43:31:44:00
    hwaddr 00:07:43:31:44:00
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-SR <full-duplex,rxpause,txpause>
    status: active
root@tubby:~ # ifconfig cxl1
cxl1: flags=8943<UP,BROADCAST,RUNNING,PROMISC,SIMPLEX,MULTICAST> metric 0 mtu 1500
    options=ac00b9<RXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6>
    ether 00:07:43:31:44:00
    hwaddr 00:07:43:31:44:08
    nd6 options=9<PERFORMNUD,IFDISABLED>
    media: Ethernet 10Gbase-SR <full-duplex,rxpause,txpause>
    status: active


I'm sure I have forgotten some stuff so I might update this post. I hope this helps someone in the future or feel free to ask questions because I spent way to much time working through everything and doing debugging.
 
Last edited:

Chris Moore

Hall of Famer
Joined
May 2, 2015
Messages
10,080
Thanks for sharing your experience.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I took the card out and tried it in a linux server with a mikrotik switch and it worked just fine on both interfaces. So I knew the card and direct attach copper cables I was using worked and are not flawed. So I just started searching and reading, it took me about a week but I eventually found this thread:
https://community.ubnt.com/t5/UniFi...nt-2m-DAC-gt-Chelsio-T520-CR-link/m-p/2494470

And it was almost my exact problem. I bought some chelsio transceivers and 10gtek ubiquity transceivers and some multi mode fiber. Hooked everything up and instantly all my problems went away. There must be some kind of compatibility issue between Ubiquity switches and Chelsio. There is talk of vendor lock in with sfp+ stuff but the feeling I got when looking things up was that was a thing of the past, well turns out it is not a thing of the past.


Really? Because if you are looking things up included here on the forum, I specifically call this out in the 10Gig Networking Primer ...

Also available are direct-attach SFP+'s, where two SFP+ modules have been permanently connected together via twinax cable. These are essentially patch cables for SFP+. The downside is that sometimes you run into compatibility issues, especially when you have two SFP+ endpoints from different manufacturers who both engage in vendor lock-in.

I mentioned twinax cables with a sinking feeling in my stomach when I did it, I still even remember the feeling. The highest level of compatibility is to go with some fiber and optics. I try to avoid the little %@$+@&>$ to the maximum extent I possibly can.

The cost on fiber stuff has dropped a good bit in recent years, and there's a number of decent vendors of "compatible" optics as well.

https://extranet.www.sol.net/files/misc/biffiber.jpg

biffiber.jpg

Lately I've been installing OM4 BIF fiber patch, which is capable of up to 100Gbps and is as thin as a strand of dry spaghetti. But the shocking thing is that normal rack-length OM4 patch cables are less than ten bucks each.
 
Joined
Dec 29, 2014
Messages
1,135
Lately I've been installing OM4 BIF fiber patch, which is capable of up to 100Gbps and is as thin as a strand of dry spaghetti. But the shocking thing is that normal rack-length OM4 patch cables are less than ten bucks each.
I am probably repeating myself here, but I have been using fiber instead of DAC/Twinax cables for two reasons. Twinax cables are really hard to manage since they tend to be large and don't bend easily. The more important reason is that you can put transceivers in each side that the card/switch likes, and then there aren't any compatibility issues. I have had all kinds of peculiar issues with Twinax compatibility which is why I almost never use them now.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
Really? Because if you are looking things up included here on the forum, I specifically call this out in the 10Gig Networking Primer ...
So my initial impression about DAC was that it was the most common solution in modern days. I thought fiber was more complicated and more work because you had to buy the special transceivers. Everyone says to just get DAC because it's easy but it turns out both solutions are easy once you figure out all the acronyms. I also didn't think there was any compatability issues anymore, this is kind of true depending on your card and switch combo, since I got my setup to work with DAC and microtik. With ubiquity there is definitely an incomparability.
 
Last edited:

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
So my initial impression about DAC was that it was the most common solution in modern days. I thought fiber was more complicated and more work because you had to buy the special transceivers. Everyone says to just get DAC because it's easy but it turns out both solutions are easy once you figure out all the acronyms. I also didn't think there was any comparability issues any more, this is kind of true depending on your card and switch combo, since I got my setup to work with DAC and microtik. With ubiquity there is definitely an incomparability.

I don't know who "everyone" is but anyone who works with this stuff on a regular basis gets bitten on a semi-regular basis.

"Compatibility issues" are generally inflicted upon us by vendors deliberately, and are not like software bugs that eventually get "fixed" so that it is no longer a problem. The crappy part about it is that it is actually just a cynical ploy to monetize the accessories you need to make your expensive gear work. There is a certain amount of truth that not every SFP+ module will work in every SFP+ slot, because these things really are little microdevices with some logic and firmware complications, so even if vendors didn't restrict compatibility, there'd be some room for problems. But I have a bin full of FTLX8571D3BCV vendor-coded for a bunch of different vendors, and really the only difference here is the vendor code. The SFP+ itself is the same.

You will definitely find combinations of things that work as-expected with unlocked ports. It is also definitely possible to find situations that don't, and twinax cables have an uncomfortably high degree of that happening.

However, as a cautionary tale, I should also mention that I've seen a firmware upgrade on a chassis take the unit from "permissive" using unbranded optics to "locked" requiring vendor optics - and suddenly several ports wouldn't link. Fortunately it was just a few generics that had gotten mixed in, probably when we were out of the correct ones, but it would have been craptacular if I didn't make a general practice of buying the "correct" optics even for apparently unlocked ports. These things are available cheaply on eBay and I rarely see failures. Just buy the optics - the CORRECT optics - and move on with a highly compatible lifestyle.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
And because this is the kind of real-world example that people find compelling, I've just now linked to this thread from the 10Gig Networking Primer as an example of compatibility issues. Hopefully your encounter spares someone else...
 

Gcon

Explorer
Joined
Aug 1, 2015
Messages
59
I have a Ubuquiti Edgeswitch 16 XG (SFP+) connecting to an Intel X520-DA2 (SFP+) via a 3m 10GTek SFP+ DAC cable which is branded "for Ubiquiti". That was working fine - life is good.

Then I decided to test out a Chelsio T520-CR. I thought that would be a nice upgrade going from PCIe 2.0 to 3.0, and Chelsio supposed to have great FreeBSD support. I couldn't help myself, so placed the eBay order and waited.

When the card arrived today I updated the Chelsio Unified Boot BIOS to the latest at the time of writing which is v2.0.0.29, keeping all defaults. Unfortunately the link didn't come up - I'm assuming that the Chelsio doesn't approve of the DAC cable.

I've now just ordered a 3m DAC cable "for Chelsio" and see how that goes. Will take a few weeks to get here. I wouldn't be surprised if the Chelsio likes the new cable, but the Edgeswitch doesn't. Based on the info from OP, I'll probably end up having to go the optical transceiver route rather than DAC. Will see how I go when the new cable arrives.
 

SweetAndLow

Sweet'NASty
Joined
Nov 6, 2013
Messages
6,421
I have a Ubuquiti Edgeswitch 16 XG (SFP+) connecting to an Intel X520-DA2 (SFP+) via a 3m 10GTek SFP+ DAC cable which is branded "for Ubiquiti". That was working fine - life is good.

Then I decided to test out a Chelsio T520-CR. I thought that would be a nice upgrade going from PCIe 2.0 to 3.0, and Chelsio supposed to have great FreeBSD support. I couldn't help myself, so placed the eBay order and waited.

When the card arrived today I updated the Chelsio Unified Boot BIOS to the latest at the time of writing which is v2.0.0.29, keeping all defaults. Unfortunately the link didn't come up - I'm assuming that the Chelsio doesn't approve of the DAC cable.

I've now just ordered a 3m DAC cable "for Chelsio" and see how that goes. Will take a few weeks to get here. I wouldn't be surprised if the Chelsio likes the new cable, but the Edgeswitch doesn't. Based on the info from OP, I'll probably end up having to go the optical transceiver route rather than DAC. Will see how I go when the new cable arrives.
I'm waiting in anticipation for your results! I like going with the brand specific transceivers because it just took any guess work out of it.

Since my original post I have also added a netgear MS510TX switch to the mix and it's connecting from it's 10gig sfp+ port to the sfp+ port on my usg pro using one of the original DAC cables i bought and ended up not using with the chelsio nic in my server. So I kinda think the chelsio is the thing that doesn't like the dac cables. Since you are getting a chelsio DAC cable it might just work.

the dac cables I originally bought where labeled 'Compatible with Cisco, Ubiquiti, Huawei, Netgear, & Supermicro Devices 3m'.
 
Joined
Dec 29, 2014
Messages
1,135
So I kinda think the chelsio is the thing that doesn't like the dac cables.
I can say that I had issues with a Chelsio T520 being quite particular about the SFP/DAC modules. I have abandoned using DAC cables unless they are between two devices from the same manufacturer. I also like that it is easier to router fiber patch cables in cable management than DAC cables.
 

JeffNAS

Dabbler
Joined
Oct 18, 2023
Messages
19
I don't know who "everyone" is but anyone who works with this stuff on a regular basis gets bitten on a semi-regular basis.

"Compatibility issues" are generally inflicted upon us by vendors deliberately, and are not like software bugs that eventually get "fixed" so that it is no longer a problem. The crappy part about it is that it is actually just a cynical ploy to monetize the accessories you need to make your expensive gear work. There is a certain amount of truth that not every SFP+ module will work in every SFP+ slot, because these things really are little microdevices with some logic and firmware complications, so even if vendors didn't restrict compatibility, there'd be some room for problems. But I have a bin full of FTLX8571D3BCV vendor-coded for a bunch of different vendors, and really the only difference here is the vendor code. The SFP+ itself is the same.

You will definitely find combinations of things that work as-expected with unlocked ports. It is also definitely possible to find situations that don't, and twinax cables have an uncomfortably high degree of that happening.

However, as a cautionary tale, I should also mention that I've seen a firmware upgrade on a chassis take the unit from "permissive" using unbranded optics to "locked" requiring vendor optics - and suddenly several ports wouldn't link. Fortunately it was just a few generics that had gotten mixed in, probably when we were out of the correct ones, but it would have been craptacular if I didn't make a general practice of buying the "correct" optics even for apparently unlocked ports. These things are available cheaply on eBay and I rarely see failures. Just buy the optics - the CORRECT optics - and move on with a highly compatible lifestyle.
Does this mean you have to make your own cables if you want to use DAC and have different brand devices on each end? Does doing so solve these issues? PS I am assuming whaenbpoeple use the term twinax that means Dac?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
Does this mean you have to make your own cables if you want to use DAC and have different brand devices on each end?

Most SFP+ modules for DAC/twinax are "generic" and most gear will be more forgiving and tolerate the generic even if it otherwise wants a vendor-locked module. Please note I said MOST. There are some examples, especially where a single vendor provides something like a complex storage system, where DAC cables require vendor coding. It's a bit rare because it'd be stupid for Cisco (for ex.) to vendor lock its DAC cables; makes it tough to hook up a Dell or HP server. In any case, you would not easily be able to make your own cables, as DAC cables typically use twinax and have relatively strict RF requirements. It would be easier to reprogram the SFP+ modules with the appropriate vendor codes.

PS I am assuming whaenbpoeple use the term twinax that means Dac?

No. DAC is "Direct Attach Cable" (formerly "Direct Attach Copper" because most of them use copper). DAC is often implemented over twinax, which is dual coax. However, it can also be implemented over fiber, so twinax and DAC are not interchangeable terms. DAC cables are further subdivided down to passive or active DAC cables, the latter which contains additional electronics to boost signal strength to allow longer transmission lengths. Active DAC cables FURTHER subdivide down to active copper cable and active optical fiber cable, the latter of which is essentially two optical SFP+ modules with a permanently cemented-into-place multimode fiber between them, which I think is relatively stupid since you then are locked in on length.

I personally prefer to just buy the correct optics, either used or new-generic-coded-for-device, and then use fiber like the designers intended. DAC generally sucks because the cabling is bulky. You can get some amazing stuff in the fiber optic market these days, such as

biffiber.jpg
 

JeffNAS

Dabbler
Joined
Oct 18, 2023
Messages
19
Most SFP+ modules for DAC/twinax are "generic" and most gear will be more forgiving and tolerate the generic even if it otherwise wants a vendor-locked module. Please note I said MOST. There are some examples, especially where a single vendor provides something like a complex storage system, where DAC cables require vendor coding. It's a bit rare because it'd be stupid for Cisco (for ex.) to vendor lock its DAC cables; makes it tough to hook up a Dell or HP server. In any case, you would not easily be able to make your own cables, as DAC cables typically use twinax and have relatively strict RF requirements. It would be easier to reprogram the SFP+ modules with the appropriate vendor codes.



No. DAC is "Direct Attach Cable" (formerly "Direct Attach Copper" because most of them use copper). DAC is often implemented over twinax, which is dual coax. However, it can also be implemented over fiber, so twinax and DAC are not interchangeable terms. DAC cables are further subdivided down to passive or active DAC cables, the latter which contains additional electronics to boost signal strength to allow longer transmission lengths. Active DAC cables FURTHER subdivide down to active copper cable and active optical fiber cable, the latter of which is essentially two optical SFP+ modules with a permanently cemented-into-place multimode fiber between them, which I think is relatively stupid since you then are locked in on length.

I personally prefer to just buy the correct optics, either used or new-generic-coded-for-device, and then use fiber like the designers intended. DAC generally sucks because the cabling is bulky. You can get some amazing stuff in the fiber optic market these days, such as

biffiber.jpg
Lots of people seem to have differing opinions on the subject of which spf+ connection type is the most reliable. My question refers to short runs from 1 foot to 15 feet. It would seem that a passive dac twinax or copper cable would be the most reliable as I assume there is no electronics needed in the GBIC? I see on Amazon they even have DACs with GBICs included on the cable (not sure if they are removable). If someone has a link with a long explanation of all this that has more details about DACs and inter vendor compatibility please post it. I've read the 2017 overview here on the forum but need more info. Thanks!
 

JeffNAS

Dabbler
Joined
Oct 18, 2023
Messages
19
Lots of people seem to have differing opinions on the subject of which spf+ connection type is the most reliable. My question refers to short runs from 1 foot to 15 feet. It would seem that a passive dac twinax or copper cable would be the most reliable as I assume there is no electronics needed in the GBIC? I see on Amazon they even have DACs with GBICs included on the cable (not sure if they are removable). If someone has a link with a long explanation of all this that has more details about DACs and inter vendor compatibility please post it. I've read the 2017 overview here on the forum but need more info. Thanks!
One other question is about power consumption of the nic in the host computer. Do any of the cable /gbic options impact nic power consumption and heat?
 

ChrisRJ

Wizard
Joined
Oct 23, 2020
Messages
1,909
SFP+ is a new area for me, but according to what I have read, DAC cables have a disadvantage relative to fiber optics when it comes to latency as well as power consumption. Whether or not this is relevant for you, is a separate matter.

As to locks on SFP+ modules, I recently purchased 2 used X520-DA2 cards that came with 2 SFP+ transceivers each. The transceivers were a mix of IBM and HP and TrueNAS as well as XCP-ng complained about them being unsupported. On both systems, there are boot parameters for the driver to disable the check and this is how got them working.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
I assume there is no electronics needed in the GBIC?

Ah, wha...?

No one uses GBIC anymore. GigaBit Interface Converters were a '90's thing and went out of style by the mid-2000's as the much better SFP technology allowed much better density. SFP and GBIC serve basically the same purpose. And yes there is substantial electronics in the GBIC, which is why they ran hot and tended to burn out sometimes. They convert the GBIC electrical interface to some sort of optical interface, usually SC or LC fiber. Improvements in electronic and optical technology meant that SFP's were much smaller and lower power.

Then along came 10G ethernet and SFP+'s, making all that old crap mostly irrelevant except to those of us that had some sort of investment in the older technology. And now we've moved on to SFP28.

SFP* generally has a power budget of 1.5W per module (per spec) and many common fiber modules use 0.8W or less. This can be a problem in 48 or 96 port switch blades, so specific ratings are usually available in the spec sheet for a given module.

Even though these are often referred to as "passive" cables, they are referring to the signal handling -- the coax is soldered directly to the PCB with maybe a few resistors, so the signal path is effectively passive. However, there should also be an I2C bus EEPROM in there that holds the configuration information about what sort of SFP+ this is. This is an extremely low power component and is negligible from that perspective. It is this EEPROM which identifies the SFP+ to the host device as a passive cable, and which might hold a vendor identifier as I previously discussed. This means that passive cables consume almost no power on their own. Active cables, of course, have amplifiers and filtering so they do consume some power, but probably very similar to what optical SFP+'s would use.

Lots of people seem to have differing opinions on the subject of which spf+ connection type is the most reliable.

Sure. You know what they say, opinions are like ... well you know. Reliability can be measured in a variety of ways. Is passive DAC more reliable? Sure, maybe, until one day you push in a server and the cable gets caught up in the rails and is damaged because it is big, fat, and inflexible. Then you may have to order a replacement and it can take a week to get there. I score that low because of the downtime. We generally populate every incoming device with the correct vendor coded optics when procurement hands them to engineering, but that's just because we have a low tolerance for downtime as a network operator. I generally run that BIF fiber shown above inside racks for interconnect, but on the shop benches where we build servers and stuff with some risk of something getting inadvertently sat on a fiber, I've been favoring FS's armored cable product for its relative indestructability. Fiber with SFP+ optics means that if an SFP+ fails, you just replace that one part, you don't have to pull a fat SFP+ plug through your neatly groomed fiber cables. And unlike DAC, you don't need to stock multiple lengths of cables you might not need, just multiple lengths of fiber. That carries another kind of reliability with it.

If someone has a link with a long explanation of all this that has more details about DACs and inter vendor compatibility please post it. I've read the 2017 overview here on the forum but need more info. Thanks!

I don't know what 2017 overview you're referring to. I don't know that you're going to find a better source of discussion on the topic than here on these forums. The vendors don't like talking about it because their attitude is that they just want you to buy new stuff from them, and have little incentive in explaining to you how to do networking on the cheap or what their vendor lock policies are. I've been doing this a hella long time (still have 10base5 cabling supplies) and have a tendency to be willing to talk endlessly about stuff I happen to know; most of this discussion we're having is above the heads of 90% of the 10G+ users on these forums.

Do any of the cable /gbic options impact nic power consumption and heat?

Yes, ... well I'd say "obviously" but then I shouldn't assume people took electrical engineering. :smile: 10GBASE-T is by far the most awful, often taking well beyond 1.5 watts per port (it's a horrible hack job, maybe 2.5W). Some of the long distance laser products come in next. Then comes 10GBASE-LR (shorter distance 220m-10km) and 10GBASE-SR (up to 220m), which are similar to active optical, and then DAC which are mostly passive and probably like 0.05 watts or something silly like that.

Also I guess I should point out that your question doesn't ask what I think you think it does. Pedantically, network card power consumption isn't affected by the modules in the SFP+ slot, because you wouldn't account something that you plug into the SFP+ slot to the network card. I mean, you don't attribute your server's power consumption to the network card either, right, even though your server is connected to the network card? :smile: But I am guessing you are trying to figure out the lowest power option for your overall networking solution.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,681
a DAC as it's not doing much

Think I said that several times. Was I not clear or did you just not believe me?

states that power dray from the power supply in the server/switch is impacted much less

But if you have a power supply that is being "impacted" by the draw of SFP+ modules, you need a better class of server or switch. This should be a nonissue. Even a large switch with hundreds of SFP+ should only be some hundreds of watts additional consumption for the SFP+ modules, and in such a configuration, you are probably not going to be able to have hundreds of endpoints within the distance limitations of DAC cabling. At some point you gotta put on the big boy pants and do it right.

Okay well thinking about it a bit I suppose if you had three 60U racks and you placed a high density switch in the middle of the middle rack, you would have plausible reachability to about 160U of usable rack space. If you then stuck a dual 10G card in a 1U server in each rack U, I guess you MIGHT be able to cable 320 DAC cables. You could potentially argue for additional racks on each side seeing as how the DAC cable length is about 7m, but I think you'd find that you'd quickly go insane on the cabling density. That's one of the reasons that ToR switches became so popular.
 
Top