Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
10 Gig Networking Primer

10 Gig Networking Primer

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,069
Just received 2x of my Solarflare SFN5122F to replace my old 10G card Chelsio cc2-s320e-sron SFP+ on Supermicro X10SL7-F ( TrueNAS Core 12 Beta) and X9SRL-F ( FreeNAS 11.3-U4). I can confirm they all work well without any loading drivers or anything just plug and play. Solarflare gives you almost full speed while Chelsio gets you half the speed.
Yeah, uh, just to clarify this, that's what happens when you use a really old janky Chelsio PCIe 1.1 card from 2006, and replace it with a much newer PCIe 2.0 card from 2010.

I do ask that people do not make generalizations if they're going to post in my thread here.

The highly recommended Chelsio T520-CR is a PCIe 3.0 card and will beat on the 5122. :smile:

The PCIe 3.0 SFN6122F (also PCIe 3.0) has been available on eBay for $20-$40 for at least a few years. I haven't tried it on FreeNAS as bare metal but don't have any reason to think it wouldn't work. I use these under ESXi and they're great, they work as well as the X520 (PCIe 2.0) and they generate less heat.
 

Sasquatch

Member
Joined
Nov 11, 2017
Messages
35
@jgreco great article, THANK YOU for stopping me from making mistake and buying into 10Gbase-t


I'm a total fibre noob so forgive noob-ish questions
I have only one PCI-e 2.0 x4(mechanical x16) available in my freenas box. In theory it is more than enough to saturate one 10Gb link on a card like sfn5122/t420/x520, how does it translate to practice?

I have 2 corners to go around and cables sticking out are not acceptable(females, heh), is this fiber really so bendy? Can i get away with 2x 7.5mm radius bends?
Or should I break out sds drill and cut come corners, in a good way for once :wink:.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,069
Yeah, well, 10Gbase-T is a bit more viable than it used to be, but it is still basically pretty expensive.

The Intel X520 and Chelsio T5x0-CR cards are known to be the best-supported cards. I've been mentioning the Solarflare SFN6122, big brother of the 5122, as I've got a bunch of these on ESXi hosts, working as well as or better than the X520 IMO. Reports from FreeNAS forum members are that they work fine on FreeNAS, but seeing as how I am generally reluctant to endorse things I haven't done, well, that's about as far as I'll commit to. The 5122/6122 have been showing up on eBay at incredible $20-$40 prices though, so it's probably not catastrophic to take a chance. The 5122 is PCIe 2.0 while the 6122 is PCIe 3.0, if the price difference isn't much, then I prefer the 3.0 even if your current board doesn't run that fast. Actual performance is a matter of so many factors that I generally don't try to make predictions, I just buy the best stuff I can and see what happens.

Definitely buy bend-insensitive fiber if you're a fiber n00b. Take a look at the initial post in this thread at the BIF OM4, it is duplex-fiber-in-single-jacket which makes for somewhat better options for hiding it. That comes at the cost of being somewhat more delicate, of course.

 

Sasquatch

Member
Joined
Nov 11, 2017
Messages
35
Somehow i missed OM4 being single jacket it costs just 10% more than OM3, would be silly to not use it.
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
894
Just be sure to buy the right transceiver for a given cable type. While fiber offers better throughput, longer distances, lower-power / heat, electrical isolation, and so on than copper, one does have to pay attention to transceiver and cable type.

Additionally, thanks to rent-seeking by some vendors, you also have to ensure that whatever SFP+ port you're plugging into is compatible with your transceiver. Thankfully, once the stuff is in the fiber, all is well, but the transceivers on either end may have to have the same vendor lock ID as the equipment they're plugged into. So, buy a Mikrotik compatible transceiver on one end and a Cisco on the other, etc. depending on what you're using.

As @jgreco pointed out, there is no shortage of fast, reputable OEM stuff available for great prices on eBay. FWIW, I thought the drivers for Chelsio on the Mac were absolute stinkers, cannot recommend. Myricom works great. Undoubtedly, there are other "great" brands to consider.
 

jgreco

Resident Grinch
Moderator
Joined
May 29, 2011
Messages
13,069
No, it's not OM4 that's single jacket, it's just this particular product. Fiber is generally simplex-fiber-per-jacket because this makes it easy to terminate, especially in the field. The duplex-fiber-double-jacket stuff is somewhat more difficult to work with, having worked with it for some decades gives me some perspective on that.

This from APNIC is interesting.

When you have a double jacket, the best case is that you have a vertical LC or SC connector, because most cable management is horizontal, and this tends to give you equal jacket. But you can still bend and break a fiber if you're not careful, as outlined above.

It's nice for fiber to all line up neatly like this.

But when you're cabling to a switch, the ports and plugs are usually horizontal, and this leads to a sucky situation where the dual jacket stuff has to rely on the split in the cable. I found a great but probably copyrighted image at thumbs.dreamstime.com/z/cable-network-fiber-optic-cable-connect-to-switch-port-server-room-concept-network-management-cable-network-fiber-optic-cable-162833048.jpg that clearly shows this.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,228
...snip...
I've been mentioning the Solarflare SFN6122, big brother of the 5122, as I've got a bunch of these on ESXi hosts, working as well as or better than the X520 IMO. Reports from FreeNAS forum members are that they work fine on FreeNAS, but seeing as how I am generally reluctant to endorse things I haven't done, well, that's about as far as I'll commit to. The 5122/6122 have been showing up on eBay at incredible $20-$40 prices though, so it's probably not catastrophic to take a chance. The 5122 is PCIe 2.0 while the 6122 is PCIe 3.0, if the price difference isn't much, then I prefer the 3.0 even if your current board doesn't run that fast. Actual performance is a matter of so many factors that I generally don't try to make predictions, I just buy the best stuff I can and see what happens.
I've accumulated 5 of the SolarFlare SFN6122F NICs at a net cost $21.90 each. Gotta love fleaBay! I've installed them in two of my ESXi 6.x servers, where they work fine (as you noted above).

Curious about how well they'd work with FreeNAS running on-the-metal, this afternoon I installed FreeNAS 11.2-U8 onto a flash drive in my somewhat elderly X8SIE-based system ('BACON'). I can report that they work right out-of-the-box with no intercessions or hand-waving necessary.

ifconfig:
Code:
root@bacon:~ # ifconfig -vm sfxge0
sfxge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: connected to 9 peers
        options=ec07ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=ec07ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0f:53:0c:05:b0
        hwaddr 00:0f:53:0c:05:b0
        inet 172.16.10.12 netmask 0xffffff00 broadcast 172.16.10.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-SR <full-duplex,rxpause,txpause>
        status: active
        supported media:
                media 10Gbase-SR mediaopt full-duplex,rxpause,txpause
                media 10Gbase-SR mediaopt full-duplex,rxpause
                media 10Gbase-SR mediaopt full-duplex
                media 1000baseSX mediaopt full-duplex,rxpause,txpause
                media 1000baseSX mediaopt full-duplex,rxpause
                media 1000baseSX mediaopt full-duplex
        plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
        vendor: Arista Networks PN: CAB-Q-S-2M SN: XHC1314LG06P DATE: 2013-04-02
root@bacon:~ # ifconfig -vm sfxge1
sfxge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 9000
        description: connected to brocade-6610-u1
        options=ec07ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        capabilities=ec07ab<RXCSUM,TXCSUM,VLAN_MTU,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWTSO,LINKSTATE,RXCSUM_IPV6,TXCSUM_IPV6>
        ether 00:0f:53:0c:05:b1
        hwaddr 00:0f:53:0c:05:b1
        inet 10.0.58.12 netmask 0xffffff00 broadcast 10.0.58.255
        nd6 options=9<PERFORMNUD,IFDISABLED>
        media: Ethernet 10Gbase-SR <full-duplex,rxpause,txpause>
        status: active
        supported media:
                media 10Gbase-SR mediaopt full-duplex,rxpause,txpause
                media 10Gbase-SR mediaopt full-duplex,rxpause
                media 10Gbase-SR mediaopt full-duplex
                media 1000baseSX mediaopt full-duplex,rxpause,txpause
                media 1000baseSX mediaopt full-duplex,rxpause
                media 1000baseSX mediaopt full-duplex
        plugged: SFP/SFP+/SFP28 1X Copper Passive (Copper pigtail)
        vendor: Arista Networks PN: CAB-Q-S-2M SN: XPV131830184 DATE: 2013-05-03
root@bacon:~ #

iperf:
Code:
root@bacon:~ # iperf -c bandit -t30 -i5
------------------------------------------------------------
Client connecting to bandit, TCP port 5001
TCP window size: 4.01 MByte (default)
------------------------------------------------------------
[  3] local 172.16.10.12 port 22973 connected with 172.16.10.10 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0- 5.0 sec  5.46 GBytes  9.38 Gbits/sec
[  3]  5.0-10.0 sec  5.75 GBytes  9.87 Gbits/sec
[  3] 10.0-15.0 sec  5.75 GBytes  9.87 Gbits/sec
[  3] 15.0-20.0 sec  5.75 GBytes  9.87 Gbits/sec
[  3] 20.0-25.0 sec  5.75 GBytes  9.87 Gbits/sec
[  3] 25.0-30.0 sec  5.75 GBytes  9.87 Gbits/sec
[  3]  0.0-30.0 sec  34.2 GBytes  9.79 Gbits/sec
root@bacon:~ #

pciconf:
Code:
root@bacon:~ # pciconf -lv | grep -A1 -B3 network
sfxge0@pci0:1:0:0:      class=0x020000 card=0x62271924 chip=0x08031924 rev=0x00 hdr=0x00
    vendor     = 'Solarflare Communications'
    device     = 'SFC9020 10G Ethernet Controller'
    class      = network
    subclass   = ethernet
sfxge1@pci0:1:0:1:      class=0x020000 card=0x62271924 chip=0x08031924 rev=0x00 hdr=0x00
    vendor     = 'Solarflare Communications'
    device     = 'SFC9020 10G Ethernet Controller'
    class      = network
    subclass   = ethernet
--
 

Sasquatch

Member
Joined
Nov 11, 2017
Messages
35
I've accumulated 5 of the SolarFlare SFN6122F NICs at a net cost $21.90 each. Gotta love fleaBay! I've installed them in two of my ESXi 6.x servers, where they work fine (as you noted above).

Curious about how well they'd work with FreeNAS running on-the-metal, this afternoon I installed FreeNAS 11.2-U8 onto a flash drive in my somewhat elderly X8SIE-based system ('BACON'). I can report that they work right out-of-the-box with no intercessions or hand-waving necessary.
Thanks for the report, will sleep better until my cards arrive.
 

Spearfoot

He of the long foot
Moderator
Joined
May 13, 2015
Messages
2,228

ianrm

Junior Member
Joined
Aug 22, 2020
Messages
21
Hi,
first post as with beginners usually a question.
I have installed my freenas and after a day of stuffing around I have it working. A bit confused about IPMI how ist appears on one the standard interface. My question is how do I install my 10GB ASUS XG-C100C network adapter? It is connected and talks to the 10gb port on the switch.

My system is
FreeNAS 11.3.U4-1
SuperMicro X10SLM-F CPU E3-2140v3
Memory 16gb
6 disks a mix of SSSD and spinning.

Ian
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
894
You plug it in, restart the machine, and then it should show up as one of the network interfaces in the networking section. For the Chelsio and the built-in SFP+ ports I have dealt with, the 10GbE ports were labeled "ix" (followed by a number) whereas gigabit ports (all built in) were labeled "ig" followed by a number. However, this may vary by board.

If you have a plethora of ports and you don't deal with this stuff professionally and hence know by heart which port is which, I would consider testing and labeling each one individually. That way, you will know immediately which port is on strike or whatever when your NAS is in trouble.

IIRC, the availability of the IPMI interface on non-dedicated ports is set in the UEFI BIOS for X10 and X11-series supermicro boards - see the relevant manual here. For greater security, I would ensure that the "shared lan selection" is off (see page 2-7), i.e. that the IPMI interface availability is limited to the dedicated IPMI ethernet port.

Given the many known security issues with IPMI, you don't want that port to be exposed to anyone on your LAN. In more professional settings, the IPMI interface is commonly connected to either a separate VLAN or air gapped network. I simply leave my IPMI network cable unplugged unless I need the IPMI, which is just about never in a home setting.
 

ianrm

Junior Member
Joined
Aug 22, 2020
Messages
21
[Hi Constantin,
thank you for the detailed explanation. Your answer is what I was expecting should happen. The motherboard appears to be faulty more than I thought, originally this was a working VM with 24gb of ram, now it will only work with 16gb in the first slots.
I have tried to perform a BIOS upgrade through the IPMI, indicated it was successful, but deeper investigation shows no change.
Looks like a lot more experimenting and a new MB etc is required.
Ian
 

ianrm

Junior Member
Joined
Aug 22, 2020
Messages
21
Well, back to the drawing board, the MB has basically died, now cannot find the memory. I have checked the equivalent dollar value commercial replacements and will spend the money on buying new hardware and building a new FreeNAS. When I get that far I will also need to understand how to create an NTP server in the FreeNAS if that is possible.

Ian
 

Constantin

Vampire Pig
Joined
May 19, 2017
Messages
894
I had good luck with my embedded motherboard and recommend it highly for use cases similar to mine (no transcoding, storage-oriented, etc). The less-expensive, 2-core, 4-thread version using a D-1508 (X10SDV-2C-7TP4F) is likely even better for a SOHO setting due to the higher clock speed (SMB & AFP are single-threaded).

I did find its acceptance of memory modules to be incredibly picky, however. Even something as remote as die revision of the RAM chips could mean the difference between all 128GB RAM being recognized vs. not. So you are not alone!
 

ianrm

Junior Member
Joined
Aug 22, 2020
Messages
21
Hi folks,
decided to try a different approach,, I recovered and intel server MB, installed the ram, 4 disks and the 10gb card.
Installed freeNAS 11.3, upgrade to 11.3.u4.1. Interestingly with an older CPU this installation runs much smoother.
I am back where I started though with the original post, how to get the 10GB card recognised? It shows in the BIOS but not in FreeNAS.

New system
Motherboard: Intel S1200KP
CPU: e3-1245v2
Memory: 16gb (2 by 8gb) (not Kingston)
Disks: system 256gb SSD Intel, 900gb slow shinning and two 750gb slow spinning.
NICs: 2 by Intel that show up as EM
 

Yorick

Dedicated Sage
Joined
Nov 4, 2018
Messages
1,700

ianrm

Junior Member
Joined
Aug 22, 2020
Messages
21
Hi York,
I checked the supplied CD and it has a linux folder and build instructions for Debian and Red-hot, but no actual drivers. I am not that confident of being able to build a driver within the FreeNAS environment. I suppose try and learn will apply.
Ian
 

ianrm

Junior Member
Joined
Aug 22, 2020
Messages
21
Hi,
found the answer in another thread with details about the driver installation.
Ian
 

Eric Schrauth

Member
Joined
Jun 26, 2017
Messages
37
... they work as well as the X520 (PCIe 2.0) and they generate less heat.
Was wondering, I have a Asus P8B WS motherboard and the only PCIe 2.0 slots that are available are two PCIe 2.0 x16 (@ x4). Will those be fast enough for a card in each of them?
 

Eric Schrauth

Member
Joined
Jun 26, 2017
Messages
37
I asked MikroTik to send me its 8 port big brother and I'm incredibly impressed at the value. It's absolutely not the enterprise grade five-figure switchgear I normally use, but it's definitely a big step up compared to the normal Netgear etc. stuff, and, holy mackerel, the price. Can you go wrong? I think you can't.
Were you ever able to get your hands on one of these? Didn't know if you had any further input/experience with these.
 
Top