10GB NIC direct connected can't ping.

[Brendonb]

Hello,
I'm having a strange issue. I'm using an HP Microserver Gen8 with an HP NC550SFP Dual 10GB Card (made by Emulex). The card is correctly identified and shows alive. Because I'd rather not buy an $800 10GB switch for 4 servers, I'm directly connecting this NC550 to the identical card in my 2 ESX hosts; dual port card so each port goes direct to one of the 2 hosts. Only one of the ESX hosts is online right now to simplify testing. I've configured the network as follows:
FreeNAS oce0 = 10.0.12.9 /30 (255.255.255.252 = allows 2 usable IPs)
ESX = 10.0.12.10 /30

Again, I see link and alive on both sides, but I can not ping the other side from either machine. This exact same configuration works perfectly well on another storage server running Solaris.

Other stats:
FreeNAS version: FreeNAS 9.2.1.8-Release 64-bit
System = HP Microserver Gen8 w/ Xeon E3-1220v2 & 16GB RAM.

Any thoughts? Let me know if there is any other info or output needed.

Below is the ifconfig from FreeNAS (oce0 is the 10GB in question).

Code:

[root@kalimdor ~]# ifconfig
oce0: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
ether 3c:4a:92:da:1a:18
inet 10.0.12.9 netmask 0xfffffffc broadcast 10.0.12.11
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (10Gbase-SR <full-duplex>)
status: active
oce1: flags=8043<UP,BROADCAST,RUNNING,MULTICAST> metric 0 mtu 1500
options=507bb<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,JUMBO_MTU,VLAN_HWCSUM,TSO4,TSO6,LRO,VLAN_HWFILTER,VLAN_HWTSO>
ether 3c:4a:92:da:1a:1c
inet 10.0.12.13 netmask 0xfffffffc broadcast 10.0.12.15
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect
status: no carrier
bge0: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether a0:48:1c:b8:b1:50
inet 10.0.0.105 netmask 0xffffff00 broadcast 10.0.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex,master>)
status: active
bge1: flags=8843<UP,BROADCAST,RUNNING,SIMPLEX,MULTICAST> metric 0 mtu 1500
options=c019b<RXCSUM,TXCSUM,VLAN_MTU,VLAN_HWTAGGING,VLAN_HWCSUM,TSO4,VLAN_HWTSO,LINKSTATE>
ether a0:48:1c:b8:b1:51
inet 10.0.0.228 netmask 0xffffff00 broadcast 10.0.0.255
nd6 options=9<PERFORMNUD,IFDISABLED>
media: Ethernet autoselect (1000baseT <full-duplex>)
status: active
ipfw0: flags=8800<SIMPLEX,MULTICAST> metric 0 mtu 65536
nd6 options=9<PERFORMNUD,IFDISABLED>
lo0: flags=8049<UP,LOOPBACK,RUNNING,MULTICAST> metric 0 mtu 16384
options=600003<RXCSUM,TXCSUM,RXCSUM_IPV6,TXCSUM_IPV6>
inet6 ::1 prefixlen 128
inet6 fe80::1%lo0 prefixlen 64 scopeid 0x9
inet 127.0.0.1 netmask 0xff000000
nd6 options=21<PERFORMNUD,AUTO_LINKLOCAL>

 

[cyberjock]

Hate to break it to you, but those Emulex cards have been a mixed bag. Yours may not be compatible with FreeBSD.

Unfortunately, those outrageously expensive NICs you don't want to buy are the only ones I recommend at present. Even the previously "tried and true.. and relatively cheap" Intel 10Gb NICs are intermittent right now due to a driver bug.

Unless an Emulex user has first-hand knowledge of how to fix your problem (assuming there is a fix) you are probably "on your own". :( One of the drawbacks about going with your own stuff is nobody has walked the path so you are trailblazing it.

 

[Brendonb]

Thanks for the reply. I've seen in your other posts that Chelsio is the preferred vendor for Freenas 10GB (never heard of Chelsio before). I had hoped that because of the driver seeing it, etc that it is working and just some stupid thing I was missing (not a BSD expert), but perhaps not. It really seems that it is just a problem with 'auto-mdix' but if its a driver problem...

Does no one else use 10GB with freenas with non-chelsio cards?

 

[mjws00]

Haven't seen much talk of Emulex beyond "I hope the drivers come soon." The Intel 10Gbe cards were starting to make headway until we hit a driver issue with them. So the fall back position is Chelsio which iX relies on and is known good, these have the highest chance of a driver tweak and or bug fix. As the other options make headway in FreeBSD hopefully we'll have more 10GB users. FreeNAS also lags behind FreeBSD in some areas significantly. So your card might be perfect under 10.0... we just aren't there yet. Limited resources yada yada... :)

If I knew of a known good solution I'd use FreeNAS all day long with a non-chelsio card. I've even held off the Intel x540's for now. But not forever. ;)

Sorry, rarely are the suggestions based on brand zealotry. Truth is there are few highly proven options, and most of us aren't comfortable recommending solutions with known problems.

 

[Brendonb]

I assumed the response wasn't around bias or anything. :)

I was actually just digging on updating the emulex driver directly, but I'm running a bit thin on knowledge. Emulex's manual says "pkg_info | grep -i 'oce driver'", but I get a response of no packages installed...so now I'm stuck. Any thoughts on trying to determine the current driver version and whether a current FreeBSD v9.1 driver direct from emulex might help?

As always thank you for the input.

 

[mjws00]

Unfortunately adding a driver to FreeNAS requires you to compile the drivers against the kernel. That is the nature of how they have designed it as an appliance. There is a decent learning curve if you aren't doing it frequently. I've never bothered with bsd, sorry.

What I would try is taking the 9.3 Alpha and seeing if it works. I'd try a livecd of freebsd 10.0 just to see if the card is already addressed and working perfectly. At that point at least you have an idea of future support. If it works great in a version that is near term... it might be worth adding that driver, or just choose to wait it out.

Hard to say what is the best choice for you. My gut says if you sent me the cards, and Emulex has a bsd driver. I'd go through the trouble to add it and test. But I've compiled a few kernels in my day... even if I'm rusty ;)

 

[Brendonb]

Hello. In trying to take some advice and use Chelsio adapters, I found a good price on the Chelsio S320E-LP-CR. Can someone confirm if these are supported/work correctly in FreeNAS? I see cyberjock refers to them often in posts, but I want to make sure this model is good before I purchase.

Thank you as always.

 

[cyberjock]

Per the hardware compatibility matrix for FreeBSD it appears that only the following is supported:

[i386,amd64] The cxgb(4) driver supports 10 Gigabit and 1 Gigabit Ethernet adapters based on the T3 and T3B chipset:

• Chelsio 10GBase-CX4
  
  
• Chelsio 10GBase-LR
  
  
• Chelsio 10GBase-SR

A developer at iX said that it doesn't appear to have been tested on FreeNAS.

So I wouldn't buy one unless you are okay with potentially not being able to use it.

 

[jgreco]

The first numeric in the Chelsio part number indicates the revision of the Terminator ASIC on a board. A part number beginning with "S3" is therefore a Terminator 3 based card. This is not suggesting that the card is guaranteed to work, but it would appear to qualify as potentially supported under the cxgb(4) driver. As usual some research is a good idea.

 

[cyberjock]

I'd also like to add to this thread that the Intel 10Gb LAN cards have been validated to work properly in 9.2.1.8. There are currently no known outstanding issues with Intel 10Gb cards, so if they don't work or aren't consistently working then bug tickets need to be filed as appropriate.

 

[Brendonb]

Nice new avatar cyberjock!

I actually went ahead and bought the s320 based on my own small amount of research figuring that it is a pretty safe bet. I will be installing this weekend and rebuilding with freenas. Server is currently installed with illumos variant, but I want the gui and functionality that freenas brings. I need stable storage that works and a reasonable ability to troubleshoot. My main concern is about the smart reporting on the drives as that is obviously the highest concern for failure.

I will report back about the s320 after my testing.

Oh! The emulex did seem to work in fn 10 alpha, but I didn't get very good perf out of it so I moved on.

 

[aufalien]

I'd suggest SolarFlare. They OEM to HP and are used in high transaction; ie; stock exchanges were low latency is key. I only use them, better price and performance than Intel etc.. That doesn't solve your particular issue with Emulex but it gives you options. I've been using the Server Performant adapters. They have some high dollar ones that seem over kill.

 

[cyberjock]

Better price? What model works? I'm interested in buying one from ebay or something so I can test it out. ;)

 

[aufalien]

Hi CJ,

I use the SNF5162F duel port in servers and SNF5152F single port in various stations. Both there Onload and Performant models share the same driver so you're all good in FreeBSD/FreeNAS.

However there Flareon cards look wicked, unsure about FreeBSD support but both FreeBSD and FreeNAS communities were very helpful in getting my Performant adapters working properly.

http://www.solarflare.com/Product-Overview

 

[Brendonb]

UPDATES:
So I purchased the Chelsio S320E-LP-CR as I stated before. I ran a slew of VMWare IO analyzer perf tests with the server running Nexenta & using the HP/Emulex 10GB card and recorded those perf #s.
The datastore is mounted into esx using NFS.

Now I've reinstalled with FN9.2.1.8 and swapped the Emulex for the Chelsio card. I set up all the options the same as before including mtu 9000 (though I was just reading about -tso on this cards?) and thankfully, this one actually passes traffic and can ping.

So I did a similar perf test and where before I was getting 600 IOPS (this is on my 5400rpm laptop drives), now I'm only getting 200. I ran the exact same test using first the 10GB card and then again with an intel 1GB card and the network cards show roughly the same bandwidth output!?

I then did a dd on the same disk (its a 4disk RAIDz) and here's the output. By my math, that's about 2.5GB/s...so the disks aren't the problem. The cpu never went above 23% during DD.

Question: Why is the 10GB card passing the same amount of bandwidth as the 1GB card? Thoughts?

Thank you as always.

 

[jgreco]

Question: Why is the 10GB card passing the same amount of bandwidth as the 1GB card? Thoughts?

Because your system isn't capable of passing more than a few dozen megabits as it is configured. I would have thought that to be obvious. You won't see a speed bump from an upgrade to 100Gbit ethernet either.

One of the hard things to wrap your head around in this business is that you don't get the fastest possible speeds that are possible out of each individual component. There's a saying about chains being as strong as their weakest link, and this translates to complexity in storage systems, but for storage systems it also turns out that the more links you have, that is also a weakness.

We try to strengthen the individual links in the chain by pumping up and optimizing each individual subsystem, but overall adding complexity tends to counteract that.

At the speeds you are probably hoping to achieve, latency and overhead are critical factors. Back-to-back SFP+ based 10GbE is a good way to optimize the network hardware layer (10GbT adds latency) but basically you've brought a very fat firehose up to a yard hydrant and you're wondering why more water isn't coming out of the hose. A MicroServer with E3-1220v2 and 16GB of RAM is several times smaller than what you'd need to be able to really push past 1Gbps. The 1220v2 has 4 cores without hyperthreading and a 3.1GHz-3.5GHz speed so it probably has the CPU oomph to get in the ballpark of where that needs to be, but the 16GB is going to be very limiting with a VM workload, and there's no indication what your I/O subsystem looks like.

Now, NFS is always problematic with ESXi, so my *guess* is that you'd get at least a tenfold increase in speeds by disabling sync, which effectively short-circuits all the hard work your storage system is dumping into guaranteeing that your VM data is safely and correctly stowed on disk. It "shortens the chain" and counteracts the lack of a SLOG device by telling ZFS not to care about guaranteeing the VM data is written to disk in a timely fashion. If that works for you, then we've essentially identified that your MicroServer is unsuitable for the task, but I think I kinda guessed that before the discussion started.

For VM storage, you probably want something capable of at least 64GB RAM and a bunch of fast disks.

 

[Brendonb]

Ah. I see where the disconnect is because I started this discussion off talking about the Microserver which was meant to be a test bed before moving to my main storage. I've moved forward and purchased the Chelsio card and installed that to my main storage server which is as follows:

• HP DL380 G6 (dual Xeon CPU w/ 130GB RAM)
  
• 8 x 300GB 10K drives (raid10)
• 12 x 1TB SAS drives (raid50) w/ 1 x 128GB Samsung 840 Pro slog
• 4 x 500GB 5400RPM 2.5" (raidz)

I apologize for leaving that significant detail out. I've since done MORE testing and ISCSI is pushing far more IOPS even over 1GB than NFS over the same line. This is using a 150GB test file so it is outside ARC.

• NFS = 200 IOPS
• ISCSI = 1100 IOPS

So it appears the slowdown is NFS vs ISCSI so far. Does that make sense? Any advice or tuning to be done? I'm going to test 10GB iscsi to see if that is equivalently faster.

 

[jgreco]

So it appears the slowdown is NFS vs ISCSI so far. Does that make sense? Any advice or tuning to be done? I'm going to test 10GB iscsi to see if that is equivalently faster.

I think someone wrote a sticky about this.

https://forums.freenas.org/index.ph...xi-nfs-so-slow-and-why-is-iscsi-faster.12506/

 

[cyberjock]

Silly rabbit.. you linked to your own sticky!

 

[Mlovelace]

The chelsio T4 and T5 ASICs are fully supported by the current version of freenas.
"Updated by Josh Paetzel 6 months ago
FreeNAS 9.2.1.5 supports the Chelsio T3 with the cxgb driver. It supports the T4/T5 with the cxgbe driver. We keep these drivers current with the latest code from the vendor. By far and away the chelsio cards are the best FreeBSD option for 10Gbe."

Here's a link to the bug report: https://bugs.freenas.org/issues/4560#change-23492

Code:

[root@clnas]~# pciconf –l –v
t5nex0@pci0:2:0:4:      class=0x020000 card=0x00001425 chip=0x54011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    class      = network
    subclass   = ethernet
none18@pci0:2:0:5:      class=0x010000 card=0x00001425 chip=0x55011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    class      = mass storage
    subclass   = SCSI
none19@pci0:2:0:6:      class=0x0c0400 card=0x00001425 chip=0x56011425 rev=0x00 hdr=0x00
    vendor     = 'Chelsio Communications Inc'
    class      = serial bus
    subclass   = Fibre Channel 

Code:

[root@clnas] ~# sysctl dev.t5nex.0.%desc
dev.t5nex.0.%desc: Chelsio T520-CR RNIC (rev 0), S/N:XXXXXXXXXXXX, E/C:0000000000000000

Code:

[root@clnas] ~# dmesg
t5nex0: <Chelsio T520-CR> mem 0xf8300000-0xf837ffff,0xf7000000-0xf7ffffff,0xf8b04000-0xf8b05fff irq 32 at device 0.4 on pci2
t5nex0: PCIe x8, 2 ports, 22 MSI-X interrupts, 55 eq, 21 iq
cxl0: <port 0> on t5nex0
cxl0: Ethernet address: 00:07:43:2c:72:e0
cxl0: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)
cxl1: <port 1> on t5nex0
cxl1: Ethernet address: 00:07:43:2c:72:e8
cxl1: 8 txq, 8 rxq (NIC); 8 txq, 2 rxq (TOE)