Chelsio T310 10Gb card seems to be flapping

[David Dyer-Bennet]

I'm getting over 1Gb throughput, so I'm pretty sure I've actually got the card configured.

(Just two of the cards, connected with a Twinax cable; both sides are running FreeNAS 9.10-Stable.)

But I'm getting this flapping going on:

Code:

Apr 29 08:14:31 fsfs devd: Executing '/etc/rc.d/dhclient quietstart cxgb0'
Apr 29 03:14:31 fsfs kernel: cxgb0: link state changed to UP
Apr 29 03:14:31 fsfs kernel: cxgb0: link state changed to UP
Apr 29 03:14:34 fsfs kernel: cxgb0: link state changed to DOWN
Apr 29 03:14:34 fsfs kernel: cxgb0: link state changed to DOWN
Apr 29 08:14:35 fsfs devd: Executing '/etc/rc.d/dhclient quietstart cxgb0'
Apr 29 03:14:35 fsfs kernel: cxgb0: link state changed to UP
Apr 29 03:14:35 fsfs kernel: cxgb0: link state changed to UP
Apr 29 03:14:36 fsfs kernel: cxgb0: link state changed to DOWN
Apr 29 03:14:36 fsfs kernel: cxgb0: link state changed to DOWN
Apr 29 08:15:04 fsfs devd: Executing '/etc/rc.d/dhclient quietstart cxgb0'
Apr 29 03:15:04 fsfs kernel: cxgb0: link state changed to UP
Apr 29 03:15:04 fsfs kernel: cxgb0: link state changed to UP

Across 50Gb, I got almost 1.5Gb throughput -- so I'm really using the 10Gb card, but not very effectively. And the "reporting" window shows only low megabits on the other link (gigabit ethernet).

It's rsync driving it -- but multiple parallel runs on different filesystems, using rsyncd on the other end so encryption isn't an issue. So I ought to be able to get rather more throughput than that (I get 850Mb with encryption on a 1Gb port, about 950Mb without).

So, do I have a defective card or cable, or do I need to check firmware versions, or do I need to look for more information, or what?

This is my first attempt to run the pair of cards I bought. They came up fine, and I remembered how to configure the network apparently (static IPs in a different subnet on the two 10G ports, don't need a static route when it's just a point-to-point link like this). I can ping the 10G address of the other system, too.

 

[David Dyer-Bennet]

With further testing -- definitely getting traffic through the 10G channel successfully; this is a synthetic test, just copying /dev/zero to /dev/null over the 10G channel using netcat:

(The earlier case was actual rsync of disk data; the disks on one end or the other were presumably the limiting factor, given that the rsync processes weren't maxing out their cpus.)

As you see, it's bouncing the connection about every 5 minutes, though. It's a short bounce -- in particular it doesn't break the netcat connection. And the log entries as shown above occur at the times of those drops. I've looked at both sides, and the logs aren't showing any more info than given above; no apparent cause for the drop, just a record that it occurred.

 

[dlavigne]

Were you able to figure out the cause of this?

 

[David Dyer-Bennet]

No, I have no idea what's going on.

But...maybe a recent update fixed it. At least, it seems to have stopped happening. I haven't made any manual change in physical or software config for the 10G, but I have upgraded to latest 9.10-Stable versions as they came along (currently on FreeNAS-9.10-STABLE-201605021851 on both ends of that Twinax cable). I think that's at least two upgrades.

I don't see it happening with either replication streams, or with my synthetic netcat workload (same two cases as I tested originally). (Just re-ran the netcat benchmark, since I haven't used that since the first time I ran it to verify the cards worked all the way up to 10G, graph below.)