10GB Direct Connect XEN/FreeNas

[aribs]

Hi,
I am trying to make a direct connection between Freenas and Xen server. I have Chelsio N320E dual nics in each machine. Configs are as follows:
Freenas:
192.168.10.10
255.255.255.0
mtu 9000
Xen Server
192.168.10.15
255.255.255.0
mtu 9000

I have only included one set of config in the post but the picture contains the config on the second port of the nics.

I am not getting any connectivity, Freenas Media Status says "no carier" please see the picture

I have 4 DAC and i have tried different cable same result. Can someone let me know what I am doing wrong?
Thanks

 

[acp]

I had similar issues with mellonax cards. Sometimes they wouldn't see the carrier despite the other side seeing it. If I brought the interface on freenas down and back up again it would work. The ultimate solution was put in switch in the middle and the problem went away.

 

[aribs]

I actually have 2 intel gigabit lan port lag together. I wanted to use the 10gb for only iSCSI targets. I have tried deleting the lag restarting it but I had the same result so I end up restoring the lag and I am still at the same point with no connectivity.

Aside from adding a switch can you suggest something that I can do to resolve the issue

 

[acp]

ifconfig (interface) down
Wait a couple of seconds

ifconfig (interface) up

This worked for me.

 

[acp]

ifconfig (interface) down
Wait a couple of seconds

ifconfig (interface) up

This worked for me.

And I would call this a work around and not a resolution.

 

[aribs]

ifconfig (interface) down
Wait a couple of seconds

ifconfig (interface) up

This worked for me.

Ya, it didn't work. Any other suggestions?

 

[Elliot Dierksen]

I suspect that the cards don't like the DAC cables, or those cables are bad. It depends on the switch/NIC, but some are much more fussy than others. You said you have two sets, so that seems the most likely cause is the former. I would suggest getting some Chelsio short range optics, and fiber patch cords (OM3 or OM4).

 

[aribs]

I suspect that the cards don't like the DAC cables, or those cables are bad. It depends on the switch/NIC, but some are much more fussy than others. You said you have two sets, so that seems the most likely cause is the former. I would suggest getting some Chelsio short range optics, and fiber patch cords (OM3 or OM4).

I got the original cable recommended by Chelsio for this card.

1m (3ft) CHELSIO TAPCABLE1M Compatible 10G SFP+ Passive DAC Twinax Cable

3m TAPCABLE3M CHELSIO Compatible 10G SFP+ Passive Direct Attach Copper Cable

Any other suggestions?

 

[Elliot Dierksen]

I would suggest you try to connect the DAC cables to the two ports on the same card. It could be one of the cards, but it is hard to tell for sure without finding a way of validating the individual components.

 

[aribs]

I would suggest you try to connect the DAC cables to the two ports on the same card. It could be one of the cards, but it is hard to tell for sure without finding a way of validating the individual components.

Yes, I had tried that too. So far I had no luck. I have exhausted all options that I can think of. I have ordered another DAC cable to see if that works. It might be a possibility that the Chelsio branded cable may not work without a switch. So I am going to test that option the cable arrives next week but in the mean time if you can think of anything else please let me know. Thanks.

 

[Geek Baba]

@aribs - did this setup worked? Ordered few Chelsio N320E to do the similar direct connection.

 

[aribs]

@aribs - did this setup worked? Ordered few Chelsio N320E to do the similar direct connection.

Yes and no, as far as I remember I read a compatibility list of freebsd and Chelsio N320E was not listed, which means that it is not fully supported on freebsd. In my situation I had Freenas on one side and ESXi and Xenserver on the other side which needed to communicate with Freenas. This card is definitely not supported on ESXi (there are some workaround suggested of installing a driver for ESXi but that didn't work for me instead it froze my system).

Long story short I end up getting Intel x520 dual port adapter and that worked flawlessly and it is still operational. So I believe the issue I was having was related to the incompatibility of the hardware. If you have a compatible hardware the configuration should work fine for you.

 

[jgreco]

I got the original cable recommended by Chelsio for this card.

1m (3ft) CHELSIO TAPCABLE1M Compatible 10G SFP+ Passive DAC Twinax Cable

3m TAPCABLE3M CHELSIO Compatible 10G SFP+ Passive Direct Attach Copper Cable

Any other suggestions?

The problem is that the DAC cable needs to be compatible at BOTH ends. If you have a FooBrand card at one end and a BarBrand at the other, and the DAC is only compatible with one of them, things won't work.

This is one of the reasons I'm a strong proponent of avoiding DAC cables and just using the optics, and that's what we suggest doing in the 10 Gig Networking Primer. Fiber and optics always work, and are often just as cheap as the DAC solution, with the added advantage that you can change the fiber length at a later date.

Most of the Chelsio N320E's I'm seeing on eBay right now seem to be sourced from China, which isn't a strong endorsement as a lot of knockoffs source from China. It's better to pay the higher price for a decent card like the Chelsio T520-CR, which works swimmingly well with FreeNAS, and also has a driver for ESXi that works fine. I don't know offhand about Xenserver. There's a warning about knockoff cards in the 10 Gig Networking Primer too.

 

[Geek Baba]

Yes and no, as far as I remember I read a compatibility list of freebsd and Chelsio N320E was not listed, which means that it is not fully supported on freebsd. In my situation I had Freenas on one side and ESXi and Xenserver on the other side which needed to communicate with Freenas. This card is definitely not supported on ESXi (there are some workaround suggested of installing a driver for ESXi but that didn't work for me instead it froze my system).

Long story short I end up getting Intel x520 dual port adapter and that worked flawlessly and it is still operational. So I believe the issue I was having was related to the incompatibility of the hardware. If you have a compatible hardware the configuration should work fine for you.

Great to hear, I already had x520 and its connected via Unifi switch, ordered more x520 for dedicated connection, I hope the direct connection have significant improvement.