Intel 10Gb NIC no link, laggy console (Figured it out)

[Visseroth]

OK, so I've been struggling with this server for a while but I can't afford to just toss it after putting money into it already.
Prior problems include random reboots when transferring data at 80+% saturation on the on-board 1Gb NICs,
OEM Critical Interrupt Software NMI @ BUS:0 /Dev:1 /Func:0 - Asserted

The system is a X8DTH motherboard, 48GB of RAM. 5 LSI 9811-8i controllers and a Intel EXPX9501AFXSR 10GbE XF SR FIBER CHANNEL SERVER ADAPTER CARD. Upon startup the NIC was recognized, I configured it's IP but there was no link on the switch. After a while the console started becoming laggy then the NIC disappeared. I sent a graceful shutdown and it took forever to shutdown. When it did start to go down I saw that IX0 link kept going up and down before it shutdown.

I'm currently working on installing the NIC in my desktop to confirm that it is working correctly. But does anyone have any ideas on WTF is going on?

 

[Visseroth]

Figured out what was going on. The 10Gb NIC is bad. I put it in my desktop and had the same results. Laggy desktop, unresponsive applications, no link to the switch, ect.
I swapped the back plates to another NIC and it linked perfectly, same model, ect.

 

[jgreco]

The early generation 10G stuff were watt-burners and often had heatsinks on the optics and the chipset because they could be burning 10-20 watts of power.

I specifically avoided suggesting those kind of cards in the 10G primer, in part because many of the pre-SFP cards either have nonremovable optics or XFP, which makes them far less flexible. Normally its the optics on these suckers that go fail-y, but yours sounds like the entire thing is going bad.

Why don't you wander on over to the Intel warranty support page and see if maybe it's still covered under warranty?

 

[Visseroth]

Really? Good to know. I could find some SFP+ replacement cards instead. I still have SFP+ modules I haven't put to use yet.
It would be better to have something solid instead of a heater.
What would you suggest?

 

[jgreco]

Seriously, check the warranty on your existing card first. Intel warrants a lot of those for "lifetime", which is a funny term in the computer world that actually means something like "until we decide to declare them all dead."

The Intel X520-SR2 (or X520-DA2) is the go-to card around here for general 10G, but unfortunately Intel has it set to only work with Intel SFP+ optics. This card works well with FreeNAS but may need a tweak or two to get there.

Your best choice is one of the Chelsios, which also require Chelsio branded optics. The T420-CR is a pretty good option (works perfectly with FreeNAS out of the box) but you may not require dual ports. Unfortunately they don't make any single port cards in that generation. You can move down to the N310E, I *think*, but do some research first.

 

[Visseroth]

ahh, those Chelsio cards are spendy little cards! :P

Well I haven't put the server or the switch on the network yet so there's plenty of time of changes. I'll probably dump these cards and go get a couple of those X520-DA2 cards

But yea, I tried to check the warranty on the card, couldn't find anything so I sent intel a inquiry into the card. If I can warranty it then sell it to pay for one of those other cards I will.

Good to know about the cards though, I didn't realize those were the go-to cards!

 

[jgreco]

Yeah, apparently iXsystems uses them so of course they're going to be problem-free out of the box.

Spendy would be the nice 40Gbps card (T580-CR), I've got 40G ports open but I can't really even get the filer to peak out 20Gbps so what's the point. Need a big upgrade to all-SSD or something like that. ;-)

 

[Visseroth]

lol! I currently don't see how one could use that much bandwidth. Large file transfers would be done pretty dang quick!
I could definitely see the purpose in a corporate environment though!
and all-SSD? Yea, I've thought about that one myself as well. I'm still waiting on those 4 to 10TB SSD drives at a decent price.
But good to know on the compatibility. You have definitely informed me of more than I knew before!

 

[Visseroth]

Well I took the leap and picked up a couple of these...
http://www.ebay.com/itm/231812102720?_trksid=p2060353.m1438.l2649&ssPageName=STRK:MEBIDX:IT
I'll have to see if my Finisar modules work and if not then I'll go find some modules

Thanks for the help BTW!

 

[jgreco]

The Chelsios require Chelsio optics, http://www.chelsio.com/nic/t4-unified-wire-adapters/t420-cr/

"SFP+ optics sold separately. Only Chelsio-supplied modules may be used"

But if you look around eBay you may find someone marketing "compatible" optics which are usually just Finisars someone ran through a programmer, which should be just fine (may void warranty but you bought it on eBay anyways).

 

[Visseroth]

Ineed!

But a guy posted here he had some success. May be that his was re-programmed, but eh, worth a shot. Would rather have something compatible that works than something that might work and contributes to my heating bill. LOL

But yea, if it's used it doesn't always come with a warranty, but worth a shot.

 

[Visseroth]

Found these...
http://www.ebay.com/itm/New-Chelsio...477524?hash=item56811ef694:g:mTUAAOSwpdpVdg6h

So I grabbed a couple. Can't complain about the price.

 

[Visseroth]

Well the NICs came in. Thank you BTW for the recommendation. BTW the Finisar modules did indeed work, both of them. I don't know if they were previously flashed to work with Chelsio, but I have a 10Gb fiber link to my switch. Now to do some stress testing (again) on my new (new to me) server. Currently I'm trying to transfer all my data from my old server to the new one over scp but I'm only getting 40MB/s. Funny thing is I can transfer from my Windows Desktop to my old server at 80 to 120MB/s. My new server my read and write speeds vary between about 60MB/s to 90MB/s, but I would think that I could achieve at least 100MB/s between the two servers being they are both connected at 10Gb through the switch. Maybe it's a SCP issue?
Basically I'm trying to migrate from the old server to the new one but I also need to hammer the new one as hard as I can to make sure it'll handle the load and because when I was using the onboard Gb NICs at 800Mb/s for longer than a few hours the system would reboot. I need to determine if this reboot issue was related to something other than the NICs.

 

[Visseroth]

I created a new thread for the network slowness issue as iPerf showed only 1Gb/s throughput.
https://forums.freenas.org/index.php?threads/10gb-nics-and-iperf-is-showing-1gb-speeds.41218/

 

[jgreco]

Sorry for the delayed reply. On the road. I'll check in on your other thread.