FC Target slow performance?

[d0kt0r]

Hi guys!

First of all, my first post here!!!

I was trying to use a replaced motherboard to try to make some lab environment for Windows Server Hyper-V 2012.

I've follow the very nice guide : https://forums.freenas.org/index.php?threads/freenas-9-3-fc-fibre-channel-target-mode-das-san.27653/

The setup :
1x Brocade 200E 4GB
1x Gigabyte z87 Motherboard 4GB RAM DDR3 CPU Celeron G1820 + 1SSD 250 Samsung Evo 840 + Qlogic FC QLE2464 (Quad Port FC 4GB) - Installed with Freenas 9.3.1 and booting from a USB 8GB drive


I haven't follow some of the recommended guidelines, so, Im initially guessing that the slow performance is from there (specially from RAM, but I will try to upgrade this afternoon and see if I get better results).

And everything worked fine, like a charm indeed I can see the storage from another PC (using a Emulex 4GB LPe1150). The problem is in fact in the performance, using the ATTO disk benchmark on the new SAN volume I can only get around 200MB/sec. If the FC card is 4GB, shouldn't be more close to the 400mb/sec?
Having in mind that the SSD is not a bottleneck, Im only guessing the FC card is the bottleneck here, but still dont know why.
And another thing, I get the same speed using 1 FC cable connected than two. I've installed MPIO on windows Server 2012 R2, but, still the same performance.

My next step will be upgrading the memory from 4GB to 16GB and see if I get better performance.

Anything else I should check for that performance improvement?

Should the MPIO through iscsi should work by default?

Cheers and thanks!!!

 

[mav@]

Can you try to update to latest nightly builds of FreeNAS 9.3.1? Nightly branch received substantial list of changes in its SCSI target subsystem, that most significantly affected performance of Fibre Channel. On some load patterns it gave me up to double speedup. This code is going to be merged to stable at some point soon. Now it is still in testing, so you can participate and report your results.

 

[Ericloewe]

My next step will be upgrading the memory from 4GB to 16GB and see if I get better performance.

4GB is half the minimum requirement, and the minimum requirement is a bit low for a pleasant iSCSI experience. So please do so.

 

[d0kt0r]

Hi,

First test and first failure arg.

I've installed from fresh the "server" with the last nightly build from yesterday ... and the FC target wasn't working. I can view the targets OK from the switch perspective (and on the console they seem up) but when I try to access from the server 2012, I can't see any disk (well , I did the same configuration on the FN 9.3 and it was working OK). I think that nightly build is not working as good as it should.

I will try with another version tonight I guess.

And for Ericloewe, of course, if I get working the FC targets on 10.2, I will update the RAM (Im not that confident that upgrading the RAM will double the speed, but it's worth a try anyway).

Cheers guys!! And thanks for answer!!

 

[mav@]

Sorry, nightly of which branch did you took? I was talking about nightly of 9.3. I have no idea what is in 10.2 now, but I would be very surprised if it would work -- it is not even declared alpha now.

 

[d0kt0r]

Yes hehe You said 9.3.1 ... The idea of the build was a nice one, so I was trying to go two steps forward. I guess I've got two steps behind instead :P

I will try today on the last 9.3.1 nightlies. I will report the results!!

Cheers and thanks again!
Daniel.

 

[d0kt0r]

Same results with the 9.3.1 last nightly.

I will edit this post with the memory upgrade.

 

[mav@]

May be it is small memory, may be slow CPU, may be old cards ... I don't know. But couple weeks ago on nightly build on decent hardware with dual-port 8Gbps FC Qlogic card I've reached full 1.5-1.6GB/s in the same benchmark.

 

[d0kt0r]

I've upgraded the server with 20GB RAM (4+16) and unfortunately the same result.

Mav, two questions from your deployment :

* From your speed 1.6GB/s sounds that you are using both FC Ports 8GB. You just configured MPIO on the windows machine? Have you done something else on the FN?
* Are you using a FC switch between, or it is a direct connection?

Cheers and thanks guys!
Daniel.

 

[mav@]

* From your speed 1.6GB/s sounds that you are using both FC Ports 8GB. You just configured MPIO on the windows machine? Have you done something else on the FN?

Yes, I had two ports connected on both sides. I've configured MPIO on Windows 2012R2, rebooted, after reboot IIRC MPIO policy set to round-robin by itself.

* Are you using a FC switch between, or it is a direct connection?

There was 8Gbps Qlogic FC switch in between, which I zoned to disable extra paths.

 

[HoneyBadger]

I'd look at your physical hardware (SFPs, cables, etc) and its BIOS/configuration in this case. Your speeds hitting a wall where they are strongly implies that something is forcing the link to negotiate at 2Gbps only.

 

[d0kt0r]

Hi Honeybadger,
That's exactly what I think. That speed is clearly 2GB.
The weird thing is that the Qlogic HBA Bios is forced to 4GB/sec, and I did the same on the brocade switch, forcing all the ports to 4GB. Exactly the same output.
I guess if the SFPs are 2gb, something should definitely go wrong and not connecting.
And for cables I guess they are good (if they are dirty I would look for disconnecting issues, and for speed ... I think I stole them for another HP disk array) they are brand new. Orange cables with LC/LC connectors.

Maybe the problem is on the client :
So my next step is trying to get an updated driver or installing another server OS or installing a spare qlogic HBA single port and replace the emulex one.

I know Im missing something here :-/

Thanks for the info guys.
Daniel.

 

[HoneyBadger]

Is it OM1? If you've got mismatched core sizes that could easily do it (62.5um cable, 50um SFPs)

 

[d0kt0r]

Is it OM1? If you've got mismatched core sizes that could easily do it (62.5um cable, 50um SFPs)

Good point.
Both cables are 50/125 and the SFP is 850nm shortwave. At the testing it was with the original brocade SFP, but I've swapped to avago. Same results.
Should I get new cables or should be good with those ones?

Edited : Tried with a direct connection between the Proliant and the FN ... Same result!!! Meh!!

Maybe I should get a better cable?

The next try is removing the emulex lpe1150e on the proliant and put a qlogic if that makes a difference.

 

[HoneyBadger]

It sounds like you're chasing fibre gremlins. Try a new cable just to see; if the one you're using has been twisted/bent it could have internal breakage that results in a lot of loss.

Afraid nothing jumps out at me. Just have to start swapping components until you find the offending party.

 

[fips]

Hi d0kt0r!
I have exactly the same issue.
Could you fix it somehow?

 

[d0kt0r]

Yes, I fixed it.

The problem was on the client HBA. Using another Qlogic card on the client side it works ok. (Tested with the 50/125 new cables)

So, the hardware was :

Server side - Qlogic FC QLE2464 (Quad Port FC 4GB)
Client side - Qlogic FC QLE2460 (Single Port FC 4GB) working at 4GB // with Emulex LPE1150 4GB only works at 2GB (tested with a brocade switch and direct connection)

Hope it helps!!

 

[fips]

hmm... my client HBA is a Qlogic QLE2460...
Well I can change it to another QLE2460 (I only have those cards).

 

[fips]

I am still investigating... So maybe my FreeNAS doesn't provide enough performance.

Is there a simple way to benchmark local disk performance?

 

[sfcredfox]

There are a bunch of posts from jgreco that talk about disk burn in and performance testing.

You can do read/write tests with DD, but you'll need to be very careful since it can destroy data if you target devices. If you want to leave data/pool intact, you can do file level.

Example or something similar:
dd if=/dev/zero of=/mnt/poolname/somefilepath/filename.testfile bs=1M count=how big you want it

you'll need to turn off compression on the pool you're testing though or the write test will be bogus.

You can turn around and do a read test from the file created
dd if=/mnt/poolname/somefilepath/filename.testfile of=/dev/null

research this more, there's better parameters to use than my example, but it will do local disk testing the best way I know how.