iSCSI Xenserver Multipath performance over gigabit LAN

[rptl]

I finally have finished Xenserver build with Multipath in place. But still struggling with performance.

FreeNAS setup (2 nics, 2 different subnets):
1 portal - listening on 10.1.0.1 and 10.2.0.1
1 extent (2T)

XenServer iSCSI setup:
[root@accxen01 boot]# iscsiadm -m session
tcp: [3] 10.1.0.1:3260,3 iqn.2011-03.org.accsan1.istgt:accnas02 (non-flash)
tcp: [4] 10.2.0.1:3260,3 iqn.2011-03.org.accsan1.istgt:accnas02 (non-flash)

[root@accxen01 boot]# multipath -ll
36589cfc00000009e65ba89b64650b625 dm-1 FreeBSD,iSCSI Disk
size=1.8T features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
|- 24:0:0:0 sdc 8:32 active ready running
`- 23:0:0:0 sdb 8:16 active ready running

/etc/multipathf.conf
device {
vendor "FreeBSD"
product "iSCSI Disk"
path_grouping_policy multibus
path_selector "round-robin 0"
rr_min_io 100
}

I am getting only 67MB/s Write and 75MB/s Read rate. I was expecting, at least, twice this speed, since it's a multipath with 2 nics.

At last, when I run traceroute from FreeNAS to Xenserver IPs, I am getting following result:
traceroute to 10.1.0.2 (10.1.0.2), 64 hops max, 52 byte packets
1 10.1.0.2 (10.1.0.2) 0.113 ms !Z 0.095 ms !Z 0.087 ms !Z

What '!Z' means? Is that a problem?

 

[cyberjock]

The !Z meas that an error occurred while getting the latency. This likely means your network configuration is broken, or your hardware is broken.

You didn't specify the hardware on your FreeNAS server, but you're going to need beefy hardware to get good performance. If other users are any indicator, your hardware probably isn't beefy enough to expect more.

 

[L]

The definition in the traceroute man page for !Z says
!Z Communication with destination host administratively
prohibited.

Sub millisecond latency is really good though. Multipathing with round robin won't give you twice the speed. It will give you twice the clients throughput.

 

[rptl]

FreeNAS is Dell PE 830, 8GB.

iSCSI Nics:
RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet

Could be a driver issue?

I also have 1 Nic "RealTek 8169/8169S/8169SB(L)/8110S/8110SB(L) Gigabit Ethernet" which seems to work fine (no !Z in traceroute).

 

[rptl]

The definition in the traceroute man page for !Z says
!Z Communication with destination host administratively
prohibited.

Sub millisecond latency is really good though. Multipathing with round robin won't give you twice the speed. It will give you twice the clients throughput.

But, you think I could get better rates than 67/75Mb, using 2 nics? I see somes folks saying they got about 90~110Mb/s using only one nic, that is why I am asking.

 

[cyberjock]

You didn't specify the hardware on your FreeNAS server, but you're going to need beefy hardware to get good performance. If other users are any indicator, your hardware probably isn't beefy enough to expect more.

FreeNAS is Dell PE 830, 8GB.

iSCSI Nics:
RealTek 8168/8111 B/C/CP/D/DP/E/F/G PCIe Gigabit Ethernet

I called it! What do I win?

Honestly, if you want good performance with iSCSI for VMs, you need at least 64GB of RAM, an L2ARC, and a CPU that isn't 10+ years old. Yes, those things started in 2005...

Feel free to read the forum about iSCSI and VMs. It's not pretty. There's lots of graveyards where people gave up because they couldn't get appropriate hardware to run VMs.

 

[rptl]

I called it! What do I win?

Honestly, if you want good performance with iSCSI for VMs, you need at least 64GB of RAM, an L2ARC, and a CPU that isn't 10+ years old. Yes, those things started in 2005...

Feel free to read the forum about iSCSI and VMs. It's not pretty. There's lots of graveyards where people gave up because they couldn't get appropriate hardware to run VMs.

Thanks for your advices. I do not see FreeNAS struggling on CPU, WaitIO or memory usage when Network is "heavily" active. Not sure if the impact is 'transferred' to Network performance, though.

In the other hand, I just realized that I can get better rates using only one nic, non RTL, but a broadcom used for admin, placed in the same server.

Will do some more tests.

 

[cyberjock]

Oh, don't worry. Just wait until your zpool has a few VMs running. The IO will bottleneck all the heck and back, with your datastore potentially dropping out regularly due to timeouts.

Just don't say we didn't warn you. ;)

 

[rptl]

I was comparing nics and can see that some media settings are different from working and troubled cards:

Good nics:
media: Ethernet autoselect (1000baseT <full-duplex)

Troubled:
media: Ethernet autoselect (1000baseT <full-duplex,master>)

I know it's non related directly to FreeNAS, but How can I remove 'master' setting from media? I know that is possible to add nic options using FreeNAS webconsole. Is there a way to remove it ?

 

[Ericloewe]

Ar-Tee-Ell-Eight-Triple-One

We meet again.

 

[rptl]

I was able to remove 'master' from media options manually. Nothing has changed, though.

Quick update about !Z. It's something on Xenserver side. I am getting same result from other boxes, even using Admin Xenserver Nic as target.

From Linux, I get !X.

From man page:
!X means "communication administratively prohibited" and !Z "communication with destination host administratively prohibited"

 

[rptl]

Just wanted to report my performance after few adjustments.
What has changed:
- Test was running over DRDB replicated file system over HD block mapping.

Migrating to Zpool increased drastically my MPIO rates

###########################################################################
-----------------------------------------------------------------------
CrystalDiskMark 3.0.4 x64 (C) 2007-2015 hiyohiyo
Crystal Dew World : http://crystalmark.info/
-----------------------------------------------------------------------
* MB/s = 1,000,000 byte/s [SATA/300 = 300,000,000 byte/s]

Sequential Read : 180.224 MB/s
Sequential Write : 38.334 MB/s
Random Read 512KB : 183.293 MB/s
Random Write 512KB : 38.067 MB/s
Random Read 4KB (QD=1) : 22.452 MB/s [ 5481.5 IOPS]
Random Write 4KB (QD=1) : 16.253 MB/s [ 3968.0 IOPS]
Random Read 4KB (QD=32) : 61.422 MB/s [ 14995.6 IOPS]
Random Write 4KB (QD=32) : 23.836 MB/s [ 5819.4 IOPS]

Test : 100 MB [T: 91.0% (66.3/72.8 GB)] (x3)
Date : 2015/04/27 20:59:45
OS : Windows 7 Professional SP1 [6.1 Build 7601] (x64)
###########################################################################

And yes, it is still old PE 830 6GB ram and 8111 crap nics used to multipath. Not bad... I just wanted a little bit more "Sequential Write" performance, but I can live with that. MPIO throughput is winning.

 

[cyberjock]

That is such great news. Ok, I lied.

Your test was 100MB. Really? You realize that on the first test you cached all that in RAM, and then all the subsequent tests used RAM for reads?

Oh, how I wish people understood that benchmarks don't reflect reality unless your benchmark is a reflection OF reality.

Your test proved absolutely nothing.

 

[rptl]

I do understand what you did expect with this hardware and the reason for your warning, but like I said, I am not complaining about the performance I am getting.

Oh and I have forgotten to mention: There are +15 VMs running fine and absolutely no Timeout for almost 1 week.