ISCSI Multipath, speed is the same with one or two paths active

[danypd69]

Hello,
I'm doing some experiments in my lab with iscsi multipath.
I have a server with freenas 8.3RC1 (installed today); the server is a dell 2940 with 2 xeons, 12Gb of memory and 2 320gb sas disks.

I want to do some speed tests so I configured the disks as raid0.
A simple dd on the freenas server shows that the disk can transfer about 300MB/sec

Code:

[root@freenas] /mnt/Raid0# ls -lh Disk00
-rw-r--r--  1 root  wheel    80G Mar 20 23:01 Disk00
[root@freenas] /mnt/Raid0# dd if=test of=/dev/null bs=32k count=524288
80000+0 records in
80000+0 records out
2621440000 bytes transferred in 8.653268 secs (302942191 bytes/sec)
[root@freenas] /mnt/Raid0#

Then I created a scsi target with 2 interfaces on different subnets (192.168.0.35/24 and 192.168.10.1/24)

On another server with proxmox vm I configured a scsi disk using multipath; the configuration seems correct and it shows both paths as active

Code:

root@server:~# multipath -ll
330000000eab5e18d dm-17 FreeBSD,iSCSI Disk
size=80G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=2 status=active
  |- 7:0:0:0 sde 8:64 active ready running
  `- 8:0:0:0 sdf 8:80 active ready running

This is my /etc/multipath.conf

Code:

blacklist {
        wwid *
}

blacklist_exceptions {
        wwid "330000000eab5e18d"
}

defaults {
        polling_interval        2
        selector                "round-robin 0"
        path_grouping_policy    multibus
        getuid_callout          "/lib/udev/scsi_id -g -u -d /dev/%n"
        rr_min_io               1000
        failback                immediate
        no_path_retry           fail
}

I tried to measure the read speed with dd on the mounted disk and I got about 116MB/sec

Code:

root@server:~# dd if=/dev/dm-17 of=/dev/null bs=32k count=524288
524288+0 records in
524288+0 records out
17179869184 bytes (17 GB) copied, 148.239 s, 116 MB/s

If I disable one of the interfaces I get about the same speed

Code:

root@server:~# ifconfig eth1 down
root@server:~# multipath -ll
330000000eab5e18d dm-17 FreeBSD,iSCSI Disk
size=80G features='0' hwhandler='0' wp=rw
`-+- policy='round-robin 0' prio=1 status=active
  |- 7:0:0:0 sde 8:64 failed faulty running
  `- 8:0:0:0 sdf 8:80 active ready  running
root@server:~# dd if=/dev/dm-17 of=/dev/null bs=32k count=524288
524288+0 records in
524288+0 records out
17179869184 bytes (17 GB) copied, 149.271 s, 115 MB/s
root@server:~#

Here is the iperf output for the interfaces

Code:

root@server:~# iperf -c 192.168.0.35
------------------------------------------------------------
Client connecting to 192.168.0.35, TCP port 5001
TCP window size: 23.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.252 port 55732 connected with 192.168.0.35 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes    941 Mbits/sec
root@server:~# iperf -c 192.168.10.1
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
TCP window size: 23.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.2 port 60092 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes    942 Mbits/sec

Is it a normal behaviour ? From what I read the multipath should increase the performance using more than one connection but it does not seem to work in my case.

 

[jgreco]

You ought to see increased performance if everything is capable of it. Most likely, something isn't set up right or possibly you just happened to be around the capacity your server could handle.

You should probably check to see that your multipath is actually doing what you think it is doing, for example by checking on the FreeNAS server to see if the connections you expect actually exist and appear to be in use.

 

[danypd69]

I tried to disconnect the other path removing the network cable, the disk work so it mean that both paths works ok but the speed is still 116 MB/sec

Code:

root@server:~# dd if=/dev/dm-17 of=/dev/null bs=32k count=524288
524288+0 records in
524288+0 records out
17179869184 bytes (17 GB) copied, 148.357 s, 116 MB/s

On the freenas server I have created a single portal with two ip addresses, is this the correct way to do it?

I have done another test, the network graphs during the copy shows about 500Mb/sec for both interfaces, so it seems that both paths are used but at 50% capacity.

 

[cyberjock]

Now that 8.3.1-RELEASE is out, I'd try upgrading to that and see if that helps.

 

[danypd69]

Upgrade done, no changes :-(

 

[jgreco]

It's kind of unlikely that your NAS would be capable of delivering almost exactly one gigabit of traffic. I still vote "underlying issue with configuration" and to me it smells like maybe the target system is getting all the traffic on a single interface.

Why don't you walk us through the actual configuration a bit more. Could you describe the two network segments (ethernet interface types, switch types, are they completely segregated or is there a router that might be complicating the issue, etc)? What happens if you run two iperf sessions simultaneously, so you see something well in excess of 1Gbps, or does it seem mysteriously capped at around 1Gbps?

What happens when you do "netstat -an"? Do you see the expected multipath sessions?

That's just a start, I can name a dozen more questions that are dependent on the answers to the above.

 

[danypd69]

I also think it is a config problem but I don't know what to look at.

The network is very simple, one of the path (192.168.10.0/24) is a simple crossover cable between the two servers;
the other path (192.168.0.0/24) is connected to an unmanaged 1gb switch.

Jumbo frames are not enabled because the switch does not support them.
I tried to enable them on the second interface but I did not get any change in the transfer rate.

Here is the iperf output, I launched iperf -s on the freenas server and two iperf -c on the client

Code:

[root@freenas] ~# iperf  -s 
------------------------------------------------------------
Server listening on TCP port 5001
TCP window size: 64.0 KByte (default)
------------------------------------------------------------
[  4] local 192.168.0.35 port 5001 connected with 192.168.0.252 port 33156
[  5] local 192.168.10.1 port 5001 connected with 192.168.10.2 port 33280
[ ID] Interval       Transfer     Bandwidth
[  4]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec
[  5]  0.0-10.0 sec  1.10 GBytes   939 Mbits/sec

Code:

daniele@server:~$ iperf -c 192.168.10.1
------------------------------------------------------------
Client connecting to 192.168.10.1, TCP port 5001
TCP window size: 23.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.10.2 port 33280 connected with 192.168.10.1 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes    942 Mbits/sec

Code:

daniele@server:~$ iperf -c 192.168.0.35
------------------------------------------------------------
Client connecting to 192.168.0.35, TCP port 5001
TCP window size: 23.8 KByte (default)
------------------------------------------------------------
[  3] local 192.168.0.252 port 33156 connected with 192.168.0.35 port 5001
[ ID] Interval       Transfer     Bandwidth
[  3]  0.0-10.0 sec  1.10 GBytes    942 Mbits/sec

netstat -an on the freenas host shows the connections as active

Code:

tcp4       0      0 192.168.10.1.3260      192.168.0.252.44125    ESTABLISHED
tcp4       0      0 192.168.0.35.3260      192.168.0.252.44247    ESTABLISHED

 

[Got2GoLV]

multipath should increase the performance using more than one connection

This.

How are you initiating multiple connections to the iSCSI target ?

It seems you are only initiating one, thus you will only see the transfer bandwidth equivalent to of one of your path links.

And even if you initiate multiple connections, odds are they will probably be close in timing where you wont notice much of an improvement.

Multipath does better with many, many connections. Then you increase the likelihood of benefiting from the multi path setup.
(as far as bandwidth aggregation...the redundancy benefit is always there)

Each connection will start using one of the links until the configured number of IOS threshold is met, then it switches to the other link.

Ex:
If the initiator is configured for 1k IOPS (ESX default), then every 1k IOPS, the active path will be cycled to the next link.
This is for each connection.

So, for testing a 2 link MPIO setup, if you can start 2 connections that are using a different link from each other at any time, you should be able to double your transfer speed.
Ex:
Conn 1 on link 1
Conn 2 on link 2
Then after each connection does 1k IOPS, they swap connections to:
Conn 1 on link 2
Conn 2 on link 1

That way each connection is using its own link.

But, in real world, this is unlikely, unless you have man connections happening all the time.

Try setting up many simple VMs where you can run dd on each at the same time, and you should see that the sum of all the bandwidth will be more than one gig link.


NOTE: one synthetic way to see more bandwidth is to decrease the IOPS threshold on the initiator so that it cycles links more often.
I dont know if that is possible on your iSCSI initiator, but look into it, see what the default is, and try changing it to something extreme like, 1.
For testing( in real world/production, that might not be a good idea)

 

[danypd69]

Here are the commands that I'm using to create the connections

Code:

root@server:/home/daniele# iscsiadm  -m node --targetname "iqn.2011-03.org.example.istgt:freenas" --portal 192.168.10.1:3260 --login
Logging in to [iface: default, target: iqn.2011-03.org.example.istgt:freenas, portal: 192.168.10.1,3260]
Login to [iface: default, target: iqn.2011-03.org.example.istgt:freenas, portal: 192.168.10.1,3260]: successful
root@server:/home/daniele# iscsiadm  -m node --targetname "iqn.2011-03.org.example.istgt:freenas" --portal 192.168.0.35:3260 --login
Logging in to [iface: default, target: iqn.2011-03.org.example.istgt:freenas, portal: 192.168.0.35,3260]
Login to [iface: default, target: iqn.2011-03.org.example.istgt:freenas, portal: 192.168.0.35,3260]: successful
root@server:/home/daniele# 

The initiator that I'm using is open-iscsi on linux, the number of operations between switching the links is set to 10 for testing (parameter rr_min_io).

I will try to create two different vm.

 

[danypd69]

I tried to create two identical VMs.
Two dd at the same time gives 57.1 MB/s for each virtual machine.

 

[jgreco]

Code:

tcp4       0      0 192.168.10.1.3260      192.168.0.252.44125    ESTABLISHED
tcp4       0      0 192.168.0.35.3260      192.168.0.252.44247    ESTABLISHED

Well that there's your problem. You're running both the connections to/from one interface on the remote host. You're hitting the remote host's limit for traffic on a single interface. Fix your remote host to use two interfaces.

 

[cyberjock]

So what you're saying is.. not a FreeNAS problem :P

 

[danypd69]

Well that there's your problem. You're running both the connections to/from one interface on the remote host. You're hitting the remote host's limit for traffic on a single interface. Fix your remote host to use two interfaces.

Ok, I recreated the connections
This is the netstat of the freenas host

Code:

[root@freenas] ~# netstat -an|grep 3260
tcp4       0 163776 192.168.0.35.3260      192.168.0.252.45204    ESTABLISHED
tcp4       0    800 192.168.10.1.3260      192.168.10.2.50634     ESTABLISHED
tcp4       0      0 192.168.0.35.3260      *.*                    LISTEN
tcp4       0      0 192.168.10.1.3260      *.*                    LISTEN

and this is on the server

Code:

root@server:~# netstat -an|grep 3260
tcp        0      0 192.168.10.2:50634      192.168.10.1:3260       ESTABLISHED
tcp        0      0 192.168.0.252:45204     192.168.0.35:3260       ESTABLISHED

Transfer speed is still the same

 

[jgreco]

You're going to have to get more proactive about proving what's actually going on here, because the facts you've provided to this point so strongly suggest that you've got a single-interface bottleneck at some point.

For each physical system you have, determine how to measure current speeds at the network interface. For example, for a FreeBSD based system with an em0 and bge0 interface, that'd be "netstat -I em0 1" and "netstat -I bge0 1".

I don't really know what the equivalent would be on your Proxmox server, but you need to figure that out. This isn't a FreeNAS issue, it's a complicated-networks deployment issue. You've done *something* that isn't correct, but it isn't clear what.

 

[danypd69]

Ok I will do the netstat as soon as I'm in office.

The next step will be to use a different network interface for one of the paths, I'm currently using a dual port nic but I have another one installed in the server. I will try to use another port.

 

[danypd69]

Hello
I have changed one of the two interfaces and I have done the netstat commands while doing a transfer ; the netstat command have been done at the same time in two different ssh connections.
Here is the output

Code:

[root@freenas] ~# netstat -I bce0 1
            input         (bce0)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
         4     0     0        300          4     0        560     0
         4     0     0        270          4     0        500     0
      3118     0     0     217056       2319     0   30901182     0
      2918     0     0     209658       1543     0   45891992     0
      3429     0     0     247086       1831     0   54491324     0
      3328     0     0     239310       1767     0   53876022     0
      3331     0     0     239466       1771     0   53614310     0
      3434     0     0     246412       1810     0   54140884     0
      3326     0     0     239220       1765     0   53875890     0
      3219     0     0     231558       1700     0   52560400     0
      3352     0     0     240840       1752     0   53744374     0
      3318     0     0     238698       1761     0   54006284     0
      3629     0     0     261150       1911     0   59134470     0
      3340     0     0     241810       1803     0   53923946     0
      3576     0     0     257928       1905     0   60041876     0
      3550     0     0     255534       1891     0   58217060     0
      3716     0     0     268254       1958     0   60346108     0
      3648     0     0     262782       1928     0   60180606     0
      3588     0     0     257964       1884     0   58079534     0
      3693     0     0     265974       1952     0   60444232     0

Code:

[root@freenas] ~# netstat -I em0 1
            input          (em0)           output
   packets  errs idrops      bytes    packets  errs      bytes colls
         0     0     0          0          0     0          0     0
         2     0     0        180          2     0        180     0
         0     0     0          0          0     0          0     0
         2     0     0        180          2     0       4276     0
         0     0     0          0          0     0          0     0
      4457     0     0     312450       3058     0   50158680     0
      2896     0     0     209040       1571     0   46738232     0
      3445     0     0     247578       1804     0   55320584     0
      3457     0     0     247842       1760     0   53875162     0
      3484     0     0     249864       1769     0   54269314     0
      3247     0     0     233742       1727     0   53479822     0
      3295     0     0     237150       1737     0   53873842     0
      3189     0     0     229242       1679     0   51251758     0
      3345     0     0     240450       1728     0   53873248     0
      3571     0     0     256710       1870     0   57554178     0
      3558     0     0     255842       1842     0   55491262     0
      3465     0     0     249954       1838     0   55045506     0
      3646     0     0     262428       1937     0   59787244     0
      3716     0     0     266856       1944     0   59132502     0
      3755     0     0     269718       1975     0   59757182     0

I really don't understand what happens. It seems that the two interfaces are used correctly but the total bandwidth is always 1gb
I tried to do the netstat after disconnecting the 192.168.10.1 interface and the netstat shows about 1gb on the active interface

 

[jgreco]

Yes, it's been clear for awhile now that the problem isn't the FreeNAS box, which appears to be doing the correct things. As I said, you'll need to figure out similar testing on the Proxmox side to determine what is happening there.

 

[danypd69]

Tomorrow I will install another server without proxmox to do some testing. What do you think I could use ? Linux ? Freebsd ? I usually use linux but it would not be a problem to install a bsd box for testing.

 

[jgreco]

We're more familiar with FreeBSD here. I occasionally have to cope with Linux, but usually I end up a bit frustrated at the Linux-does-it-different-itis. I have no idea why there seems to be no sane equivalent to "netstat -I xx0 1" for example, and it sucks to have to devise a shell script to repeatedly run "ifconfig xx0" and extract the "RX bytes" / "TX bytes" and then do math and then print it in a formatted fashion just to do what in FreeBSD, Solaris, etc., is a quick and obvious thing.

A command like "netstat -I xx0 1" is so fundamental to see traffic levels on your interfaces, and it is ideally suited to help debug this sort of thing.

 

[danypd69]

Hello, I have tried to connect to the iscsi target from my workstation (different version of linux) but the problem is still the same.
The interfaces are both used at 50% of their capacity.
If I install a bsd box is there someone that can help me with the iscsi initiator configuration?