Iscsi performance over mpio to datastore on Esxi.

[devops33]

Hello gents,

As the title says, post here if you are unsure.

I am going to right to it so I don't waste people time.

The issue at hand i not getting the performance I should be getting from iscsi MPIO and I could use some else perspective.

example. Direct file copy is at 2.5Gigabyte /s
Iscsi performance is as 70MB/s

Can anyone shot some pointers, its like if the nics were operating at 100 mbit but they are not.
I changed RR path change to 1 from 1000 for testing.


Freenas 9.3

BuildFreeNAS-9.3-STABLE-201501241715
PlatformAMD FX(tm)-6300 Six-Core Processor
32GB ram

pool: zfs-pool8tb
state: ONLINE
scan: scrub repaired 0 in 6h21m with 0 errors on Sun Feb 1 09:21:25 2015
config:

NAME STATE READ WRITE CKSUM
zfs-pool8tb ONLINE 0 0 0
raidz1-0 ONLINE 0 0 0 <--- I know I'm moving to raidz2 soon.
da5p1 ONLINE 0 0 0
da7p1 ONLINE 0 0 0
da4p1 ONLINE 0 0 0
da6p1 ONLINE 0 0 0

Seagate 7200 rpm drives with 32M cache on them and I have 2 spares. This is on a reflashed h200 to lsi firmware and no raid, device is pass through, 6Gbit card 8 port and 6Gbit drives

1 intel gbit for management and one quad port gbit for iscsi.
Each on its own subnet " 50.22 , 51.22, 52.22 53.22.



ESXi Host.

Networking

iscsi MPIO 8 nics on a vlan to isolate traffic and all 4 ip's of the iscsi nics are in the portal. Management on another link.

Lun presented to esxi is in round robin with 4 targets, 1 device and 4 paths.

cisco

sw1>show vlan brief

VLAN Name Status Ports
---- -------------------------------- --------- -------------------------------
1 default active Gi0/1, Gi0/2, Gi0/11, Gi0/12
Gi0/13, Gi0/14, Gi0/15, Gi0/16
Gi0/17, Gi0/18, Gi0/19, Gi0/20
Gi0/21, Gi0/22, Gi0/23, Gi0/24
Gi0/25, Gi0/26, Gi0/27, Gi0/28
Gi0/29, Gi0/30, Gi0/31, Gi0/32
Gi0/33, Gi0/34, Gi0/35, Gi0/36
Gi0/37, Gi0/38, Gi0/39, Gi0/40
Gi0/41, Gi0/42, Gi0/43, Gi0/44
Gi0/45, Gi0/46, Gi0/47, Gi0/48
Gi0/49, Gi0/50, Gi0/51, Gi0/52
2 esxi-iscsi active Gi0/3, Gi0/4, Gi0/5, Gi0/6
Gi0/7, Gi0/8, Gi0/9, Gi0/10

4 ports on each host making it 8 for iscsi on the same vlan.

Iscsi Traffic

All interfaces are in (1000baseT <full-duplex>) I checked the switch and 1gbit link, checked esxi and 1gbit link.

What else should I be looking at?

Gstat says at 70MB drives are not even half busy.

Thanks in advance for any pointers.

 

[cyberjock]

For starters, you don't want to move to RAIDZ2. You want to move to mirrors so you have more IOPS available.

Secondly, for VMs you don't *want* high throughput. You want to tune for low latency (which will also give you higher throughput as a result).

Third, you want more RAM on your FreeNAS box and you'll want a right-sized L2ARC for your working data set. Yes, I know you're about to tell me you can't because you can't put more than 32GB of RAM on that motherboard (I've seen these discussions plenty, so I know exactly how they go...). Well, I don't know what to say. Should have done more reading on using VMs on FreeNAS before you bought your hardware.

Lastly, I can promise you for 1000% certainty you did NOT truly get 2.5GB/sec from 4 disks in a RAIDZ1. Why do I know this? Because 3 disks at saturation speed for 6Gb SATA/SAS can only kick out about 1.5GB/sec. Your 2.5GB/sec is lying to you. I have no clue what "direct file copy" is in your definition of it, but I'm betting you did a dd write test from /dev/zero and compression gave you about 10KB/sec to the disk but the raw write was 2.5GB/sec. So you look kind of silly for not catching an obvious mistake on your part. ;)

I setup MPIO on my FreeNAS machine and ESXi server last week. It works just fine for me (I don't get the kinds of speeds I wanted since I have only 48GB of RAM and no L2ARC) but it does work properly for me. I have no clue if you've actually made a mistake in the MPIO setup or not, but I can tell you that 70MB/sec is about what I'd expect for that hardware (I'd give a range of 60-120MB/sec as the range I'd have expected you to fall into as a best-case scenario).

 

[devops33]

Cyberjock,

I know where you are coming from and I understand the logic behind every though you put into this response I do appreciate your input and I am taking some of those suggestions to heart and probably implementing them. However, maybe I need to rephrase my question or perhaps add some more information so you can understand where I am coming from. First, That same server NAS was running 9.2 over LACP no Iscsi and using nfs and it would saturate all 4 links using 8 VMs over nfs. The only thing that changed was that we are trying to implement MPIO over Esxi to consolidate servers and a fresh install of 9.3. I did the following tests. I placed 4 centos vm's on the esxi and assigned each an IP on the different subnets as the MPIO Iscsi network. I can tell you that ZFS is not the issue. I can saturate those links with no problems. I am not looking at getting 2.5GB per second. my dd test i know its flawed but I should be able to get at least 2gbit or more with this configuration over MPIO Iscsi. Now, I have some concerns. I know that Iscsi implementation is sync on every write and this causes a huge penalty in write performance. To mitigate this, I have installed 2 SSD 120GB in mirror for Zil which are currently not implemented but it will be. I have not determined how big I should make it since I have more reading to do since this will be my first implementation.I never needed it before when I was just using NFS. Second, I know I can saturate close to 4gbit over nfs, but this is Iscsi over MPIO and that perhaps the IOPS over ISCSI might be higher due to the fact that over NFS, drives are 4k but over Iscsi esxi has a limitation of 512 to be able to present to the lun to esxi. So I have to look into what the performance implications are on these facts. However, I would be curious to see if perhaps I missed something. I just don't get why I am having these huge bottle neck. Funny thing is , I did another test and I leaded a few more VMs over the datastore and I stated testing writing files local datastore to iscsi datastore and using one VM I would get about 700mbit, if I add another VM and did both it pretty much doubled the writes. So I am leaing towards something is not setup properly. Its not ZFS, I know that for sure. I think i'm leaning towards the MPIO implementation. I have read the dummy guide, the dell best practices implementation and many posts, articles before posting this. So I feel its probably something very simple that I did not do that's probably not in the guides or howtos I read.

Maybe i'm under the wrong impression about MPIO. I seen lots of post where one VM can saturate a 2gbit MPIO iscsi. I know LACP is driven per connection or mac and you need multiple stream to reach 4gbit which is what I was doing over NFS. How Ever in Iscsi, I was under the impression that via MPIO and Round robin over 4 links 1 vm can use more than 1gbit of traffic over iscsi. Correct me if I am wrong please.

Case in point,

Last login: Wed Feb 4 22:23:17 2015 from nas.lan
[root@mail ~]# dd bs=512 count=4000000 if=/dev/zero of=test
4000000+0 records in
4000000+0 records out
2048000000 bytes (2.0 GB) copied, 38.7722 s, 52.8 MB/s
[root@mail ~]# dd bs=1M count=2048 if=/dev/zero of=test
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 37.9201 s, 56.6 MB/s
[root@mail ~]# dd bs=2M count=1024 if=/dev/zero of=test
1024+0 records in
1024+0 records out
2147483648 bytes (2.1 GB) copied, 37.5235 s, 57.2 MB/s
[root@mail ~]# dd bs=2M count=1024 if=/dev/zero of=test
1024+0 records in
1024+0 records out
2147483648 bytes (2.1 GB) copied, 30.8196 s, 69.7 MB/s
[root@mail ~]#

at the same time


[root@franciscodev ~]# dd bs=512 count=4000000 if=/dev/zero of=test 4000000+0 records in
4000000+0 records out
2048000000 bytes (2.0 GB) copied, 36.4352 s, 56.2 MB/s
[root@franciscodev ~]# dd bs=1M count=2048 if=/dev/zero of=test
2048+0 records in
2048+0 records out
2147483648 bytes (2.1 GB) copied, 37.0964 s, 57.9 MB/s
[root@franciscodev ~]# dd bs=2M count=1024 if=/dev/zero of=test
1024+0 records in
1024+0 records out
2147483648 bytes (2.1 GB) copied, 37.7834 s, 56.8 MB/s
[root@franciscodev ~]#




dT: 1.002s w: 1.000s
L(q) ops/s r/s kBps ms/r w/s kBps ms/w %busy Name
0 0 0 0 0.0 0 0 0.0 0.0| da0
0 0 0 0 0.0 0 0 0.0 0.0| da1
0 0 0 0 0.0 0 0 0.0 0.0| da2
0 0 0 0 0.0 0 0 0.0 0.0| da3
1 62 0 0 0.0 60 723 0.2 10.4| da4
0 61 1 4 17.6 57 719 0.2 16.9| da5
1 57 0 0 0.0 55 655 0.2 11.2| da6
0 64 0 0 0.0 61 687 0.2 15.9| da7
0 0 0 0 0.0 0 0 0.0 0.0| ada0
0 0 0 0 0.0 0 0 0.0 0.0| da0p1
0 0 0 0 0.0 0 0 0.0 0.0| da0p9
0 0 0 0 0.0 0 0 0.0 0.0| da1p1
0 0 0 0 0.0 0 0 0.0 0.0| da1p9
0 0 0 0 0.0 0 0 0.0 0.0| da2p1
0 0 0 0 0.0 0 0 0.0 0.0| da2p9
0 0 0 0 0.0 0 0 0.0 0.0| da3p1
0 0 0 0 0.0 0 0 0.0 0.0| da3p9
1 62 0 0 0.0 60 723 0.2 10.4| da4p1
0 0 0 0 0.0 0 0 0.0 0.0| da4p9
0 61 1 4 17.6 57 719 0.3 17.0| da5p1
0 0 0 0 0.0 0 0 0.0 0.0| da5p9
1 57 0 0 0.0 55 655 0.2 11.3| da6p1
0 0 0 0 0.0 0 0 0.0 0.0| da6p9
0 64 0 0 0.0 61 687 0.2 16.0| da7p1
0 0 0 0 0.0 0 0 0.0 0.0| da7p9
0 0 0 0 0.0 0 0 0.0 0.0| ada0p1
0 0 0 0 0.0 0 0 0.0 0.0| ada0p2

As you can see, its barely using the drives to its potential, ios are minimal yet performance over ISCSI is not scaling like it should.



Any input would be appreciated it.

and thank for taking the time to reply.


PS. Maybe i'm under the wrong impression about MPIO. I seen lots of post where one VM can saturate a 2gbit MPIO iscsi. I know LACP is driven per connection or mac and you need multiple stream to reach 4gbit which is what I was doing over NFS. How Ever in Iscsi, I was under the impression that via MPIO and Round robin over 4 links 1 vm can use more than 1gbit of traffic over iscsi. Correct me if I am wrong please. Pool is using lz4 compression.

 

[devops33]

Sorry forgot to mention.
[root@mail ~]# dd bs=2M count=1024 if=/dev/zero of=test
1024+0 records in
1024+0 records out
2147483648 bytes (2.1 GB) copied, 30.8196 s, 69.7 MB/s
[root@mail ~]# This test was by itsef.

Thanks..

 

[devops33]

Well Folks,

I am updating my thread so that if you are in the same path as I am, you can actually get some answers to the same questions I asked. Why Was I not getting the speed I wanted out of my setup. I was told that my setup would yield 60 MB to 120MB at max per second on my hardware and this information is not correct. Yes, there was a limitation on the hardware and it came down to IOPS but this is something that can be easily overcome. If your pool is not yielding the throughput you want over iscsi MPIO with 4 links dedicated to iscsi. What you would want to do is add a log/Zil raid 1 of good quality SSD's and make your iscsi storage on freenas is sync=always. Do this only in your iscsi storage. What this will do is that it will write the intent log into the zil raid1 of SSD's and since these have a higher IOPS than my pool of drives it speeds up dramatically your read and writes operations. For me it increased 4+ times the speed I had. It went from 70MB writes to 307 to 320MB writes depending on your sector setup and how you present the volume.

Here are some nice graphs and terminal output. By the way, this as using only one Host with one dd session. so I can only imagine how nicely this will perform with multiple hosts. I would probably reach 350MB but I dont think it would go any higher.

[root@kvm1 iscsi]# dd if=/dev/zero of=testfile bs=4096 count=599999
599999+0 records in
599999+0 records out
2457595904 bytes (2.5 GB) copied, 7.6862 s, 320 MB/s
I reduced the ram to 1GB on this test host just for this test.

 

[zambanini]

scnr: zfs porn.

 

[Redglobal]

Well Folks,

I am updating my thread so that if you are in the same path as I am, you can actually get some answers to the same questions I asked. Why Was I not getting the speed I wanted out of my setup. I was told that my setup would yield 60 MB to 120MB at max per second on my hardware and this information is not correct. Yes, there was a limitation on the hardware and it came down to IOPS but this is something that can be easily overcome. If your pool is not yielding the throughput you want over iscsi MPIO with 4 links dedicated to iscsi. What you would want to do is add a log/Zil raid 1 of good quality SSD's and make your iscsi storage on FreeNAS is sync=always. Do this only in your iscsi storage. What this will do is that it will write the intent log into the zil raid1 of SSD's and since these have a higher IOPS than my pool of drives it speeds up dramatically your read and writes operations. For me it increased 4+ times the speed I had. It went from 70MB writes to 307 to 320MB writes depending on your sector setup and how you present the volume.

Here are some nice graphs and terminal output. By the way, this as using only one Host with one dd session. so I can only imagine how nicely this will perform with multiple hosts. I would probably reach 350MB but I dont think it would go any higher.

[root@kvm1 iscsi]# dd if=/dev/zero of=testfile bs=4096 count=599999
599999+0 records in
599999+0 records out
2457595904 bytes (2.5 GB) copied, 7.6862 s, 320 MB/s
I reduced the ram to 1GB on this test host just for this test.






View attachment 6609

View attachment 6610

Hi,

Thanks so much for the info.. Could you please share the FreeNAS side MPIO/network settings that you have.. Some screen shots would be really helpful.

Best