Bottleneck in iSCSI lans speed?

[newlink]

Hi,

I'm running a Freenas 9.2.18 based server (Supermicro 1027r-wrf, 16GB ECC ram, 4 128GB Samsung 850 pro SSD, 4 Gigabit lans) connected to an ESXI 5.5 host with a Raid10 full SSD storage.

DD speed tests on the storage are pretty impressive, the disapponting part is when I have to benchmark the ZFS volume. I configured a MPIO iSCSI portal in Freenas using all the 4 lans (I segmented my Procurve swith with 4 different subtnets VLANs) and I managed to saturate all the paths (VMware round robin) only when I adjusted the iSCSI IOPS setting to 1 from the default 1000 IOPS limit.

Now speed is pretty decent but it seems like I can't max out the Gigabit lan speed, limited to 95,3 Mb/s for each path (I'd expect about 125 Mb/s)

Is a bottleneck on the Freenas config side? I tried all setting, experimental iSCSI included, but I cound't sort it out, any advice from the experts would be very appreciated!

 

[cyberjock]

There is a penalty involved with using multipath. It scales, but not quite linearly (just like most things in IT). Personally, I'd consider 95MB/sec for each of 4 connections to be pretty good. You might want to try playing with the IO setting. Instead of 1000 maybe try 64 or 128 or maybe even going bigger like 2048.

 

[newlink]

Hi cyberjock,

thanks for the tips. Yesterday I ran other tests, using "fixed" and "most recently used" in VMware, both are single paths since I have just one target and one LUN. I had similar results, so it's not multipathing, MPIO balancing was poor only before tweaking with the IOPS limit in the iSCSI, 1 is the optimal setting for Freenas according to their knowledge base:

http://kb.vmware.com/selfservice/mi...nguage=en_US&cmd=displayKC&externalId=2069356

This morning I had a look at the VMware performance monitor and, surprise, realtime monitor shows a maximum of 111274 KBps (111,2 Mb/s) so each lan is performing extremely well. If you look at the graph I didn't make my homeworks, it show a max RX of 907.5 million of bits per second (113,4 Mb/s).

I definitely have a bottleneck but it's because 4 gigabit lans are not enough (a single SSD can reach 600 Mb/s read speed, a raid10 should double that score...).

Do you think that I need a 10 gigabit lan to keep up with the raid reading/writing speed? A dual port 10 gigabit lan (I need to connect 2 VMware hosts) is very expensive, about 650 euros, and at least the same price for a couple of single 10 gigabit lans for the hosts, so I need to be sure that are not wasted money

 

[mav@]

Whether you need 10GigE probably depends on your workload requirements. I can only say that from my own experience on powerful enough hardware the new iSCSI target is able to saturate 2x10Gbps network links at almost any request sizes, assuming your backing storage is able to handle that too.

 

[cyberjock]

I can't really say if 10Gb would help or not. You are running SSD, but 16GB of RAM is extremely small for someone that is trying to use iSCSI and having high expectations. Before I'd even consider 10Gb I'd go to at least 32GB of RAM, and 64GB of RAM would probably be an even better idea. Also for VM workloads, you are probably going to want an L2ARC, and that almost universally also means 'I got 64GB of RAM or more'. In fact, if you add an L2ARC and don't have enough RAM you can actually make your server slower.

As you can see, it's totally possible to do what you want with just 16GB of RAM. But as the manual, my noobie guide, and about 500 threads on this forum will attest to, if ZFS performance is slow the one single thing you can do to speed it up is add more RAM. Yes, SSDs are fast. But if they are having to read lots of data into the ARC only to be evicted before you can use it because of the ARC you're like a race car that's idling at high RPM at the starting line. Going nowhere and spending huge amounts of resources getting there.

Can you post a debug file on your server?

 

[HoneyBadger]

@cyberjock, his entire back-end is SSD, I don't think L2ARC is necessary in this case.

32GB is a good place to start.

 

[cyberjock]

@cyberjock, his entire back-end is SSD, I don't think L2ARC is necessary in this case.

32GB is a good place to start.

Actually, there are benefits to an L2ARC even in a full-SSD array. I'm not privvy to first-hand experience but apparently pulling data into the ARC incurs a penalty that is greater than retrieving data from an L2ARC. Of course, this is assuming the L2ARC isn't overloaded and is actually of appropriate speed to be an L2ARC.

Case and point, look at the OP. He's got 16GB of RAM and is having performance problems. ;)

 

[HoneyBadger]

There's the obvious case of having less contention on the underlying vdevs since your most frequent reads come from L2ARC and don't have to fight with any concurrent IOPS to the vdevs. But I'd argue that the money needed for an L2ARC that could add value to a 4-drive RAID10 SSD setup would be better spent on a big stack of RAM. In that config I'll take 64GB/no L2ARC over 32GB/L2ARC.

 

[cyberjock]

You don't need to argue. I agree. That's why I said:

Also for VM workloads, you are probably going to want an L2ARC, and that almost universally also means 'I got 64GB of RAM or more'. In fact, if you add an L2ARC and don't have enough RAM you can actually make your server slower.

 

[HoneyBadger]

Obviously the best solution is 64GB + L2ARC. ;)

Though that board does handle up to 1TB ... thoughtfully strokes goatee ...

 

[newlink]

mav@ I had at look at the supported 10 Gbit lans, but the ones I could find from my dealer are based on the Intel X540-T2 chipset that seems to have issues with the lastest version of Freenas:

https://bugs.freenas.org/issues/4560

Any suggestion about good ones that you tested?

 

[cyberjock]

I just talked to the lead developer for FreeNAS. The official answer is all bugs related to the Intel 10Gb NICs *should* be resolved in 9.2.1.8. They are unaware of any problems that exist that are currently outstanding for Intel 10Gb NICs and they should be safe and reliable to use.

 

[newlink]

I can't really say if 10Gb would help or not. You are running SSD, but 16GB of RAM is extremely small for someone that is trying to use iSCSI and having high expectations. Before I'd even consider 10Gb I'd go to at least 32GB of RAM, and 64GB of RAM would probably be an even better idea. Also for VM workloads, you are probably going to want an L2ARC, and that almost universally also means 'I got 64GB of RAM or more'. In fact, if you add an L2ARC and don't have enough RAM you can actually make your server slower.

As you can see, it's totally possible to do what you want with just 16GB of RAM. But as the manual, my noobie guide, and about 500 threads on this forum will attest to, if ZFS performance is slow the one single thing you can do to speed it up is add more RAM. Yes, SSDs are fast. But if they are having to read lots of data into the ARC only to be evicted before you can use it because of the ARC you're like a race car that's idling at high RPM at the starting line. Going nowhere and spending huge amounts of resources getting there.

Can you post a debug file on your server?

From the tests I ran I can say that I have issues with small files, lets say under 4Mb and smaller. Probably it's because the extra redundancy of the raid10 array that adds some unwanted latency, I'd like to test a raidz to see if I can have better scores when I have to deal with 4K and small files in general. At the other side, sequential read and write are very good, I can symmetrically reach 450 Mb/s, so basically I saturate the iSCSI paths when I read or write a file bigger than 4Mb.

I can upgrade to 32GB and see what happens, it's a good idea, while I have no experience with L2ARC caches. I'd like to try it too, do you suggest two mirrored SSD disks?

Here you are, I attached the debug file, I'm curious to know what you think about the server

 

[cyberjock]

Small files are always slow. If that's your only problem then I'd say everything is just fine. Small files hinder throughput because of all of the I/O generated on the source and destination plus network protocol overhead. There is no fix for dealing with small files except to "suck it up buttercup".

If you are happy with the performance except for small files you are probably not going to see much of an improvement with more RAM or anything else. Personally I'd upgrade to 32GB of RAM because it's iSCSI and it won't hurt. But whether you want to spend the $150 or so on RAM is something you'll have to decide on.

 

[HoneyBadger]

Any kind of RAIDZ will inherently have worse latency than a RAID10, so you can skip that test entirely.

L2ARC is best added as a stripe, since it will just go to the pool if it loses a device. But heed cyberjock's warning and bump up the RAM to 64GB first, because L2ARC consumes RAM.

 

[newlink]

@cyberjock thank you, if I'm not mistaken the chipset is the 82599ES, I had some concerns because I read that only 82598EB is fully supported, not sure if they share the same driver.

Of course, I was just a little upset because the same Samsung SSD, attached to his sata3 controller, has a 50 Mb/s transfer speed rate in both read and write tests with a 512k file, while I have a poor 8,6 Mb/s. With 8Mb and bigger I'm pretty close so I can't complain considering the iSCSI i/o and the lans latency

@HoneyBadger Do you think that 32GB are not enough? So if one of the striped disks fails in the L2ARC, I won't lose data? Cool

 

[cyberjock]

As my noobie presentation says, L2ARCs shouldn't be mirrored normally. If an L2ARC fails it just means the performance benefits are lost. On the other hand, if a ZIL fails at an inopportune time you can lose data. ;)

 

[mjws00]

I just talked to the lead developer for FreeNAS. The official answer is all bugs related to the Intel 10Gb NICs *should* be resolved in 9.2.1.8. They are unaware of any problems that exist that are currently outstanding for Intel 10Gb NICs and they should be safe and reliable to use.

This is VERY good news. Thought we might be waiting until 9.3.

 

[cyberjock]

I thought so too. Went to the horse's mouth. :D

 

[Ericloewe]

This is VERY good news. Thought we might be waiting until 9.3.

Very, very good news! It brings 10GbE down to rather expensive from ludicrously expensive.
It also simplifies advice-giving - since Intel supposedly stopped making motherboards and their 10GbE is now reliable, I can just say "You can't go wrong with Intel".