ESX / iSCSI - Improve 4k Read / Writes, Latency & Optimize for large reads

[msignor]

The speed you are talking about, "80/90 MB/sec", is much slower than it could be if you didn't have those 12 disk laid out as a single large RAID-z2 vdev. I have 12 disks in both of my systems and laid them out as two vdevs for the purpose of greater speed. On my 10GB network they will transfer (large files) around 550MB/s and they completely max the wire speed on the 1GB network. Many small files slow everything down, even when you have high capacity for sequential transfer, small files kill the performance. So, the thing that may be causing you more trouble than you realize is the kind of file access you are dealing with. I know you said it is a big file, but you also said the goal is to search for data inside the file and that changes things because it isn't like just moving one big file, which would be fast, it is more the kind of work of accessing many small files.
Like @Stux said above, you might need to put the data for this on SSD or a bunch of mirrors like I suggested to get reasonable performance.
Also a SLOG device might help. Your current hardware is just not going to do it, not without making some changes.

Thanks. Appreciate the detail.

One last thing came to mind.. A year ago, I set my TCP control algorithm to 'cubic' to handle better wifi speeds. Is there a chance that this is messing up my iSCSI? ( https://forums.freenas.org/index.ph...very-slow-only-to-freenas-sanity-check.43811/ )

 

[msignor]

OK, so the fact we have established that sync=disabled works for you, you might be a perfect candidate for a SLOG device. While the pool you have set-up is sub-optimal (mirrored vdevs will give you better random IO), you can at least improve it with a SLOG.

A SLOG device however cannot be any old SSD. Ideally, it must have power loss protection in order to safeguard your data from (you've probably guessed already), power loss. I suggest you take a look at Intel's SSD offerings as some of theirs have this feature.

Hi - For giggles I got a 6TB external USB3 drive, plugged it into the ESX host, then my VM, and then was able to perform the same writes and reads at 60MB+/sec no problem to one disk. Is this horrible iscsi performance I am seeing literally due to the ZVOL?! Again, I assume why things are "fast" after a reboot over iscsi is due to the RAM? and that would be the SLOG?

 

[Chris Moore]

Hi - For giggles I got a 6TB external USB3 drive, plugged it into the ESX host, then my VM, and then was able to perform the same writes and reads at 60MB+/sec no problem to one disk. Is this horrible iscsi performance I am seeing literally due to the ZVOL?! Again, I assume why things are "fast" after a reboot over iscsi is due to the RAM? and that would be the SLOG?

I understand, you keep asking the question because you don't like the answer and you are hoping someone will give you a different one.

One last thing came to mind.. A year ago, I set my TCP control algorithm to 'cubic' to handle better wifi speeds. Is there a chance that this is messing up my iSCSI?

That could be causing problems for you, you could try changing it to see if there is a measurable change.

Here is an online calculator that is designed to estimate the performance of a RAID set. The results are not exactly the same for ZFS, but because you can do a side by side comparison of two different configurations, it will help you understand how different configurations affect performance.

http://wintelguy.com/raidperf2.pl