Need help to understand slower read than write performance - iscsi/esx

[vctender]

Dear all,

we used an older server to build up a Freenas system to be used as ESX datastore over ISCSI.

Some hardware details:
- FreeNAS-11.3
- 2 CPUs (Xeon X5672 @ 3.20GHz)
- 32 GB ECC RAM
- IT Mode SAS Controller (No Hardware RAID)
- 12 Discs 7.2K (3 striped vdevs, each RAIDZ2 with 4 discs)
- Compression lz4, DeDup off, Pool usage < 20% (storage)
- ARC Hit ratio around 80%
- 4x GBit Ports, no LACP, MPIO used, Jumbo packets enabled on ESX + Freenas + Switch

I've used a virtual machine for some tests. The vm is stored on Freenas of course.
When writing 8 GByte with blocksize 1M we measured around 510 MByte/s. I'm satisfied with this value, altough I know this blocksize is more theoretical.
But reading those 8 GByte gives only around 190 MByte/s. "top" and "gstat" seems to be fine. We tested each network link in both directions with iperf3, all of them provided 990 MBit - which is fine.

I do not understand, why reading is much slower than writing. Can anyone explain this?. Because of the high ARC hit ratio, I would assume, that disk usage is quite slow.
Any chance to investigate the problem? I will try increasing RAM, but I would not expect more read performance by the current reported values. I appreciate any other hint to tweak performance.

Thanks in advance,
St

 

[vctender]

I've found the solution. For anybody experiencing the same issue:

The NAS integration (datastore) was new for ESX(i). As stated, we use MPIO via 4 links (Gbit) in round-robin mode for parallelized data transfers.
Because of the new integration, ESX(i) applied the default value (1000) for iops until link is changed. Usually using value 1 provides better performance with round-robin. Doing so, we measured now ~500 MB/s in both directions, as expected.

For anybody not knowing the esx shell command:

Code:

esxcli storage nmp device list -d naa.<naa-id>

esxcli storage nmp psp roundrobin deviceconfig set --type=iops --iops=1 --device=naa.<naa-id>

You have to replace <naa-id>with your values. You can find the id under ESX(i) Web Console -> Storage -> Devices