iSCSI performance low

[KMR]

Hey folks,

I have been playing with my FreeNAS box for a little while now and am quite happy. I have added some new equipment and am playing around with iSCSI as VM storage for my ESXi whitebox. I bought a new 16 port switch that supports jumbo frames, LACP, etc, as well as two more dual port Intel server NICs. I have installed one NIC in the FreeNAS box and one in the ESXi box so that I can have two dedicated gigabit ports for iSCSI traffic. I have created a file extent on my existing pool for testing purposes (I have an IBM M1015 on order so I can create a dedicated RAID 10 pool from 4 1TB drives I have laying around) which is connected to my ESXi box. ESXi is configured for round robin and both paths are active and being used. My issue is that when I transfer VMs between the local SSD datastore on the ESXi box and the file extent on the FreeNAS box I am only seeing about 200 - 400mbps across both interfaces, or 100 - 200mbps per interface. I know that I won't get 2gbps but this is much lower than I was expecting. My disk subsystem isn't the issue as I am routinely able to get 950mbps with CIFS on my other NICs.

I know there are a couple of different ways to configure ESXi for iSCSI MPIO so I was wondering if anyone could give me some pointers on how they configured their systems. Please feel free to ask config questions, I'm just not sure what would be relevant here.

Thanks,

 

[jgreco]

It would seem logical for you to describe the sort of throughput you got without the MPIO complications, so we can more effectively figure out what is affecting you.

 

[KMR]

With only one path active (before changing it to round robin) I was getting about 600mbps throughput on a single NIC. Although this seems low as well given that I am transferring too and from an SSD.

 

[KMR]

My previous tests had basically been moving VM's between datastores which resulted in low performance numbers. I tried creating a vmdk on the iSCSI datastore and attaching it to a VM and transferring a .mkv file between disks within the VM. The VM itself is stored on the local SSD so this would essentially be the same test but more realistic. This resulted in more acceptable results of 120MB/s or so. Checking the reporting page in FreeNAS shows that the load is being distributed across both iscsi interfaces although not evenly.

These are back of the envelope tests because I am still getting my head around analyzing iostat and esxtop output.
Transferring a 5GB file from iSCSI extent to local SSD yields about 125MB/s on average.
Transferring a 5GB file from local SSD to iSCSI extent yields different results. Windows reporting shows 400+MB/s at the beginning of the transfer which tapers steadily down to about 90MB/s when the transfer completes. Obviously something is off here as 400 MB/s isn't even possible for two gigabit NICs.

I'll do some more digging. I may try to remove MPIO to get some perspective. Honestly I bought the used Dell X3959 NICs (I love these things. This makes 4 in total I have) and configured MPIO right away. Might be a good idea to eliminate some of the extra complexity. I also plan on playing with the round robin settings in ESXi as well as read more on ZFS sync writes.

 

[KMR]

Sample IOSTAT -XD 5 output of transfer from iSCSI to SSD:

Code:

                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
md0        0.0   0.3     0.0     0.0    0   0.0   0
md1        0.0   0.0     0.0     0.0    0   0.0   0
md2        0.0   6.4     0.0     0.0    0   0.2   0
md3        0.0   0.1     0.0     0.0    0   0.1   0
ada0      25.7  18.5  2468.4  1554.7    1   5.0   3
ada1      26.1  18.6  2450.3  1557.4    0   4.9   3
ada2      28.6  18.5  2697.5  1554.8    0   5.0   3
ada3      29.8  18.5  2685.1  1554.8    1   4.8   3
ada4      28.7  18.5  2566.0  1554.7    1   4.8   3
ada5      24.3  18.6  2340.5  1557.3    1   5.1   3
da0        0.4   0.0     7.8     0.3    0   2.4   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
pass4      0.0   0.0     0.0     0.0    0   0.0   0
pass5      0.0   0.0     0.0     0.0    0   0.0   0
pass6      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
md0        0.0   0.0     0.0     0.0    0   0.0   0
md1        0.0   0.0     0.0     0.0    0   0.0   0
md2        0.0   0.8     0.0     0.0    0   0.0   0
md3        0.0   0.0     0.0     0.0    0   0.0   0
ada0     361.0 117.2 11563.3 11928.8    0   1.8  25
ada1       3.6 119.6   115.2 11960.8    0   4.8   7
ada2       3.6 119.2   115.2 11959.2    0   5.8   8
ada3     364.6 118.4 11678.5 11957.6    0   2.0  27
ada4     364.6 117.6 11678.5 11926.4    0   1.8  26
ada5     361.0 117.6 11563.3 11953.6    0   1.8  28
da0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
pass4      0.0   0.0     0.0     0.0    0   0.0   0
pass5      0.0   0.0     0.0     0.0    0   0.0   0
pass6      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
md0        0.0   0.0     0.0     0.0    0   0.0   0
md1        0.0   0.0     0.0     0.0    0   0.0   0
md2        0.0   1.2     0.0     0.0    0   0.0   0
md3        0.0   0.0     0.0     0.0    0   0.0   0
ada0       0.0 252.8     0.0 30443.6    0   7.4  19
ada1       0.0 252.6     0.0 30454.7    0   7.4  19
ada2       0.0 252.0     0.0 30438.0    0   7.6  20
ada3       0.0 252.0     0.0 30438.0    0   7.6  20
ada4       0.0 252.8     0.0 30443.6    0   7.7  20
ada5       0.0 253.0     0.0 30461.9    0   7.6  20
da0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
pass4      0.0   0.0     0.0     0.0    0   0.0   0
pass5      0.0   0.0     0.0     0.0    0   0.0   0
pass6      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
md0        0.0   0.0     0.0     0.0    0   0.0   0
md1        0.0   0.0     0.0     0.0    0   0.0   0
md2        0.0   0.6     0.0     0.0    0   0.0   0
md3        0.0   0.0     0.0     0.0    0   0.0   0
ada0       0.0 261.4     0.0 31640.5    0   7.8  21
ada1       0.0 262.2     0.0 31658.1    0   7.5  20
ada2       0.0 262.0     0.0 31654.9    0   7.7  21
ada3       0.0 262.0     0.0 31654.9    0   7.6  20
ada4       0.0 261.4     0.0 31640.5    0   7.4  20
ada5       0.0 262.0     0.0 31658.9    0   7.8  21
da0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
pass4      0.0   0.0     0.0     0.0    0   0.0   0
pass5      0.0   0.0     0.0     0.0    0   0.0   0
pass6      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
md0        0.0   0.8     0.0     0.0    0   0.0   0
md1        0.0   0.0     0.0     0.0    0   0.0   0
md2        0.0   3.2     0.0     0.0    0   0.0   0
md3        0.0   0.0     0.0     0.0    0   0.0   0
ada0       0.0 257.6     0.0 31843.9    0   7.6  20
ada1       0.0 258.2     0.0 31865.5    0   7.7  20
ada2       0.0 258.0     0.0 31859.9    0   7.8  21
ada3       0.0 258.0     0.0 31859.9    0   7.8  21
ada4       0.0 257.6     0.0 31843.9    0   7.9  21
ada5       0.0 258.2     0.0 31869.5    0   7.9  21
da0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
pass4      0.0   0.0     0.0     0.0    0   0.0   0
pass5      0.0   0.0     0.0     0.0    0   0.0   0
pass6      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
md0        0.0   0.0     0.0     0.0    0   0.0   0
md1        0.0   0.0     0.0     0.0    0   0.0   0
md2        0.0  23.8     0.0     0.0    0   0.1   0
md3        0.0   0.0     0.0     0.0    0   0.0   0
ada0       0.0 251.7     0.0 31090.9    0   8.1  21
ada1       0.0 252.1     0.0 31109.3    0   7.7  20
ada2       0.0 252.1     0.0 31102.9    0   7.6  20
ada3       0.0 252.1     0.0 31102.9    0   7.9  20
ada4       0.0 251.7     0.0 31090.9    0   7.6  20
ada5       0.0 252.3     0.0 31108.5    0   7.9  21
da0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
pass4      0.0   0.0     0.0     0.0    0   0.0   0
pass5      0.0   0.0     0.0     0.0    0   0.0   0
pass6      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s wait svc_t  %b
md0        0.0   0.0     0.0     0.0    0   0.0   0
md1        0.0   0.0     0.0     0.0    0   0.0   0
md2        0.0   4.0     0.0     0.0    0   0.0   0
md3        0.0   0.0     0.0     0.0    0   0.0   0
ada0       0.0 262.5     0.0 31769.3    0   7.4  20
ada1       0.0 262.5     0.0 31776.5    0   7.4  20
ada2       0.0 261.7     0.0 31765.3    0   7.3  20
ada3       0.0 261.5     0.0 31765.3    0   7.5  20
ada4       0.0 262.3     0.0 31769.3    0   7.7  21
ada5       0.0 262.5     0.0 31777.3    0   7.7  21
da0        0.0   0.0     0.0     0.0    0   0.0   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
pass4      0.0   0.0     0.0     0.0    0   0.0   0
pass5      0.0   0.0     0.0     0.0    0   0.0   0
pass6      0.0   0.0     0.0     0.0    0   0.0   0

Sample IOSTAT -XD 5 output of transfer from SSD to iSCSI:

Code:

device    r/s  w/s    kr/s    kw/s wait svc_t  %b
md0        0.0  0.0    0.0    0.0    0  0.0  0
md1        0.0  0.0    0.0    0.0    0  0.0  0
md2        0.0  25.4    0.0    0.0    0  0.1  0
md3        0.0  0.0    0.0    0.0    0  0.0  0
ada0      0.0  19.4    0.0  1597.3    0  4.3  1
ada1      0.0  19.6    0.0  1598.1    0  4.8  1
ada2      0.0  19.6    0.0  1596.5    0  5.7  1
ada3      0.0  19.6    0.0  1596.5    0  5.1  1
ada4      0.0  19.4    0.0  1597.3    0  5.4  1
ada5      0.0  19.4    0.0  1600.5    0  4.9  1
da0        0.0  0.0    0.0    0.0    0  0.0  0
pass0      0.0  0.0    0.0    0.0    0  0.0  0
pass1      0.0  0.0    0.0    0.0    0  0.0  0
pass2      0.0  0.0    0.0    0.0    0  0.0  0
pass3      0.0  0.0    0.0    0.0    0  0.0  0
pass4      0.0  0.0    0.0    0.0    0  0.0  0
pass5      0.0  0.0    0.0    0.0    0  0.0  0
pass6      0.0  0.0    0.0    0.0    0  0.0  0
                        extended device statistics
device    r/s  w/s    kr/s    kw/s wait svc_t  %b
md0        0.0  0.0    0.0    0.0    0  0.0  0
md1        0.0  0.0    0.0    0.0    0  0.0  0
md2        0.0  3.6    0.0    0.0    0  0.0  0
md3        0.0  0.0    0.0    0.0    0  0.0  0
ada0      0.0  14.6    0.0  142.4    0  0.6  0
ada1      0.0  14.6    0.0  139.2    0  0.6  0
ada2      0.0  14.6    0.0  136.8    0  0.7  0
ada3      0.0  14.6    0.0  136.8    0  0.6  0
ada4      0.0  14.6    0.0  142.4    0  0.6  0
ada5      0.0  14.6    0.0  149.6    0  0.6  0
da0        0.0  0.0    0.0    0.0    0  0.0  0
pass0      0.0  0.0    0.0    0.0    0  0.0  0
pass1      0.0  0.0    0.0    0.0    0  0.0  0
pass2      0.0  0.0    0.0    0.0    0  0.0  0
pass3      0.0  0.0    0.0    0.0    0  0.0  0
pass4      0.0  0.0    0.0    0.0    0  0.0  0
pass5      0.0  0.0    0.0    0.0    0  0.0  0
pass6      0.0  0.0    0.0    0.0    0  0.0  0
                        extended device statistics
device    r/s  w/s    kr/s    kw/s wait svc_t  %b
md0        0.0  0.0    0.0    0.0    0  0.0  0
md1        0.0  0.0    0.0    0.0    0  0.0  0
md2        0.0  1.2    0.0    0.0    0  0.0  0
md3        0.0  0.0    0.0    0.0    0  0.0  0
ada0      0.0  8.0    0.0    63.2    0  0.8  0
ada1      0.0  8.4    0.0    72.0    0  0.8  0
ada2      0.0  8.2    0.0    66.4    0  0.8  0
ada3      0.0  8.6    0.0    65.6    0  0.8  0
ada4      0.0  8.4    0.0    62.4    0  0.8  0
ada5      0.0  8.8    0.0    64.8    0  0.9  0
da0        0.0  0.0    0.0    0.0    0  0.0  0
pass0      0.0  0.0    0.0    0.0    0  0.0  0
pass1      0.0  0.0    0.0    0.0    0  0.0  0
pass2      0.0  0.0    0.0    0.0    0  0.0  0
pass3      0.0  0.0    0.0    0.0    0  0.0  0
pass4      0.0  0.0    0.0    0.0    0  0.0  0
pass5      0.0  0.0    0.0    0.0    0  0.0  0
pass6      0.0  0.0    0.0    0.0    0  0.0  0
                        extended device statistics
device    r/s  w/s    kr/s    kw/s wait svc_t  %b
md0        0.0  0.0    0.0    0.0    0  0.0  0
md1        0.0  0.0    0.0    0.0    0  0.0  0
md2        0.0  1.4    0.0    0.0    0  0.0  0
md3        0.0  0.0    0.0    0.0    0  0.0  0
ada0      0.0  7.8    0.0    65.6    0  0.5  0
ada1      0.0  7.6    0.0    68.0    0  0.5  0
ada2      0.0  7.6    0.0    64.8    0  0.5  0
ada3      0.0  7.6    0.0    64.8    0  0.5  0
ada4      0.0  7.8    0.0    65.6    0  0.5  0
ada5      0.0  7.8    0.0    67.2    0  0.5  0
da0        0.0  0.0    0.0    0.0    0  0.0  0
pass0      0.0  0.0    0.0    0.0    0  0.0  0
pass1      0.0  0.0    0.0    0.0    0  0.0  0
pass2      0.0  0.0    0.0    0.0    0  0.0  0
pass3      0.0  0.0    0.0    0.0    0  0.0  0
pass4      0.0  0.0    0.0    0.0    0  0.0  0
pass5      0.0  0.0    0.0    0.0    0  0.0  0
pass6      0.0  0.0    0.0    0.0    0  0.0  0
 

 

[Setius]

My home lab worked the best with ESXi MPIO RR set to 1 iops. I did most of my testing with HDTune, crystaldiskmark and ATTO from a Win 2k8 VM on the iSCSI datastore. Must of wasted hours trying different RR iops values and benchmarking.