FreeNAS 9.2.0 NFS to ESXi 4.1

[gamedude9742]

Hello everyone,
I'm brand new to the forums and have been reading on here for a while and also google as well for help with my setup.

Here are the specs...
FreeNAS is installed on
SuperMicro X7DBT
2x Intel Xeon E5420 @ 2.5GHz
32GB of RAM
3x 2TB 7200RPM HD's (standard desktop models, not NAS or green, etc.)
1x 1 TB 7200 RPM HD (standard desktop model)

The 3x 2TB drives are configured in a RAIDZ on ZFS and shared via NFS to my ESXi host. I have been noticing lots of performance issues from the initial setup. I have sync writes disabled as I have no ZIL device. The writes are becoming painful again though. I see transfer speeds of around 20MBps on a full gigabit network.

In previous testing I have been able to get very good speeds by connecting to the storage from a Windows host and transferring test ISO's or MP4's, etc.. Usually in the ballpark of 70+MBps

So I guess what I'm really asking is if someone has done a similar setup and has any tips for me. I have read a lot of the guides online about this setup and no-one seems to do anything "special" like tuning NFS settings or anything like that and seems to get much better results than I do out of the box. I'd like to think that lack of hardware isn't the problem.

Please help! Thanks very much for reading.

 

[mav@]

For the first, NFS server got many performance improvements last time, so you may wish to update your FreeNAS to the latest release.

After that, could you take a look on what is going on with your disks while you are doing your writes? For example with `gstat -I 1s` or `iostat -zxw1`. My personal guess is that you may suffer from some Copy-on-Write issues. If your writes are not aligned to ZFS records, then writes may trigger extra reads and wait for them. In such case above tools should show you large number of read operations during your writes. It could not happen when the VM was new and so its file had many holes reads from which could be skipped.

 

[gamedude9742]

I'm currently copying information from my storage server to another PC on my network. My PC and storage server are on the same gigabit network, the storage server is virutalized on ESXi with its storage presented from FreeNAS over NFS. I ran the recommended commands as I am only getting read speeds of around 20 MBps now. Here are the results.

Code:

gstat -I 1s
dT: 1.064s  w: 1.000s
L(q)  ops/s    r/s   kBps   ms/r    w/s   kBps   ms/w   %busy Name
    0     61     18   1985    4.1     41    395    0.2   14.5| ada0
    0      0      0      0    0.0      0      0    0.0    0.0| ada0p1
    0     61     18   1985    4.1     41    395    0.2   14.5| ada0p2
    0     57     13   1263    6.5     42    391    0.1   13.9| ada1
    0     56     14   1323   23.1     40    380    0.1   21.6| ada2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3
    0      0      0      0    0.0      0      0    0.0    0.0| ada1p1.eli
    0     61     18   1985    4.1     41    395    0.3   14.6| gptid/d3d440a4-18
1f-11e4-ade0-0030487de0ba
    0      0      0      0    0.0      0      0    0.0    0.0| ada1p1
    0     57     13   1263    6.5     42    391    0.1   14.0| ada1p2
    0      0      0      0    0.0      0      0    0.0    0.0| ada2p1
    0     56     14   1323   23.1     40    380    0.1   21.6| ada2p2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3p1
    0      0      0      0    0.0      0      0    0.0    0.0| ada3p2
    0      0      0      0    0.0      0      0    0.0    0.0| ada3p1.eli
    0     57     13   1263    6.5     42    391    0.1   14.0| gptid/d3ff0455-18
1f-11e4-ade0-0030487de0ba
    0     56     14   1323   23.1     40    380    0.1   21.6| gptid/d42e4093-18
1f-11e4-ade0-0030487de0ba
    0      0      0      0    0.0      0      0    0.0    0.0| ada2p1.eli

Code:

iostat -zxw1    0      0    0.0    0.0| gptid/e5fb3b1d-1
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md0        0.0   0.1     0.0     0.0    0   0.1   0
md2        0.0   4.7     0.0     0.0    0   0.0   0
ada0       2.6   5.4   261.1    51.3    0   1.3   0
ada1       2.5   5.7   261.4    51.2    0   2.0   1
ada2       1.9   6.8   208.7   173.9    0   3.3   1
ada3       0.0   0.0     0.1     0.0    0   1.4   0
da0        0.1   0.0     0.8     0.4    0  15.6   0
pass0      0.0   0.0     0.0     0.0    0   0.0   0
pass1      0.0   0.0     0.0     0.0    0   0.0   0
pass2      0.0   0.0     0.0     0.0    0   0.0   0
pass3      0.0   0.0     0.0     0.0    0   0.0   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0  22.9     0.0     0.0    0   0.1   0
ada0      22.9   0.0  1976.3     0.0    0   5.7   5
ada1      10.9   0.0   827.1     0.0    0  10.0   4
ada2      14.9   0.0  1149.2     0.0    0  17.2  12
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0   1.0     0.0     0.0    0   0.0   0
ada0      81.9   0.0  9079.7     0.0    2   4.1  11
ada1      39.9   0.0  4030.5     0.0    3  11.0  15
ada2      37.9   0.0  3459.3     0.0    3  14.0  15
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      98.4   0.0 10525.4     0.0    0   5.1  16
ada1     127.0   0.0 14120.4     0.0    0  11.6  47
ada2     197.9   0.0 24826.9     0.0    0   8.1  53
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md0        0.0   1.0     0.0     0.0    0   0.0   0
md2        0.0   1.0     0.0     0.0    0   0.0   0
ada0       0.0  44.8     0.0   381.7    0   0.2   1
ada1       0.0  38.9     0.0   405.1    0   0.1   0
ada2       0.0  44.8     0.0   405.1    0   0.1   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      69.2   0.0  8357.8     0.0    2   2.7   7
ada1      44.5   0.0  4501.3     0.0    1   9.4  12
ada2      44.5   0.0  4750.5     0.0    1  14.0  17
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0 239.7     0.0     0.0    0   0.0   0
ada0      41.3   0.0  4464.9     0.0    0   4.6   7
ada1      63.9   0.0  7672.0     0.0    0   6.7  17
ada2      43.2   0.0  4590.6     0.0    0  13.0  19
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      54.9   0.0  5250.8     0.0    0   4.6  10
ada1      53.9   0.0  5502.5     0.0    0   8.3  14
ada2      72.9   0.0  8635.4     0.0    3  19.5  44
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada2      16.4   0.0  2032.5     0.0    0   7.5   4
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0  10.0     0.0     0.0    0   0.1   0
ada0      93.9  40.0 10111.0   399.6    0   5.4  25
ada1     110.9  36.0 12481.0   391.7    0   6.3  31
ada2     139.9  36.0 17076.9   403.6    0   9.6  48
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0       0.0   4.0     0.0    16.0    0   0.4   0
ada1       0.0   4.0     0.0    16.0    0   0.2   0
ada2       0.0   4.0     0.0    16.0    0   0.2   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0  25.0     0.0     0.0    0   0.1   0
ada0      86.9   0.0  9086.9     0.0    0   3.8  10
ada1     102.9   0.0 12024.0     0.0    0   5.9  18
ada2      71.9   0.0  7548.5     0.0    0   9.8  23
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0     108.9   0.0 11895.8     0.0    3   3.6  14
ada1      85.9   0.0  9018.8     0.0    3  11.0  33
ada2     137.9   0.0 16567.0     0.0    3  14.0  67
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      28.8   0.0  3199.5     0.0    0   4.2   4
ada1      48.1   0.0  5599.1     0.0    0   5.3   9
ada2      38.5   0.0  4122.4     0.0    0  17.1  21
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0       0.0  42.9     0.0   396.6    0   0.2   1
ada1       0.0  43.9     0.0   369.9    0   0.1   0
ada2       0.0  37.2     0.0   396.6    0   0.1   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0 151.9     0.0     0.0    0   0.0   0
ada0      81.7   0.0  8938.3     0.0    0   5.5  17
ada1      72.2   0.0  7479.6     0.0    0   6.8  17
ada2      95.9   0.0 10826.3     0.0    0   9.8  30
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      85.2   0.0  8939.3     0.0    0   4.9  15
ada1      82.2   0.0  8373.0     0.0    1  15.7  39
ada2     141.6   0.0 16924.1     0.0    3  15.4  65
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada1       0.9   0.0    60.2     0.0    0  26.6   2
ada2       3.8   0.0   421.3     0.0    0  29.4   3
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      94.2   0.0 10758.4     0.0    0   2.8   9
ada1      79.6   0.0  8773.0     0.0    0  11.2  27
ada2      77.7   0.0  8334.0     0.0    0  10.7  25
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0   7.5     0.0     0.0    0   0.0   0
ada0      72.4  41.4  7710.5   394.9    0   2.9  13
ada1      70.5  45.1  7706.7   391.2    0   7.3  26
ada2     125.1  37.6 15409.7   383.6    0   5.0  25
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0  15.4     0.0     0.0    0   0.1   0
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      86.5   0.0  9335.3     0.0    0   4.6  13
ada1      86.5   0.0  9809.2     0.0    0   6.1  18
ada2     102.5   0.0 11682.3     0.0    0  12.8  36
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      77.9  37.0  8188.4   419.6    0   3.3  13
ada1      75.9  47.0  8188.4   415.6    0   7.3  27
ada2     133.9  47.0 16376.7   387.6    0   5.5  28
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0  57.4     0.0     0.0    0   0.0   0
ada0      13.2   0.0   789.8     0.0    0   7.8   6
ada1      16.0   0.0  1387.8     0.0    0   7.1   5
ada2      11.3   0.0   725.9     0.0    0  19.4   8
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      25.6   0.0  2454.3     0.0    3   7.0   6
ada1      22.6   0.0  2328.4     0.0    2   8.4   7
ada2      18.7   0.0  1762.1     0.0    1  27.4  18
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      17.9   0.0  1866.4     0.0    0   3.4   2
ada1      18.8   0.0  1806.2     0.0    0  11.0   7
ada2      27.3   0.0  2950.1     0.0    0   5.4   5
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0     133.0   0.0 14603.3     0.0    0   4.2  19
ada1      76.7   0.0  7522.9     0.0    3  16.0  43
ada2     118.4   0.0 13858.0     0.0    3  10.0  41
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0   4.9     0.0     0.0    0   0.0   0
ada0      41.0  38.0  4370.5   398.0    0   1.5   5
ada1      90.7  40.0  9993.6   409.7    0   7.4  28
ada2      85.8  41.0  9556.5   405.8    0   8.4  35
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      80.8   0.0  8749.8     0.0    0   3.3  10
ada1      79.8   0.0  8558.3     0.0    0   7.8  19
ada2      99.7   0.0 11307.3     0.0    0  10.2  30
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0   7.0     0.0     0.0    0   0.0   0
ada0      35.0   0.0  3649.7     0.0    4   2.9   4
ada1      29.0   0.0  3074.1     0.0    4   6.5   7
ada2      16.0   0.0  1407.1     0.0    3   7.6   5
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      59.9   0.0  5789.5     0.0    0   4.6   9
ada1      68.8   0.0  7235.9     0.0    0   9.4  19
ada2      96.3   0.0 11193.9     0.0    0  24.7  79
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      92.4   0.0 10057.6     0.0    0   6.2  18
ada1      89.4   0.0 10244.5     0.0    0   7.6  23
ada2     103.4   0.0 12279.9     0.0    0  13.6  47
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      19.0  43.9  1600.4   399.1    3   2.0   7
ada1      23.9  36.9  2115.3   419.1    3   3.1   8
ada2      17.0  47.9  1472.7   407.1    3   3.6   8
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0   9.6     0.0     0.0    0   0.0   0
ada0      55.9   0.0  6290.8     0.0    0   4.8   8
ada1      58.8   0.0  6475.8     0.0    0  11.4  21
ada2     110.8   0.0 13815.1     0.0    0   8.6  32
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      94.0   0.0 10666.6     0.0    0   2.8   9
ada1     103.7   0.0 11790.6     0.0    0   7.6  25
ada2      86.2   0.0  9310.0     0.0    0  14.7  36
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0   1.0     0.0     0.0    0   0.0   0
ada0      29.1   0.0  2614.9     0.0    0   2.0   2
ada1      18.5   0.0  1495.9     0.0    0   7.2   6
ada2      24.3   0.0  2113.7     0.0    0   3.2   3
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
ada0      54.2  41.4  5734.1   409.6    0   3.4  12
ada1      63.0  42.3  6049.2   409.6    0   5.7  21
ada2      45.3  45.3  4095.8   405.6    0   9.5  30
                        extended device statistics
device     r/s   w/s    kr/s    kw/s qlen svc_t  %b
md2        0.0   0.9     0.0     0.0    0   0.0   0
ada0      44.6   0.0  4797.2     0.0    0   3.6   6
ada1      40.8   0.0  4190.5     0.0    0   9.6  14
ada2      39.8   0.0  4611.4     0.0    0  20.9  29