Horrible Performance with FreeNAS as VM Storage

[SweetAndLow]

Hi,

because the size you set in the command is probably going to take the whole night, I've set it to a smaller value. Here are the results:



10710155264 bytes transferred in 198.128380 secs (54056644 bytes/sec)



10710155264 bytes transferred in 344.650258 secs (31075431 bytes/sec)

The problem seems to be related to the installed hard disks (HP, 2.5, 300GB SAS, 10K, 6G).

You have local disk speed issues. You should be getting 10x what you are getting now. Let's test the speed of raw disk read. Try running a dd test when the if=/dev/xx

 

[fc7t]

Hi,

seen in another thread I think you mean

Code:

camcontrol identify daX

Unfortunately this doesn't give me any infos on the drives except an error message:

Code:

#camcontrol identify da0
#camcontrol: ATA ATAPI_IDENTIFY via pass_16 failed

 

[HoneyBadger]

Try "camcontrol inquiry da0"

 

[jgreco]

BTW.: The MTU size is set to 9000 on both sides (FreeNAS via interface options "mtu 9000" and ESXi-Host via vSwitch). Also FreeNAS system itself is installed on a slow USB 2.0 thumb drive.

Ditch the jumbo. https://forums.freenas.org/index.php?threads/jumbo-frames-notes.26064/

 

[jgreco]

Okay, so I want you to go over to this thread

https://forums.freenas.org/index.ph...esting-your-freenas-system.17750/#post-148773

where there's a nice array testing script. I want you to run that on your filer and see what it says. It's safe to run on your existing pool.

 

[solarisguy]

@fc7t, could you please post output of

• sas2flash -o -c 0 -list
• sas2flash -o -c 1 -list (if there are two controllers)

 

[tvsjr]

camcontrol modepage <path_to_drive> -m 0x08

This should return something like:

Code:

IC:  0
ABPF:  0
CAP:  0
DISC:  1
SIZE:  0
WCE:  1
MF:  0
RCD:  0
Demand Retention Priority:  0
Write Retention Priority:  0
Disable Pre-fetch Transfer Length:  65535
Minimum Pre-fetch:  0
Maximum Pre-fetch:  65535
Maximum Pre-fetch Ceiling:  65535

You're looking for WCE - 0 = off, 1 = on.

I had a problem with drives repurposed from a NetApp always defaulting to off (after fixing them from being 528-byte sectors), so I created this script in /usr/local/bin and added it as a post-init script in the GUI. No problems since.

Code:

#!/bin/bash
for file in /dev/da? /dev/da??; do echo "WCE: 1" | camcontrol modepage $file -m 0x08 -e; done

 

[fc7t]

Try "camcontrol inquiry da0"

Code:

[root@DE001-STR-44-2] ~# camcontrol inquiry da2
pass2: <HP EG0300FAWHV HPDF> Fixed Direct Access SCSI-5 device
pass2: Serial Number 3SE1ZTW500009042NV4K
pass2: 600.000MB/s transfers, Command Queueing Enabled

I used to specify da2 because da0 and da1 are two 146 GB 10K 6G SAS disks which I wanted to use to create a ZVOL for testing purposes only. Not done this yet.

 

[fc7t]

@fc7t, could you please post output of

• sas2flash -o -c 0 -list
• sas2flash -o -c 1 -list (if there are two controllers)

Code:

[root@DE001-STR-44-2] ~# sas2flash -o -c 0 -list
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved

    Advanced Mode Set

    Adapter Selected is a LSI SAS: SAS2008(B2)

    Controller Number              : 0
    Controller                     : SAS2008(B2)
    PCI Address                    : 00:03:00:00
    SAS Address                    : 500605b-0-0975-6cb0
    NVDATA Version (Default)       : 14.01.00.08
    NVDATA Version (Persistent)    : 14.01.00.08
    Firmware Product ID            : 0x2213 (IT)
    Firmware Version               : 20.00.04.00
    NVDATA Vendor                  : LSI
    NVDATA Product ID              : SAS9211-8i
    BIOS Version                   : 07.39.00.00
    UEFI BSD Version               : N/A
    FCODE Version                  : N/A
    Board Name                     : SAS9211-8i
    Board Assembly                 : H3-25250-02K
    Board Tracer Number            : SP52809892

    Finished Processing Commands Successfully.
    Exiting SAS2Flash.

 

[fc7t]

camcontrol modepage <path_to_drive> -m 0x08

You're looking for WCE - 0 = off, 1 = on.

Code:

[root@DE001-STR-44-2] ~# camcontrol modepage /dev/da2 -m 0x08
IC:  0
ABPF:  0
CAP:  0
DISC:  1
SIZE:  0
WCE:  0
MF:  0
RCD:  0
Demand Retention Priority:  0
Write Retention Priority:  0
Disable Pre-fetch Transfer Length:  65535
Minimum Pre-fetch:  0
Maximum Pre-fetch:  65535
Maximum Pre-fetch Ceiling:  65535

Results are the same for all drives (from da2 to da17).

 

[fc7t]

Okay, so I want you to go over to this thread

https://forums.freenas.org/index.ph...esting-your-freenas-system.17750/#post-148773

where there's a nice array testing script. I want you to run that on your filer and see what it says. It's safe to run on your existing pool.

Ok, because this is going to take a long time to complete, I will post the results of the first part of the test:

Code:

1) Use all disks (from camcontrol)
2) Use selected disks (from camcontrol|grep)
3) Specify disks
4) Show camcontrol list

Option: 2

Enter grep match pattern (e.g. ST150176): 300

Selected disks: da2 da3 da4 da5 da6 da7 da8 da9 da10 da11 da12 da13 da14 da15 da16 da17
<HP EG0300FAWHV HPDF>              at scbus0 target 10 lun 0 (pass2,da2)
<HP DG0300FARVV HPDG>              at scbus0 target 11 lun 0 (pass3,da3)
<HP EG0300FAWHV HPDF>              at scbus0 target 12 lun 0 (pass4,da4)
<HP EG0300FAWHV HPDF>              at scbus0 target 13 lun 0 (pass5,da5)
<HP EG0300FAWHV HPDF>              at scbus0 target 14 lun 0 (pass6,da6)
<HP DG0300FARVV HPDG>              at scbus0 target 15 lun 0 (pass7,da7)
<HP EG0300FAWHV HPDF>              at scbus0 target 16 lun 0 (pass8,da8)
<HP DG0300FARVV HPDG>              at scbus0 target 17 lun 0 (pass9,da9)
<HP EG0300FAWHV HPDF>              at scbus0 target 18 lun 0 (pass10,da10)
<HP EG0300FAWHV HPDF>              at scbus0 target 19 lun 0 (pass11,da11)
<HP EG0300FAWHV HPDF>              at scbus0 target 20 lun 0 (pass12,da12)
<HP EG0300FAWHV HPDE>              at scbus0 target 21 lun 0 (pass13,da13)
<HP EG0300FAWHV HPDF>              at scbus0 target 22 lun 0 (pass14,da14)
<HP EG0300FBDBR HPDA>              at scbus0 target 23 lun 0 (pass15,da15)
<HP EG0300FAWHV HPDD>              at scbus0 target 24 lun 0 (pass16,da16)
<HP EG0300FBVFL HPD9>              at scbus0 target 25 lun 0 (pass17,da17)
Is this correct? (y/N): y
Performing initial serial array read (baseline speeds)
Thu Dec 24 11:31:20 CET 2015
Thu Dec 24 12:07:21 CET 2015
Completed: initial serial array read (baseline speeds)

Array's average speed is 130.238 MB/sec per disk

Disk    Disk Size  MB/sec %ofAvg
------- ---------- ------ ------
da2       286102MB    122     93
da3       286102MB    134    103
da4       286102MB    121     93
da5       286102MB    125     96
da6       286102MB    129     99
da7       286102MB    138    106
da8       286102MB    124     95
da9       286102MB    128     98
da10      286102MB    120     92 --SLOW--
da11      286102MB    122     94
da12      286102MB    121     93
da13      286102MB    124     96
da14      286102MB    117     90 --SLOW--
da15      286102MB    163    125 ++FAST++
da16      286102MB    127     98
da17      286102MB    170    130 ++FAST++

EDIT: Part 2 of the test:

Code:

Performing initial parallel array read
Thu Dec 24 12:07:21 CET 2015
The disk da2 appears to be 286102 MB.
Disk is reading at about 116 MB/sec
This suggests that this pass may take around 41 minutes

                   Serial Parall % of
Disk    Disk Size  MB/sec MB/sec Serial
------- ---------- ------ ------ ------
da2       286102MB    122    122    101
da3       286102MB    134    132     99
da4       286102MB    121    116     96
da5       286102MB    125    122     98
da6       286102MB    129    122     95
da7       286102MB    138    126     92 --SLOW--
da8       286102MB    124    117     94
da9       286102MB    128    122     95
da10      286102MB    120    110     92
da11      286102MB    122    111     91 --SLOW--
da12      286102MB    121    109     90 --SLOW--
da13      286102MB    124    104     84 --SLOW--
da14      286102MB    117     92     79 --SLOW--
da15      286102MB    163    137     84 --SLOW--
da16      286102MB    127    108     85 --SLOW--
da17      286102MB    170    132     78 --SLOW--

EDIT: Part 3 of the test:

Code:

Awaiting completion: initial parallel array read
Thu Dec 24 13:04:54 CET 2015
Completed: initial parallel array read

Disk's average time is 2811 seconds per disk

Disk    Bytes Transferred Seconds %ofAvg
------- ----------------- ------- ------
da2          300000000000    3004    107
da3          300000000000    2604     93
da4          300000000000    3011    107 --SLOW--
da5          300000000000    2900    103
da6          300000000000    2814    100
da7          300000000000    2565     91 ++FAST++
da8          300000000000    2904    103
da9          300000000000    2686     96
da10         300000000000    2995    107
da11         300000000000    2920    104
da12         300000000000    3001    107
da13         300000000000    3075    109 --SLOW--
da14         300000000000    3453    123 --SLOW--
da15         300000000000    2108     75 ++FAST++
da16         300000000000    2855    102
da17         300000000000    2089     74 ++FAST++

 

[HoneyBadger]

WCE: 0

Drive cache seems to be disabled according to that.

Link from FreeBSD forums:

https://forums.freebsd.org/threads/...western-digital-sas-drives.40534/#post-225263

Code from that thread, but it's referring to WD drives:

Code:

echo "WCE: 1" | camcontrol modepage da0 -m 0x08 -e

 

[jgreco]

Something's fundamentally funky if the array test can do its parallel array read test and get 100MB/sec simultaneously out of all the disks in parallel.

Wait. Now that I go back and read your original message, I think I warned early on that you can't set a zvol to the size of your pool. If you've gone and written to most of the 2.1TB of the pool, ZFS is simply struggling to allocate new blocks and your read problems are probably due to excessive fragmentation.

Get rid of the zvol. Then try the dd tests you were doing in message #18 again. Bet it flies. Then recreate your zvol, this time limiting size of the zvol to no more than 1TB. Grinch bets: problem fixed.

 

[fc7t]

Hi, just to give you an update on how things are going.

I have deleted the ZVOL and created a new one with 1000 GiB. Results were the same. Exceptionally slow performance. Then I deleted the freshly created ZVOL again. Also I deleted the volume with the mirrored/stripped disks (aka RAID 10). Next I created a RAIDZ (aka RAID 5) volume with 7 disks and a resulting size of 1.6 TiB. In the last step I created a new ZVOL with about 1.2 TiB size and disabled compression on volume and zvol level.

I ran the dd tests with 100 GByte write and read each and the results were fantastic. About 630 MByte/s write and 720 MByte/s read rate. The graphs under the Reporting tab in the FreeNAS GUI also showed approximately 100-120 MByte/s per disk in the array.

Now the real world results are not that great anymore. Referring to my initial post the same copy procedure now takes much less time the transfer rate is still not the best:

 

[fc7t]

For your interest: Currently I have created a new RAIDZ array with remaining 9 disks in the storage server with the size of 2.4 TiB. Next I created a new ZVOL with a size of 2.0 TiB. Performance is absolutely miserable on this ZVOL.

 

[jgreco]

Well, trying to use any RAIDZ as VM storage is a path to much badness. I'm not sure that we've done anything here other than to prove that point the long way around.

 

[solarisguy]

@fc7t, what is the brand of your 9211-8i HBA ?

Have you changed anything in the card BIOS? And conversely, have you tried to reset it to factory defaults?

I have 9211-8i in a SilverStone made card, model SST-ECS02.

 

[fc7t]

@fc7t, what is the brand of your 9211-8i HBA ?

Have you changed anything in the card BIOS? And conversely, have you tried to reset it to factory defaults?

I have 9211-8i in a SilverStone made card, model SST-ECS02.

Hi, I bought the card as a new original LSI (Avago) 9211-8i for about 270,- EUR.

 

[fc7t]

Well, trying to use any RAIDZ as VM storage is a path to much badness. I'm not sure that we've done anything here other than to prove that point the long way around.

Yes I have read this in other posts already. I don't want to use RAIDZ because of several reasons like parity calculations (and therefore performance loss) and redundancy of one failing drive but the strange thing is that it is much more faster than the "RAID 10". :-/

 

[HoneyBadger]

the strange thing is that it is much more faster than the "RAID 10". :-/

It's only "faster" because right now you're presenting a sequential workload. Parity RAID does that quite well.

As soon as you start to put a multi-VM load or any other kind of random I/O against it, your performance drops significantly.