Performance issue

PetrZ · Feb 13, 2019

Just tried one of boxes with nextcloud. I don't like what I see.

Code:

Starting scan for user 1 out of 1 (petr)
+---------+-------+--------------+
| Folders | Files | Elapsed time |
+---------+-------+--------------+
| 338     | 40670 | 06:08:35     |
+---------+-------+--------------+

I know that nextcloud use mySQL / mariadb, I know it's not good idea to run DB on RAIDZ1 of slow drives, but 6 hours?
Most of the time CPU load was close to zero, very low . Is it normal with used HW, or is it configuration issue? Or result of pool usage (almost full)? Iocage use zpool NAS.

ProLiant DL360 G6, 2x Xeon E5620 @ 2.40GHz, DDR3 16GB (4x4GB - Speed: 1333 MT/s, Configured Clock Speed: 800 MT/s)

Code:

 Index    Type          ID      ID    Pci Address          Ven ID  Dev ID
 -----  ------------  ------  ------  -----------------    ------  ------
   0     SAS2308_2     1000h    87h   00h:07h:00h:00h      1000h   3020h

        Adapter Selected is a LSI SAS: SAS2308_2(D1)  

        Controller Number              : 0
        Controller                     : SAS2308_2(D1)  
        PCI Address                    : 00:07:00:00
        SAS Address                    : 500605b-0-071f-e7d0
        NVDATA Version (Default)       : 14.01.00.06
        NVDATA Version (Persistent)    : 14.01.00.06
        Firmware Product ID            : 0x2214 (IT)
        Firmware Version               : 20.00.07.00
        NVDATA Vendor                  : LSI
        NVDATA Product ID              : SAS9207-8i
        BIOS Version                   : 07.39.02.00
        UEFI BSD Version               : N/A
        FCODE Version                  : N/A
        Board Name                     : H220
        Board Assembly                 : H3-25278-05D
        Board Tracer Number            : SV34428252

------------------------------------------------------------------------
Controller information
------------------------------------------------------------------------
  Controller type                         : SAS2308_2
  BIOS version                            : 7.39.02.00
  Firmware version                        : 20.00.07.00
  Channel description                     : 1 Serial Attached SCSI
  Initiator ID                            : 0
  Maximum physical devices                : 1023
  Concurrent commands supported           : 10240
  Slot                                    : 2
  Segment                                 : 0
  Bus                                     : 7
  Device                                  : 0
  Function                                : 0
  RAID Support                            : No
------------------------------------------------------------------------
IR Volume information
------------------------------------------------------------------------
------------------------------------------------------------------------
Physical device information
------------------------------------------------------------------------
Initiator at ID #0

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 0
  SAS Address                             : 4433221-1-0300-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : WDZAVJNQ
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 1
  SAS Address                             : 4433221-1-0200-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : WDZAVCBX
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 2
  SAS Address                             : 4433221-1-0100-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : WDZAR1JJ
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 3
  SAS Address                             : 4433221-1-0000-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : WDZAVDTK
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 4
  SAS Address                             : 4433221-1-0700-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : ZDZ4J9V3
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 5
  SAS Address                             : 4433221-1-0600-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : ZDZ4LAPJ
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 6
  SAS Address                             : 4433221-1-0500-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : ZDZ4LCSC
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD

Device is a Hard disk
  Enclosure #                             : 1
  Slot #                                  : 7
  SAS Address                             : 4433221-1-0400-0000
  State                                   : Ready (RDY)
  Size (in MB)/(in sectors)               : 1907729/3907029167
  Manufacturer                            : ATA     
  Model Number                            : ST2000LM015-2E81
  Firmware Revision                       : SDM1
  Serial No                               : ZDZ4JB4W
  GUID                                    : N/A
  Protocol                                : SATA
  Drive Type                              : SATA_HDD
------------------------------------------------------------------------
Enclosure information
------------------------------------------------------------------------
  Enclosure#                              : 1
  Logical ID                              : 500605b0:071fe7d0
  Numslots                                : 8
  StartSlot                               : 0
------------------------------------------------------------------------

Code:

NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
BACKUP        7.25T   722G  6.54T        -         -     0%     9%  1.00x  ONLINE  /mnt
EXT_2TB       1.81T  23.5G  1.79T        -         -     0%     1%  1.00x  ONLINE  /mnt
NAS           7.25T  6.85T   413G        -         -    40%    94%  1.00x  ONLINE  /mnt
USB3          12.5G  1.36G  11.1G        -         -     1%    10%  1.00x  ONLINE  /mnt
freenas-boot  7.44G  4.08G  3.36G        -         -      -    54%  1.00x  ONLINE  -

  pool: BACKUP
 state: ONLINE
  scan: none requested
config:

        NAME                                                STATE     READ WRITE CKSUM
        BACKUP                                              ONLINE       0     0     0
          raidz1-0                                          ONLINE       0     0     0
            gptid/b064c9d4-14fb-11e9-a9a1-0026557cfab4.eli  ONLINE       0     0     0
            gptid/b18d71c2-14fb-11e9-a9a1-0026557cfab4.eli  ONLINE       0     0     0
            gptid/b2e30fd2-14fb-11e9-a9a1-0026557cfab4.eli  ONLINE       0     0     0
            gptid/b40b2a96-14fb-11e9-a9a1-0026557cfab4.eli  ONLINE       0     0     0

errors: No known data errors

  pool: EXT_2TB
 state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(7) for details.
  scan: scrub repaired 0 in 0 days 00:04:23 with 0 errors on Sun Feb 10 00:04:25 2019
config:

        NAME                                          STATE     READ WRITE CKSUM
        EXT_2TB                                       ONLINE       0     0     0
          gptid/dbb106c3-f021-11e8-a7d0-0026557cfab4  ONLINE       0     0     0

errors: No known data errors

  pool: NAS
 state: ONLINE
  scan: scrub repaired 0 in 0 days 12:08:34 with 0 errors on Sun Feb  3 12:08:42 2019
config:

        NAME                                                STATE     READ WRITE CKSUM
        NAS                                                 ONLINE       0     0     0
          raidz1-0                                          ONLINE       0     0     0
            gptid/d34790f0-6a82-11e8-86ba-0026557cfab4.eli  ONLINE       0     0     0
            gptid/d47089ec-6a82-11e8-86ba-0026557cfab4.eli  ONLINE       0     0     0
            gptid/d5a54405-6a82-11e8-86ba-0026557cfab4.eli  ONLINE       0     0     0
            gptid/d6c872b8-6a82-11e8-86ba-0026557cfab4.eli  ONLINE       0     0     0

errors: No known data errors

  pool: USB3
 state: ONLINE
  scan: none requested
config:

        NAME                                          STATE     READ WRITE CKSUM
        USB3                                          ONLINE       0     0     0
          gptid/fde7e527-2c51-11e9-b48f-0026557cfab4  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:05:50 with 0 errors on Wed Feb  6 03:50:50 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          da10p2    ONLINE       0     0     0

errors: No known data errors

Code:

        512             # sectorsize
        2000398934016   # mediasize in bytes (1.8T)
        3907029168      # mediasize in sectors
        4096            # stripesize
        0               # stripeoffset
        243201          # Cylinders according to firmware.
        255             # Heads according to firmware.
        63              # Sectors according to firmware.
        ATA ST2000LM015-2E81    # Disk descr.
        ZDZ4JB4W        # Disk ident.
        No              # TRIM/UNMAP support
        5400            # Rotation rate in RPM
        Not_Zoned       # Zone Mode

Seek times:
        Full stroke:      250 iter in   7.903764 sec =   31.615 msec
        Half stroke:      250 iter in   5.498235 sec =   21.993 msec
        Quarter stroke:   500 iter in   5.078009 sec =   10.156 msec
        Short forward:    400 iter in   3.125580 sec =    7.814 msec
        Short backward:   400 iter in   2.756943 sec =    6.892 msec
        Seq outer:       2048 iter in   0.131884 sec =    0.064 msec
        Seq inner:       2048 iter in   1.140489 sec =    0.557 msec

Transfer rates:
        outside:       102400 kbytes in   0.820239 sec =   124842 kbytes/sec
        middle:        102400 kbytes in   0.860171 sec =   119046 kbytes/sec
        inside:        102400 kbytes in   1.755340 sec =    58336 kbytes/sec

Chris Moore · Feb 13, 2019

PetrZ said:
I know that nextcloud use mySQL / mariadb, I know it's not good idea to run DB on RAIDZ1 of slow drives, but 6 hours?

If you know it is bad, why did you do it. I have a hard time beleiving that anyone on this forum would have advised you to build a system like this.

PetrZ said:

Did you also know that ZFS performance falls off a cliff when the pool is over 90% filled and yours is at 94%?
You didn't read any of the documentation did you? Anyone knowledgeable on this forum, that answers questions regularly, would have told you that for a database application, you need a pool of mirror vdevs instead of RAIDz because more vdevs makes more IOPS where RAIDz is limited to the performance of the slowest drive and each vdev performs like a single drive.

The only solution to this is to move all the data off the system, build a new, larger pool, using mirrors instead of using RAIDz, then the data can be moved back into the system. It is always best to ask questions instead of doing things wrong and needing to do it over.

Chris Moore · Feb 13, 2019

If all your server has is those eight drive bays, you are either going to need to add another drive shelf externally, or you will need larger drives. Personally, I would suggest more drive bays by way of an external SAS attached drive shelf because you need as many drives as possible to give you IOPS for your database.

PetrZ · Feb 13, 2019

Chris Moore said:
If you know it is bad, why did you do it. I have a hard time beleiving that anyone on this forum would have advised you to build a system like this.

Did you also know that ZFS performance falls off a cliff when the pool is over 90% filled and yours is at 94%?

Originally system was built just as backup storage and for plex with two users. It worked OK for that purpose, including 4K streams.
As you can see, I already added another 4 drives and I am going to move part of data there, including nextcloud data. I've been asked by family members to backup two of their systems few days ago, so I copied their files to cloud data folders and run files scan. Till that time pool wasn't used over 75%. So soon there will be usage much lower again. Because I installed nextcloud with need of DB, I also plan to install PCI-E card with 2 or 4 msata or m.2 for RAID0 or RAID10 for DB. I don't have enough slots to use 2.5" SSDs and I am considering which card I should buy. I was just surprised that it's so bad and I wanted to be sure that root cause is just filled pool and/or RAIDZ1 and it's normal state, or if I have some other issue (e.g. wrong FW on HBA). You probably noticed that I mentioned almost full pool, so I am aware about this aspect.

Important Announcement for the TrueNAS Community.

Performance issue

PetrZ

Dabbler

Chris Moore

Hall of Famer

Chris Moore

Hall of Famer

PetrZ

Dabbler

Similar threads

Important Announcement for the TrueNAS Community.

Performance issue

PetrZ

Dabbler

Chris Moore

Hall of Famer

Chris Moore

Hall of Famer

PetrZ

Dabbler

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Performance issue"

Similar threads