Help me find bottleneck. Slow writes

lcnas · Feb 28, 2019

Can someone help me find my bottleneck. I am using this single zpool raidz2 via ISCSI to a Windows Server. It seems i can only average around 25-30 MiBs. This is hard wired and both 1GB NICs
info/specs of server below and thanks in advance!
...
sysctl hw.model hw.machine hw.ncpu
hw.model: Intel(R) Xeon(R) CPU E5-2603 v3 @ 1.60GHz
hw.machine: amd64
hw.ncpu: 6
24GB RAM
*I have ordered a SSD for ZIL and I have ordered more RAM. which come next week
...

Code:

sas2flash -listall
LSI Corporation SAS2 Flash Utility
Version 16.00.00.00 (2013.03.01)
Copyright (c) 2008-2013 LSI Corporation. All rights reserved
        Adapter Selected is a LSI SAS: SAS2008(B2)
Num   Ctlr            FW Ver        NVDATA        x86-BIOS         PCI Addr
----------------------------------------------------------------------------
0  SAS2008(B2)     20.00.07.00    14.01.00.08    07.39.02.00     00:0c:00:00

...
one raidz2 . did this for space. any other suggestions?
dedupe, compression and atime enabled
about 50% used/free right now
no snapshots on volume

Code:

  pool: nsdcrtbck2
state: ONLINE
  scan: scrub repaired 0 in 0 days 05:44:10 with 0 errors on Sun Feb  3 05:44:11 2019
config:
        NAME                                                STATE     READ WRITE CKSUM
        nsdcrtbck2                                          ONLINE       0     0     0
          raidz2-0                                          ONLINE       0     0     0
            gptid/996f73e6-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0
            gptid/9a62053a-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0
            gptid/9b68f805-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0
            gptid/9c637731-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0
            gptid/9d5b0383-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0
            gptid/9e4cd78a-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0
            gptid/9f55b3df-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0
            gptid/a04a8f88-056f-11e8-b05f-d4ae527a847c.eli  ONLINE       0     0     0

...
arcstat.py

Code:


    time  read  miss  miss%  dmis  dm%  pmis  pm%  mmis  mm%  arcsz     c
19:04:38   317    62     19    62   19     0    0    23    8    19G   19G

...
cpu average is very log
gui processes show very 800 which seems high to me for server only used as a backup storage device.
average load is around 1

Chris Moore · Feb 28, 2019

lcnas said:
raidz2 via ISCSI

That's the problem right there.

PS.

lcnas said:
Xeon(R) CPU E5-2603 v3 @ 1.60GHz

This is going to be an issue also. Clock speed matters.

lcnas · Feb 28, 2019

Chris Moore said:
That's the problem right there.

PS.

This is going to be an issue also. Clock speed matters.

Whats a good alternative for the single zpool of 8 drives? I was thinking 2 parity drives for safety and maximizing space usage when i set this up originally.

CPU might be something to consider but cpu load and average utilization are always so low at the moment that i didnt think about it.

Also are there any scripts/commands that someone has made that will tell you where the bottleneck may be on the disk array?

HoneyBadger · Feb 28, 2019

lcnas said:
dedupe enabled

Turn that off right now.

I'm not joking.

I also want to see the output of zpool status -D but I'm about 98% confident that your problem is right there.

lcnas · Feb 28, 2019

HoneyBadger said:
Turn that off right now.

I'm not joking.

I also want to see the output of zpool status -D but I'm about 98% confident that your problem is right there.

Code:

dedup: DDT entries 154583923, size 718 on disk, 232 in core
bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1     143M   17.8T   9.44T   9.44T     143M   17.8T   9.44T   9.44T
     2    4.34M    555G    324G    324G    9.21M   1.15T    692G    692G
     4     309K   38.7G   24.1G   24.1G    1.41M    181G    111G    111G
     8    55.7K   6.96G   3.14G   3.14G     575K   71.9G   32.4G   32.4G
    16    9.19K   1.15G    531M    531M     171K   21.4G   9.68G   9.68G
    32    6.88K    880M    120M    120M     348K   43.5G   5.77G   5.77G
    64    2.55K    326M   38.8M   38.8M     248K   31.1G   3.61G   3.61G
   128       86   10.8M   1.16M   1.16M    11.8K   1.47G    158M    158M
   256        5    640K     20K     20K    1.42K    182M   5.68M   5.68M
   512        8      1M     32K     32K    5.67K    726M   22.7M   22.7M
    1K        3    384K     12K     12K    3.28K    419M   13.1M   13.1M
    2K        1    128K      4K      4K    2.70K    345M   10.8M   10.8M
    4K        1    128K      4K      4K    6.78K    868M   27.1M   27.1M
  128K        2    256K      8K      8K     491K   61.4G   1.92G   1.92G
Total     147M   18.4T   9.79T   9.79T     155M   19.4T   10.3T   10.3T

lcnas · Feb 28, 2019

NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
NSDCRTBCK1 21.8T 9.93T 11.8T - 67% 45% 1.05x ONLINE /mnt

Heracles · Feb 28, 2019

Hi Lcnas,

As Chris said, RaidZ2 for serving iSCSI is a first culprit. To serve iSCSI, you need a max of IOPS and Raid10 will give you way more IOPS than RaidZ2. He is also right about the CPU being a little slow.

HoneyBadger got another one right : Deduplication will take a ton of RAM and will slow you down significantly. Usually, you achieved similar space saving with compression at a lower cost.

I will add that 50% used space in your pool is not down to the problematic level, but you are going there for sure. Because to add space in a pool can take a lot of time, you may be better to start increasing your capacity before your pool is too heavily loaded.

HoneyBadger · Feb 28, 2019

And that's a bingo.

This line right here points to your problem:

dedup: DDT entries 154583923, size 718 on disk, 232 in core

The total amount of memory used by your deduplication table (DDT) is entries x size in core - in your case that is 154583923 x 232 = 35863470136 bytes.

That's 33.4GB of RAM required - and your system has 24GB in total.

Quite frankly, I'm surprised your pool is working at all.

Additional RAM will allow it to regain some speed, but ultimately you need to turn off deduplication, and migrate the data off the pool and back on.

lcnas · Feb 28, 2019

Thanks a lot for the prompt help guys. Anyone have ebay links or alike to descent server with alot of bays. The current server i have is maxed out on HD bays.

Chris Moore · Feb 28, 2019

Please use tags like this [code] your text here [/code] so the output of these listing can be shown this way.

Code:

root@Emily-NAS:~ # zpool status
  pool: Backup
state: ONLINE
  scan: scrub repaired 0 in 0 days 03:31:34 with 0 errors on Sat Feb  9 13:58:28 2019
config:

        NAME                                            STATE     READ WRITE CKSUM
        Backup                                          ONLINE       0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/2e919d3d-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0
            gptid/2f292da6-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0
            gptid/2fb95d07-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0
            gptid/30514e6b-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0
          raidz1-1                                      ONLINE       0     0     0
            gptid/41d3312f-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0
            gptid/426b7b47-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0
            gptid/43029d18-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0
            gptid/43aa9a5b-2c1a-11e9-af8b-00074306773b  ONLINE       0     0     0

errors: No known data errors

Wth formatting intact. It makes them much easier to read.

Chris Moore · Feb 28, 2019

lcnas said:
Thanks a lot for the prompt help guys. Anyone have ebay links or alike to descent server with a lot of bays. The current server i have is maxed out on HD bays.

I have one like this that I put a newer system board in and it works extremely well:
https://www.ebay.com/itm/4U-48-Bay-...2x-Xeon-Low-Power-6-Core-2-26Ghz/132799185756

HoneyBadger · Mar 1, 2019

lcnas said:
Thanks a lot for the prompt help guys. Anyone have ebay links or alike to descent server with a lot of bays. The current server i have is maxed out on HD bays.

I'd just like to circle back on the dedup again:

Code:

dedup: DDT entries 154583923, size 718 on disk, 232 in core
bucket              allocated                       referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1     143M   17.8T   9.44T   9.44T     143M   17.8T   9.44T   9.44T
     2    4.34M    555G    324G    324G    9.21M   1.15T    692G    692G
     4     309K   38.7G   24.1G   24.1G    1.41M    181G    111G    111G
     8    55.7K   6.96G   3.14G   3.14G     575K   71.9G   32.4G   32.4G
    16    9.19K   1.15G    531M    531M     171K   21.4G   9.68G   9.68G
    32    6.88K    880M    120M    120M     348K   43.5G   5.77G   5.77G
    64    2.55K    326M   38.8M   38.8M     248K   31.1G   3.61G   3.61G
   128       86   10.8M   1.16M   1.16M    11.8K   1.47G    158M    158M
   256        5    640K     20K     20K    1.42K    182M   5.68M   5.68M
   512        8      1M     32K     32K    5.67K    726M   22.7M   22.7M
    1K        3    384K     12K     12K    3.28K    419M   13.1M   13.1M
    2K        1    128K      4K      4K    2.70K    345M   10.8M   10.8M
    4K        1    128K      4K      4K    6.78K    868M   27.1M   27.1M
  128K        2    256K      8K      8K     491K   61.4G   1.92G   1.92G
Total     147M   18.4T   9.79T   9.79T     155M   19.4T   10.3T   10.3T

You're getting about 5% data reduction benefits from deduplication. The vast majority of your data is unique.

You don't need more space. You just need to shut off dedup.

Chris Moore · Mar 1, 2019

HoneyBadger said:
You're getting about 5% data reduction benefits from deduplication. The vast majority of your data is unique.

While I agree that dedup is a resource hog and probably needs to go...

HoneyBadger said:
You don't need more space. You just need to shut off dedup.

His pool is filled to 50% in it's current RAIDz2 configuration. If it is reconfigured to a pool of mirrors, the OP probably does need more drive bays and ultimately will likely need a bit more space for the system to work smoothly.

lcnas said:
about 50% used/free right now

That based on the OP comment in an earlier post.

HoneyBadger · Mar 1, 2019

Chris Moore said:
His pool is filled to 50% in it's current RAIDz2 configuration. If it is reconfigured to a pool of mirrors, the OP probably does need more drive bays and ultimately will likely need a bit more space for the system to work smoothly.

That based on the OP comment in an earlier post.

I'll never argue against more for the sake of more (see my signature for my perspective on that) and while OP will definitely need a bunch of space to shuttle around the data to get rid of deduplication, I don't know that switching to mirrors is a foregone conclusion. It will be more performant certainly, but if this is a situation where iSCSI is being used unnecessarily where simple SMB would suffice, then OP could potentially stay with a Z2 setup.

@lcnas what is the workload on this Windows Server that is connected to the storage, and why was iSCSI selected vs. SMB?

pro lamer · Mar 28, 2019

HoneyBadger said:
why was iSCSI selected vs. SMB?

+1

Because I think getting rid of deduplication might be easier with smb. I'm thinking of a scenario:
1. Create a new dataset or pool with deduplication off.
2. Move/replicate data to the new dataset/pool
3. Destroy the old dataset

And I'm guessing these steps are "easier" with smb

Is that true?

Sent from my phone

Important Announcement for the TrueNAS Community.

Help me find bottleneck. Slow writes

lcnas

Cadet

Chris Moore

Hall of Famer

lcnas

Cadet

HoneyBadger

actually does care

lcnas

Cadet

lcnas

Cadet

Heracles

Wizard

HoneyBadger

actually does care

lcnas

Cadet

Chris Moore

Hall of Famer

Chris Moore

Hall of Famer

HoneyBadger

actually does care

Chris Moore

Hall of Famer

HoneyBadger

actually does care

pro lamer

Guru

Similar threads

Important Announcement for the TrueNAS Community.

Help me find bottleneck. Slow writes

Cadet

Hall of Famer

Cadet

actually does care

Cadet

Cadet

Wizard

actually does care

Cadet

Hall of Famer

Hall of Famer

actually does care

Hall of Famer

actually does care

Guru

Important Announcement for the TrueNAS Community.

Related topics on forums.truenas.com for thread: "Help me find bottleneck. Slow writes"

Similar threads