Disck activity falls to zero if there are file deletions

[HoneyBadger]

Help me understand how these numbers are calculated.

I'm having trouble figuring out how to anticipate needing new disks for dedup.

dedup: DDT entries X, size Y B on disk, Z B in core

To calculate size on disk, multiply X by Y. For RAM, X times Z in bytes.

dedup: DDT entries 10111421243, size 582B on disk, 188B in core

So for your numbers:

10111421243 x 582 bytes = 5,884,847,163,426 bytes on disk
10111421243 x 188 bytes = 1,900,947,193,684 bytes in RAM

But it's not just a "capacity" argument - you're also asking a lot of random 4K I/O from your dedup devices. Using iostat -x 5 will show you more detailed information including the rareq-sz and wareq-sz (read and write average request size) which are almost exclusively 4K for deduplication tables, so you may just be exhausting the capabilities of your DDT devices.

Spec sheets often show results for performance in pure read/write tests, with high queue depths - as soon as you mix reads and writes in even a 10% ratio either way, you can often lose half of the performance immediately in what's called the "bathtub curve" effect. Add in potentially a much lower queue depth and this only exacerbates the issue.

What is the client workload that's causing this level of duplication of records?

 

[vaschthestampede]

Thanks so much for the explanation.

10111421243 x 582 bytes = 5,884,847,163,426 bytes = 5.35 TB on disk
10111421243 x 188 bytes = 1,900,947,193,684 bytes = 1.73 TB in RAM

As for the disk we are at around 5.35TB out of 6.98Tb on disk.
What doesn't come back to me is that I have "only" 512GB of RAM in the system, so I don't know where 1.93 TB is stored.

What is the client workload that's causing this level of duplication of records?

3D survey processing.

 

[HoneyBadger]

For best performance the DDT should fit in RAM; overflowing it doesn't immediately break things, but will hurt performance (and if it has to go to disk, could be indistinguishable from "break things.") That's why you're seeing the reads coming from the dedup vdevs; your entire DDT can't fit in RAM, so if it doesn't find a matching hash cached in RAM, it has to go to your SSDs.

The disks are holding the permanent copy of your DDT; that's used for sizing your DDT vdevs. You should always leave space for performance reasons and unexpected growth; but ultimately, you probably need even more RAM.

There's another potentially aggravating factor here as well - have you added more HDDs as well since the initial post? Your initial posting indicates "6x 11-wide Z2 vdevs of ~20T" which accounts for around 6x9x20 = 1080T of space ... and you appear to be perilously close to actually hitting that point.

 

[vaschthestampede]

have you added more HDDs as well since the initial post?

In the system I have:
9x 11-wide Z2 vdevs of ~20T (Of which two were added just today)
2x 2-wide mirror vdevs of ~3.49T (Of which two were added after the problem first appeared)

I can "easily" increase the RAM to 1.5TB, then I will have to change all the ram banks.

 

[HoneyBadger]

In the system I have:
9x 11-wide Z2 vdevs of ~20T (Of which two were added just today)
2x 2-wide mirror vdevs of ~3.49T (Of which two were added after the problem first appeared)

I can "easily" increase the RAM to 1.5TB, then I will have to change all the ram banks.

If your workload is mostly "write once, read many, delete never" then you can probably continue with the expansions, but always trying to stay "one vdev ahead" would be a good idea.

Increasing your RAM should help mitigate the impact of this but ultimately you're running a very large capacity system with deduplication, which at this point carries certain performance overheads for the 1.2:1 reduction you're currently getting.

 

[vaschthestampede]

"write once, read many, delete never"

Nice way to describe our workflow.

but always trying to stay "one vdev ahead"

Are you referring to vdevs for data or dedup?

 

[NugentS]

At 1.2:1 I question whether dedupe is even worth it. Mind you extricating oneself from the dedupe hole the OP is down might have an "entertainment" value of high

 

[HoneyBadger]

Are you referring to vdevs for data or dedup?

Both, to be honest. You may start to hit a challenge with your metadata vdevs as well, since you will need additional SFF drive bays to continue expanding (or else obtain some 7.6T SSDs)

 

[vaschthestampede]

At 1.2:1 I question whether dedupe is even worth it.

With the current occupation there are over 270 TB of space gained, of course if I could understand how to avoid these problems it would be better.

You may start to hit a challenge with your metadata vdevs as well

We don't have Metadata VDEVs, currently.
Or are you referring to Dedup VDEVs?
In this case I would be very happy to take SATA SSD disks even larger than 8TB but, to my knowledge, there aren't any.

 

[sretalla]

I would be very happy to take SATA SSD disks even larger than 8TB but, to my knowledge, there aren't any.

Are you a human?

www.newegg.com

 

[HoneyBadger]

We don't have Metadata VDEVs, currently.
Or are you referring to Dedup VDEVs?
In this case I would be very happy to take SATA SSD disks even larger than 8TB but, to my knowledge, there aren't any.

Yes, I'm referring to the dedup vdevs here.

Are you a human?

www.newegg.com

Technically those are SAS, if true SATA is needed then it will be a challenge.

 

[NugentS]

Those are DWPD = 1 - so Read Intensive Drives. OK - so thats still 30TB /day which is a lot

 

[HoneyBadger]

Those are DWPD = 1 - so Read Intensive Drives. OK - so thats still 30TB /day which is a lot

It's more that they're using the SAS protocol, so if @vaschthestampede is counting on using internal, SATA-only connectivity rather than a SAS backplane, they will be unusable.

But in general, I still believe we're reaching close to the end of the road that the metaphorical can is being kicked down. While 270T of saved space may seem like a large amount, the issues with slow performance on delete (and most recently the system becoming non-responsive waiting for the DDT to catch up) are being experienced as a cost to it.

I'd like to see an additional zpool status -D and zpool iostat -v during the times of "system not responsive" - although this might require an already-active SSH session.

 

[vaschthestampede]

I'd like to see an additional zpool status -D and zpool iostat -v during the times of "system not responsive"

Code:

root@truenas[~]# zpool status -D
  pool: Archivio
 state: DEGRADED
status: One or more devices could not be used because the label is missing or
        invalid.  Sufficient replicas exist for the pool to continue
        functioning in a degraded state.
action: Replace the device using 'zpool replace'.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-4J
  scan: resilvered 5.68T in 1 days 09:04:29 with 0 errors on Sat Feb  3 20:04:54 2024
config:

        NAME                                      STATE     READ WRITE CKSUM
        Archivio                                  DEGRADED     0     0     0
          raidz2-0                                ONLINE       0     0     0
            b097df6a-2f27-4394-9730-76261871cec3  ONLINE       0     0     0
            0dd84988-a18f-4f5c-909f-460d6de433c0  ONLINE       0     0     0
            9ad1c3d8-ecf6-41f1-8a02-2a27441a08e8  ONLINE       0     0     0
            9e13868e-9418-4a26-b858-d480ce639fd5  ONLINE       0     0     0
            ac7d4716-ba83-4209-af42-ad7a1f7e8799  ONLINE       0     0     0
            9ad14a36-d5ad-40d3-bf11-40590853b7a8  ONLINE       0     0     0
            3062e26b-01c1-4233-8735-f235958c9d5e  ONLINE       0     0     0
            5015360e-990c-428a-a3c0-5ad57ea530f1  ONLINE       0     0     0
            cdf836cf-e017-4df8-bf4d-2b0e7b4bbf0a  ONLINE       0     0     0
            660d5ea7-049b-44be-819b-fc5866bad3ad  ONLINE       0     0     0
            361f3028-4da4-490c-a674-704918283cf4  ONLINE       0     0     0
          raidz2-1                                ONLINE       0     0     0
            3e93718b-720e-4f5f-a7f4-7b3ea3750ae2  ONLINE       0     0     0
            17a10b38-da89-4090-bad9-e7b9b58d82a1  ONLINE       0     0     0
            260476a7-a757-4293-be9e-7f94d3ad1127  ONLINE       0     0     0
            e6800b36-8d0e-41f6-8eef-cd8bda0f423b  ONLINE       0     0     0
            f0bb6ca1-3a14-479a-a3a2-3f9db1126070  ONLINE       0     0     0
            d69e75a9-76a8-419f-8fb4-964a95633db4  ONLINE       0     0     0
            873cb9f9-4b20-4cd0-a464-71b5c91669a6  ONLINE       0     0     0
            9fe2f86b-3681-4772-bcd6-dac7961a6172  ONLINE       0     0     0
            fe92d53d-a151-450c-85b4-685ea1d3d82a  ONLINE       0     0     0
            fd8a246d-5346-4b84-baf6-c1169e9238f6  ONLINE       0     0     0
            716bcb7f-d18b-4e82-866b-7afabfa68029  ONLINE       0     0     0
          raidz2-3                                ONLINE       0     0     0
            68a7a9b3-12b9-4738-9836-327b7e872bae  ONLINE       0     0     0
            851f1e90-d49a-4f44-93df-baf5cf368ad0  ONLINE       0     0     0
            8a8976b5-227a-4d2c-a7c5-b56e171e2186  ONLINE       0     0     0
            e4646155-a067-492d-b5f3-c17c8e400008  ONLINE       0     0     0
            ccfd693d-ccb2-4e87-b2fb-0458731d1c10  ONLINE       0     0     0
            a6c8d71a-26a4-40b0-aef1-7cd8a5ec31df  ONLINE       0     0     0
            27c1e6d6-274c-461f-b458-b9ecea0ab187  ONLINE       0     0     0
            ddaba7e8-5c4f-49db-86a8-3944f04d229c  ONLINE       0     0     0
            3980fb00-af74-47d0-bd6b-dc88de0e0be7  ONLINE       0     0     0
            63fb1f47-0d2a-4b11-a534-e75091418966  ONLINE       0     0     0
            ba1fa4e5-1729-4249-803b-725a5be6026d  ONLINE       0     0     0
          raidz2-4                                ONLINE       0     0     0
            ad2112a5-834f-487f-8e04-647a503bf678  ONLINE       0     0     0
            709505c7-40bc-4489-bc77-2883871dc245  ONLINE       0     0     0
            0765d45e-aeed-4300-9d28-7d8b464c5b09  ONLINE       0     0     0
            eca420ec-f3f8-4c0a-b6b2-08341a752184  ONLINE       0     0     0
            131d1bdc-be96-4365-b9bb-71cb54b7e022  ONLINE       0     0     0
            f19a7a96-2496-43af-91b0-3c12d0d534da  ONLINE       0     0     0
            5b79e55d-6d4d-43d8-863a-c741809ef504  ONLINE       0     0     0
            11f95813-4191-4b17-876e-6fb03dbf313c  ONLINE       0     0     0
            15a8a606-f117-4d35-81d2-93fcec38c451  ONLINE       0     0     0
            83aa4cf6-fa4e-4600-9d88-74d6a99a6933  ONLINE       0     0     0
            daa9f691-7ca7-4888-afea-2952f6274496  ONLINE       0     0     0
          raidz2-5                                ONLINE       0     0     0
            04056fc8-84f0-412f-8276-56fa5747f0b9  ONLINE       0     0     0
            f27444a7-f73b-45dc-91d3-07436e3d666b  ONLINE       0     0     0
            06cfc1b9-b7ab-4091-bedd-27d644bf9688  ONLINE       0     0     0
            297aca17-ffec-49e1-b70d-a9ff01289518  ONLINE       0     0     0
            91826ebe-cad1-4ed0-97f3-2b9fa4732ab6  ONLINE       0     0     0
            62b145da-f906-4096-ace3-7f5ddfe387ad  ONLINE       0     0     0
            3cc6e221-e537-4906-956f-3138ab840e62  ONLINE       0     0     0
            0ede001d-32c5-4c72-abee-493a88d495d8  ONLINE       0     0     0
            5ba8c56f-bec4-4337-9bc4-42a04025edc3  ONLINE       0     0     0
            df2be133-15da-4801-880d-e8272dd9f66d  ONLINE       0     0     0
            2e2576c6-f6a5-46a4-a81b-14e4e35c2551  ONLINE       0     0     0
          raidz2-6                                ONLINE       0     0     0
            87ca9464-1e71-4bed-93d3-f07ccef00d65  ONLINE       0     0     0
            c6091290-9990-4618-84e0-a92a83648e9e  ONLINE       0     0     0
            8e961f3b-470e-4d49-98da-60d8186afab6  ONLINE       0     0     0
            a7b9947b-1a66-4aac-9c59-25cc673e73d7  ONLINE       0     0     0
            efd33267-21bf-4647-80e4-62b3218241d5  ONLINE       0     0     0
            ecd56be3-be62-45df-9c20-f60337a4cd16  ONLINE       0     0     0
            927aed67-cb5d-4afe-8a16-f51492336bcc  ONLINE       0     0     0
            9582d73e-e2ec-4e09-899b-adee554868f5  ONLINE       0     0     0
            1d633f20-44eb-42d6-ae86-4cddbc155776  ONLINE       0     0     0
            46ddf912-20ec-470b-9ab2-66fdbd64132e  ONLINE       0     0     0
            b263089b-37e2-43b9-aee5-8cb6ec7a2126  ONLINE       0     0     0
          raidz2-8                                DEGRADED     0     0     0
            9727455391675130707                   UNAVAIL      0     0     0  was /dev/disk/by-partuuid/817a5fd8-b921-43f7-ad07-00c5057bea98
            0c0c7d3f-7e89-4de5-aac0-91141b005b80  ONLINE       0     0     0
            8b206b2e-ba0e-4833-9af2-dc24fec3a73c  ONLINE       0     0     0
            1a2eac09-a267-4fbf-8085-961208ea95ce  ONLINE       0     0     0
            9f304704-fb82-4faf-bf63-9c0d0f2a17e0  ONLINE       0     0     0
            db940a83-fb07-47b2-af7e-45e56a35cec0  ONLINE       0     0     0
            82d26dd0-4c68-43aa-9402-bdcd07bb7c53  ONLINE       0     0     0
            47137c7b-741d-44c0-85d3-80bd1a9c1957  ONLINE       0     0     0
            efe1320f-9114-4d5d-b896-57608a9ab368  ONLINE       0     0     0
            6efd4874-05f9-4300-bc6e-3588364fdbbf  ONLINE       0     0     0
            4ccace51-fcca-4cfb-b1a2-765d69efbc8d  ONLINE       0     0     0
          raidz2-9                                ONLINE       0     0     0
            63d22ff9-739d-46ab-9edb-64e225f0b5c4  ONLINE       0     0     0
            863bdd21-9f71-4c50-bb48-f7b6ab33294b  ONLINE       0     0     0
            9cbeec4f-9a8a-4ce3-9760-ae0b4988eafe  ONLINE       0     0     0
            4cd632f9-99c8-436c-a90e-66571af7fcfd  ONLINE       0     0     0
            b0491c13-964a-4e58-a539-243a3fac0779  ONLINE       0     0     0
            5fe968c0-da72-4f22-9892-d0e55524d9b2  ONLINE       0     0     0
            018e9361-dd8d-47da-8f0d-eddc317667c8  ONLINE       0     0     0
            f6079603-0f99-4563-8918-f0e55bbe4146  ONLINE       0     0     0
            c3e023db-10f8-488a-b5e9-887d9c925092  ONLINE       0     0     0
            407f09a1-3f7b-49a6-8183-ccc282f118c4  ONLINE       0     0     0
            6c961565-4add-48d7-9b1c-9a929d9dd319  ONLINE       0     0     0
          raidz2-10                               ONLINE       0     0     0
            4bf3d14f-fdc7-4f06-a6e4-3036282063a3  ONLINE       0     0     0
            c398f904-7374-4ba4-bba4-36ddca88b281  ONLINE       0     0     0
            840f2d36-c4ed-4d13-b35a-29ba6c862334  ONLINE       0     0     0
            49eb085e-dacb-4836-8e3c-c562eb92d04b  ONLINE       0     0     0
            90bae1e0-ec14-44f0-ba54-38a591f69a3d  ONLINE       0     0     0
            f214d84d-6861-481a-ab56-8ad4aef93f45  ONLINE       0     0     0
            39bbcd85-ff93-462a-ba9f-8fe8ed84da99  ONLINE       0     0     0
            4d70df60-83dc-4bbf-9f4a-6d06295e130c  ONLINE       0     0     0
            3c1581a8-cde6-4bec-a87e-8a94df9156c2  ONLINE       0     0     0
            4c35e3f9-4413-4027-ac34-c9f3cd20f08f  ONLINE       0     0     0
            b58b5035-a719-47b7-9280-c0bd818107c2  ONLINE       0     0     0
        dedup
          mirror-2                                ONLINE       0     0     0
            d8c51d4c-1dd3-4648-a990-fdf7d6972333  ONLINE       0     0     0
            abf1aa23-632f-4042-9490-103aa8529599  ONLINE       0     0     0
          mirror-7                                ONLINE       0     0     0
            e0944b41-cc69-4380-b1a3-971d19a47b49  ONLINE       0     0     0
            b7ce854b-f262-4c93-aed2-ac92f1846aa3  ONLINE       0     0     0

errors: No known data errors

 dedup: DDT entries 10432566445, size 589B on disk, 190B in core

bucket              allocated                       referenced         
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
     1    8.17G   1.02P    944T    951T    8.17G   1.02P    944T    951T
     2    1.43G    183T    171T    172T    2.97G    380T    355T    357T
     4     112M   12.6T   12.3T   12.4T     512M   54.1T   52.7T   53.4T
     8    1.05M   91.8G   67.7G   70.9G    11.4M    863G    613G    657G
    16    1.21M    154G   18.6G   24.3G    30.3M   3.76T    434G    578G
    32    1.52M    193G   24.9G   31.9G    65.1M   8.12T   1.06T   1.36T
    64     128K   15.7G   4.73G   5.19G    9.69M   1.18T    377G    412G
   128    12.4K   1.19G    492M    544M    2.03M    194G   75.9G   84.8G
   256    6.56K    187M   31.6M   81.9M    2.44M   64.0G   9.81G   28.8G
   512    3.11K   17.1M   4.57M   30.8M    1.79M   11.8G   3.01G   18.1G
    1K      841    101M   3.92M   8.28M    1.26M    156G   5.84G   12.5G
    2K       91   9.28M    588K   1.09M     280K   29.0G   1.74G   3.31G
    4K       50   5.39M    199K    494K     291K   31.1G   1.11G   2.77G
    8K       58   6.39M    466K    777K     550K   60.4G   4.64G   7.51G
   16K       44    690K   61.5K    402K     986K   15.1G   1.35G   8.80G
   32K       15    651K    283K    384K     584K   25.4G   11.3G   15.1G
   64K        3    130K   89.5K    110K     216K   9.92G   6.85G   8.24G
  128K        4    388K   15.5K   36.6K     754K   74.1G   2.86G   6.73G
  256K        3    256K   8.50K   27.4K    1.39M    116G   3.85G   12.7G
  512K       32      4M    128K    292K    18.1M   2.26T   72.4G    165G
    1M       15   1.75M   57.5K    137K    17.1M   1.94T   64.5G    157G
 Total    9.72G   1.21P   1.10P   1.11P    11.8G   1.46P   1.32P   1.33P


  pool: boot-pool
 state: ONLINE
config:

        NAME        STATE     READ WRITE CKSUM
        boot-pool   ONLINE       0     0     0
          sda3      ONLINE       0     0     0

errors: No known data errors

 dedup: no DDT entries
root@truenas[~]# 

Code:

root@truenas[~]# zpool iostat -v
                                            capacity     operations     bandwidth
pool                                      alloc   free   read  write   read  write
----------------------------------------  -----  -----  -----  -----  -----  -----
Archivio                                  1.47P   483T  9.26K    609   139M  3.64M
  raidz2-0                                 219T  1.49T     20      1  84.6K  11.8K
    b097df6a-2f27-4394-9730-76261871cec3      -      -      1      0  7.73K  1.08K
    0dd84988-a18f-4f5c-909f-460d6de433c0      -      -      1      0  7.71K  1.08K
    9ad1c3d8-ecf6-41f1-8a02-2a27441a08e8      -      -      1      0  7.75K  1.07K
    9e13868e-9418-4a26-b858-d480ce639fd5      -      -      1      0  7.72K  1.06K
    ac7d4716-ba83-4209-af42-ad7a1f7e8799      -      -      1      0  7.72K  1.07K
    9ad14a36-d5ad-40d3-bf11-40590853b7a8      -      -      1      0  7.65K  1.07K
    3062e26b-01c1-4233-8735-f235958c9d5e      -      -      1      0  7.63K  1.07K
    5015360e-990c-428a-a3c0-5ad57ea530f1      -      -      1      0  7.62K  1.08K
    cdf836cf-e017-4df8-bf4d-2b0e7b4bbf0a      -      -      1      0  7.71K  1.08K
    660d5ea7-049b-44be-819b-fc5866bad3ad      -      -      1      0  7.69K  1.08K
    361f3028-4da4-490c-a674-704918283cf4      -      -      1      0  7.71K  1.07K
  raidz2-1                                 218T  1.62T     17      0  73.1K  11.6K
    3e93718b-720e-4f5f-a7f4-7b3ea3750ae2      -      -      1      0  6.60K  1.06K
    17a10b38-da89-4090-bad9-e7b9b58d82a1      -      -      1      0  6.71K  1.07K
    260476a7-a757-4293-be9e-7f94d3ad1127      -      -      1      0  6.75K  1.05K
    e6800b36-8d0e-41f6-8eef-cd8bda0f423b      -      -      1      0  6.82K  1.06K
    f0bb6ca1-3a14-479a-a3a2-3f9db1126070      -      -      1      0  6.75K  1.05K
    d69e75a9-76a8-419f-8fb4-964a95633db4      -      -      1      0  6.60K  1.05K
    873cb9f9-4b20-4cd0-a464-71b5c91669a6      -      -      1      0  6.48K  1.05K
    9fe2f86b-3681-4772-bcd6-dac7961a6172      -      -      1      0  6.50K  1.06K
    fe92d53d-a151-450c-85b4-685ea1d3d82a      -      -      1      0  6.60K  1.06K
    fd8a246d-5346-4b84-baf6-c1169e9238f6      -      -      1      0  6.60K  1.05K
    716bcb7f-d18b-4e82-866b-7afabfa68029      -      -      1      0  6.64K  1.05K
  raidz2-3                                 219T  1.44T     12      1  53.5K  11.7K
    68a7a9b3-12b9-4738-9836-327b7e872bae      -      -      1      0  4.84K  1.07K
    851f1e90-d49a-4f44-93df-baf5cf368ad0      -      -      1      0  4.81K  1.07K
    8a8976b5-227a-4d2c-a7c5-b56e171e2186      -      -      1      0  4.85K  1.07K
    e4646155-a067-492d-b5f3-c17c8e400008      -      -      1      0  4.92K  1.08K
    ccfd693d-ccb2-4e87-b2fb-0458731d1c10      -      -      1      0  4.94K  1.07K
    a6c8d71a-26a4-40b0-aef1-7cd8a5ec31df      -      -      1      0  4.85K  1.06K
    27c1e6d6-274c-461f-b458-b9ecea0ab187      -      -      1      0  4.92K  1.06K
    ddaba7e8-5c4f-49db-86a8-3944f04d229c      -      -      1      0  4.84K  1.05K
    3980fb00-af74-47d0-bd6b-dc88de0e0be7      -      -      1      0  4.84K  1.06K
    63fb1f47-0d2a-4b11-a534-e75091418966      -      -      1      0  4.84K  1.07K
    ba1fa4e5-1729-4249-803b-725a5be6026d      -      -      1      0  4.87K  1.07K
  raidz2-4                                 218T  2.09T     17      1  73.6K  11.9K
    ad2112a5-834f-487f-8e04-647a503bf678      -      -      1      0  6.72K  1.09K
    709505c7-40bc-4489-bc77-2883871dc245      -      -      1      0  6.68K  1.08K
    0765d45e-aeed-4300-9d28-7d8b464c5b09      -      -      1      0  6.67K  1.08K
    eca420ec-f3f8-4c0a-b6b2-08341a752184      -      -      1      0  6.62K  1.08K
    131d1bdc-be96-4365-b9bb-71cb54b7e022      -      -      1      0  6.68K  1.08K
    f19a7a96-2496-43af-91b0-3c12d0d534da      -      -      1      0  6.77K  1.08K
    5b79e55d-6d4d-43d8-863a-c741809ef504      -      -      1      0  6.70K  1.08K
    11f95813-4191-4b17-876e-6fb03dbf313c      -      -      1      0  6.67K  1.08K
    15a8a606-f117-4d35-81d2-93fcec38c451      -      -      1      0  6.67K  1.08K
    83aa4cf6-fa4e-4600-9d88-74d6a99a6933      -      -      1      0  6.71K  1.07K
    daa9f691-7ca7-4888-afea-2952f6274496      -      -      1      0  6.69K  1.08K
  raidz2-5                                 218T  2.13T      5      1  27.1K  12.0K
    04056fc8-84f0-412f-8276-56fa5747f0b9      -      -      0      0  2.47K  1.09K
    f27444a7-f73b-45dc-91d3-07436e3d666b      -      -      0      0  2.48K  1.10K
    06cfc1b9-b7ab-4091-bedd-27d644bf9688      -      -      0      0  2.46K  1.10K
    297aca17-ffec-49e1-b70d-a9ff01289518      -      -      0      0  2.48K  1.09K
    91826ebe-cad1-4ed0-97f3-2b9fa4732ab6      -      -      0      0  2.44K  1.09K
    62b145da-f906-4096-ace3-7f5ddfe387ad      -      -      0      0  2.44K  1.09K
    3cc6e221-e537-4906-956f-3138ab840e62      -      -      0      0  2.44K  1.10K
    0ede001d-32c5-4c72-abee-493a88d495d8      -      -      0      0  2.45K  1.09K
    5ba8c56f-bec4-4337-9bc4-42a04025edc3      -      -      0      0  2.48K  1.10K
    df2be133-15da-4801-880d-e8272dd9f66d      -      -      0      0  2.46K  1.09K
    2e2576c6-f6a5-46a4-a81b-14e4e35c2551      -      -      0      0  2.48K  1.09K
  raidz2-6                                 218T  2.36T     19      6   169K  76.1K
    87ca9464-1e71-4bed-93d3-f07ccef00d65      -      -      1      0  15.1K  6.87K
    c6091290-9990-4618-84e0-a92a83648e9e      -      -      1      0  15.3K  6.90K
    8e961f3b-470e-4d49-98da-60d8186afab6      -      -      1      0  15.4K  6.92K
    a7b9947b-1a66-4aac-9c59-25cc673e73d7      -      -      1      0  15.6K  6.93K
    efd33267-21bf-4647-80e4-62b3218241d5      -      -      1      0  15.5K  6.93K
    ecd56be3-be62-45df-9c20-f60337a4cd16      -      -      1      0  15.3K  6.94K
    927aed67-cb5d-4afe-8a16-f51492336bcc      -      -      1      0  15.5K  6.96K
    9582d73e-e2ec-4e09-899b-adee554868f5      -      -      1      0  15.3K  6.94K
    1d633f20-44eb-42d6-ae86-4cddbc155776      -      -      1      0  15.3K  6.95K
    46ddf912-20ec-470b-9ab2-66fdbd64132e      -      -      1      0  15.2K  6.93K
    b263089b-37e2-43b9-aee5-8cb6ec7a2126      -      -      1      0  15.4K  6.88K
  raidz2-8                                 167T  52.6T     29     15   329K   178K
    9727455391675130707                       -      -      0      0      0      0
    0c0c7d3f-7e89-4de5-aac0-91141b005b80      -      -      2      1  30.2K  17.8K
    8b206b2e-ba0e-4833-9af2-dc24fec3a73c      -      -      2      1  31.8K  17.7K
    1a2eac09-a267-4fbf-8085-961208ea95ce      -      -      2      1  31.6K  17.7K
    9f304704-fb82-4faf-bf63-9c0d0f2a17e0      -      -      2      1  32.0K  17.7K
    db940a83-fb07-47b2-af7e-45e56a35cec0      -      -      2      1  31.9K  17.8K
    82d26dd0-4c68-43aa-9402-bdcd07bb7c53      -      -      2      1  33.1K  17.8K
    47137c7b-741d-44c0-85d3-80bd1a9c1957      -      -      3      1  34.0K  17.9K
    efe1320f-9114-4d5d-b896-57608a9ab368      -      -      3      1  35.2K  17.8K
    6efd4874-05f9-4300-bc6e-3588364fdbbf      -      -      3      1  35.8K  17.8K
    4ccace51-fcca-4cfb-b1a2-765d69efbc8d      -      -      3      1  33.1K  17.8K
  raidz2-9                                11.8T   208T     35     14   286K   255K
    63d22ff9-739d-46ab-9edb-64e225f0b5c4      -      -      3      1  26.2K  23.2K
    863bdd21-9f71-4c50-bb48-f7b6ab33294b      -      -      3      1  25.8K  23.3K
    9cbeec4f-9a8a-4ce3-9760-ae0b4988eafe      -      -      3      1  25.6K  23.3K
    4cd632f9-99c8-436c-a90e-66571af7fcfd      -      -      3      1  25.8K  23.2K
    b0491c13-964a-4e58-a539-243a3fac0779      -      -      3      1  26.0K  23.2K
    5fe968c0-da72-4f22-9892-d0e55524d9b2      -      -      3      1  26.2K  23.2K
    018e9361-dd8d-47da-8f0d-eddc317667c8      -      -      3      1  26.4K  23.3K
    f6079603-0f99-4563-8918-f0e55bbe4146      -      -      3      1  26.0K  23.2K
    c3e023db-10f8-488a-b5e9-887d9c925092      -      -      3      1  26.0K  23.2K
    407f09a1-3f7b-49a6-8183-ccc282f118c4      -      -      3      1  26.1K  23.1K
    6c961565-4add-48d7-9b1c-9a929d9dd319      -      -      3      1  26.5K  23.2K
  raidz2-10                               10.4T   210T     32     13   269K   246K
    4bf3d14f-fdc7-4f06-a6e4-3036282063a3      -      -      2      1  24.7K  22.4K
    c398f904-7374-4ba4-bba4-36ddca88b281      -      -      2      1  24.4K  22.4K
    840f2d36-c4ed-4d13-b35a-29ba6c862334      -      -      2      1  24.2K  22.4K
    49eb085e-dacb-4836-8e3c-c562eb92d04b      -      -      2      1  24.9K  22.4K
    90bae1e0-ec14-44f0-ba54-38a591f69a3d      -      -      3      1  24.4K  22.4K
    f214d84d-6861-481a-ab56-8ad4aef93f45      -      -      2      1  24.4K  22.4K
    39bbcd85-ff93-462a-ba9f-8fe8ed84da99      -      -      2      1  24.3K  22.4K
    4d70df60-83dc-4bbf-9f4a-6d06295e130c      -      -      2      1  24.3K  22.4K
    3c1581a8-cde6-4bec-a87e-8a94df9156c2      -      -      2      1  24.4K  22.4K
    4c35e3f9-4413-4027-ac34-c9f3cd20f08f      -      -      2      1  24.4K  22.4K
    b58b5035-a719-47b7-9280-c0bd818107c2      -      -      3      1  24.3K  22.4K
dedup                                         -      -      -      -      -      -
  mirror-2                                2.80T   700G  4.56K    283  72.5M  1.40M
    d8c51d4c-1dd3-4648-a990-fdf7d6972333      -      -  2.28K    142  36.2M   717K
    abf1aa23-632f-4042-9490-103aa8529599      -      -  2.28K    141  36.3M   717K
  mirror-7                                2.80T   699G  4.52K    270  65.5M  1.45M
    e0944b41-cc69-4380-b1a3-971d19a47b49      -      -  2.26K    135  32.8M   740K
    b7ce854b-f262-4c93-aed2-ac92f1846aa3      -      -  2.26K    135  32.7M   740K
----------------------------------------  -----  -----  -----  -----  -----  -----
boot-pool                                 2.34G   446G      2     10  32.3K   127K
  sda3                                    2.34G   446G      2     10  32.3K   127K
----------------------

 

[Etorix]

Data integrity is definitely something I want to maintain.

Which raises an additional issue: How do you backup this monster? (You have a backup, right?)
The dedup vdevs are simple 2-way mirrors, so losing two drives could destroy the pool.

At 1.2:1 I question whether dedupe is even worth it. Mind you extricating oneself from the dedupe hole the OP is down might have an "entertainment" value of high

Not really. It would just have the high cost of building another pool of at least 2.5 PB.

 

[HoneyBadger]

You're getting a high amount of reads from the SSDs with very little actual data being written, which is indicative of a deduplication bottleneck.

Please double-check my math, but I believe that since January 31st:

Data has grown at a rate of approximately 10T/day
On-disk deduplication tables have grown 34.6G/day
Deduplication RAM demand has grown by 10.8G/day

Of these, the RAM demand is likely the most challenging obstacle as your dedup tables presently want to consume 1.8T of RAM - and that would leave no room for caching and serving actual data. Even if you upgraded to the 1.5T of RAM mentioned earlier, this would mitigate but not entirely prevent the issue - and at the calculated rates above it's only a matter of time before you approach that limitation.

I'd love to get in touch with you directly to see if iX can help you solve this. Feel free to DM me.