Dedupe - What I'm doing wrong? (ratio not what was expected)

[SoMBrA]

Hello! Hola!
First of all just want to thank everyone that is always around here helping people, like me, that have been around reading but never had to regsiter because we could always find the answers ;)

The plan:
I have 2 sons, and we all play the same games (Battle.NET, Steam, EPIC, EA) so I always wanted to try dedupe (because I think for this use case is a good option!)
So I have 3 "gaming rigs" that are connected to my Truenas Server, the gaming rigs have 10Gb cards ( the only issue is that we are connecting to the server via a Ubiquiti FLEX switch, so we are sharing the 10 Gb uplink, but is something that I will solve in the future)
I've created 3 iSCSI drives, and all of have our libraries in those drives (so each one has it's own iSCSI drive, just to make it clear).
* Side note here, does the folder structure matter? For example, PC1 has Destiny2 folder in the root, and PC2 and PC3 have it under SteamLibrary\Destiny2

My Server:
- TrueNAS-SCALE-23.10.0.1
- Threadripper 1950x
- 64 GB RAM (NON ECC)
- ASUS Strix X399-E Gaming
- Inspur LSI SAS3008 9300-8i SATA / SAS HBA Controller 12Gbps IT-Mode
- Intel X520-DA2 Dual-Port 2x 10GbE-LAN SFP+ PCIe x8 E10G42BTDA
- SUN 7069200 F80 FLASH ACCELERATOR 800GB (4x200GB) PCI-Ex8

The pools:

This is the main pool that I use for other stuff (NextCloud, and personal stuff)

The cache is a NVME2 drive.

"Gaming pool", the one with the deduplication

4 Kingston A400 (2 x 2 Mirror)
2 Samsung (1 x 2 Mirror)
Sun F80 Flash (4 x 200) as the Dedup VDEVs. I was forced to use Mirror for the Data VDEVs otherwise I couldn't set the Dedup VDEVs as mirrors.
On a older TrueNAS version I could enable Dedup VDEV in Mirror but the Data VDEV as stripes (No redudancy but since it's for gaming I was happy with the extra storage, and the mirror to fast access to dedup tables)

The iSCSI share have only 680 Gb of space, to keep it under the 80% space, and that's why I wanted to put them as stripes.

The warning is because I'm missing a 3rd mirror for the Dedup VDEVs, not sure if this is the problem or is this the right way to do this?

Each Zvol has the same compression level lz4 (inherit) and Deduplication ON (inherit)

All the data was copied AFTER the setup was complete

Thanks for your time!

 

[Arwen]

* Side note here, does the folder structure matter? For example, PC1 has Destiny2 folder in the root, and PC2 and PC3 have it under SteamLibrary\Destiny2

In some ways, no, the directory structure of the file system in the iSCSI volume should not mater. But, it should be the same file system type, like NTFS.

Sun F80 Flash (4 x 200) as the Dedup VDEVs. I was forced to use Mirror for the Data VDEVs otherwise I couldn't set the Dedup VDEVs as mirrors.
On a older TrueNAS version I could enable Dedup VDEV in Mirror but the Data VDEV as stripes (No redudancy but since it's for gaming I was happy with the extra storage, and the mirror to fast access to dedup tables)

This is a bug in the TrueNAS SCALE GUI, fix is already in the works.

Their are lots of things that affect De-Dup, and for the casual user, it is not recommended. I did write a Resource on the subject, steering casual users away from De-Dup. But, some of the information is also useful for people actually wanting to use De-Dup;

Resource - ZFS de-Duplication - Or why you shouldn't use de-dup

The TrueNAS forums occasionally have people who come across ZFS de-duplicate, and want to investigate its use. Or think it is a good idea, and want to implement it. Here are some suggested configuration details: Understand that you need CPU...

www.truenas.com

Other than above, I don't have any suggests or reasons why your De-Dup ratio is so low.

 

[SoMBrA]

In some ways, no, the directory structure of the file system in the iSCSI volume should not mater. But, it should be the same file system type, like NTFS.

It's NTFS, so all good there.

This is a bug in the TrueNAS SCALE GUI, fix is already in the works.

Good to know! I was confused.

Their are lots of things that affect De-Dup, and for the casual user, it is not recommended. I did write a Resource on the subject, steering casual users away from De-Dup. But, some of the information is also useful for people actually wanting to use De-Dup;

Resource - ZFS de-Duplication - Or why you shouldn't use de-dup

The TrueNAS forums occasionally have people who come across ZFS de-duplicate, and want to investigate its use. Or think it is a good idea, and want to implement it. Here are some suggested configuration details: Understand that you need CPU...

www.truenas.com

I did saw your post when I was learning about dedup, but still wanted to experiment.

Other than above, I don't have any suggests or reasons why your De-Dup ratio is so low.

I think I will try to delete everything and try with a very basic setup, see if that makes any difference.

 

[Arwen]

It might help if the parent dataset was the one that activated De-Dup:

pool/dataset_media pool/dataset_iscsi - This one has De-Dup enabled pool/dataset_iscsi/PC1_vol pool/dataset_iscsi/PC2_vol pool/dataset_iscsi/PC3_vol

and the last 3 inherit De-Dup.

I threw in a media dataset just so it is clear you don't use the iSCSI dataset for anything non-De-Dup. Plus, it is always standard practice not to use the top level dataset in a pool, (aka "pool" dataset), for any purpose other than creating child datasets.

 

[Heracles]

It's NTFS, so all good there.

Are you using disk level encryption like bitlocker ? If you do, there will be no possibility for dedup to achieve anything significant. Same thing for compression. The content will be modified in the client before being sent to TrueNAS and each client is subject to produce different results.

Another option would be to rely on snapshots and clones instead.

Use your iSCSI only for the OS part of your 3 gaming computer.
Then, you install your software and data in another dataset (file-level, so not iSCSI).
You snapshot this dataset and clone it twice.
You mount one clone in gaming PC no2 and one in gaming PC no3

You fix the install in gaming 2 and 3 and voilà.

I consider that dedup is good only for troubles and that it should be avoided in all cases.

 

[PhilD13]

I would think dedupe would be a really bad idea for a system where gaming is involved as in general it slows even the fastest system down, requires lots of horsepower and possibly massive amounts of ram depending upon dedupe ratio. My understanding is smaller ratios like shown would a result in a huge table needing to be kept in ram and the system shown would not have near enough physical ram. Maybe the gaming data due to DRM or the way it is structured, etc. doesn't actually have enough duplicate blocks to dedupe resulting in a low ratio and so is not a good canidiate for that.

 

[SoMBrA]

Thanks everyone for the tips!

So now It's working. I did delete everything and followed some of your tips, like:

It might help if the parent dataset was the one that activated De-Dup:

...so it is clear you don't use the iSCSI dataset for anything non-De-Dup. Plus, it is always standard practice not to use the top level dataset in a pool, (aka "pool" dataset), for any purpose other than creating child datasets.

Are you using disk level encryption like bitlocker ? If you do, there will be no possibility for dedup to achieve anything significant. Same thing for compression. The content will be modified in the client before being sent to TrueNAS and each client is subject to produce different results.

I was not using encryption, but I did disable compression (lz4 was inherited)
I also did create the zvol via cli. Blocksize was set to 1M and then I format the iSCSI drives with NTFS 1M (from windows)

 

[jgreco]

You might also want to look at the block storage guide.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com