More dedup questions & block size issues.

[ThisTruenasUser]

Hi all.
I have been testing out dedup on an old machine.
The spec is 24GB 16GB optane for dedup and 1TB for pool.
I bought two old 16GB optane for this purpose. They were not exactly expensive.

I used to have freenas on an old machine (now the one I am testing with) , now as a virtual machine in my server, which is proxmox. I am looking to use another option for storage, maybe truenas scale as hypervisor, or just a turnkey solution in proxmox.
In there are 5x4TB in raidz 1 for about 16TB storage.
When setting up a pool, it says you need X GB dedup storage for every X TB of storage.
This is exactly what I have , which is the problem.
So initially setup two identicel 600GB zvols, with dedup turned on, with no compression.
Then two iscsi shares, both the same with standard values & the 4k modern os setup.

A windows 10 virtual machine copied over some data four times, two to each attached iscsi drive.

here is the output of:
sudo zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
OneTB 941G 85.2G 856G - - 0% 9% 3.47x ONLINE /mnt
b9f1db3c-e1bc-11ed-8ada-3d25fb7fec7d 930G 80.9G 847G - - 0% 8.71% - ONLINE
dedup - - - - - - - - -
b8a9af54-e1bc-11ed-8ada-3d25fb7fec7d 13.4G 4.32G 8.68G - - 0% 33.2% - ONLINE
boot-pool 31G 2.65G 28.4G - - 0% 8% 1.00x ONLINE -
sdb3 31.5G 2.65G 28.4G - - 0% 8.54% - ONLINE

The dedup works well (3.47) , but the dedup data usage s way too high (4.32GB if I reed correctly)
It is with about 4 x 80GB copied over.

I then had a different idea.

I recreated destroyed & recreated the pool & zvols, with 128 block sizes.
In windows 11, formatted in ntfs with 128k block size.
New iscsi shares were created also.

I copied data over again, 196GB unique data to each disk.

The new output of: sudo zpool list -v:
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
OneTBVthree 941G 195G 746G - - 0% 20% 2.03x ONLINE /mnt
02f4f10b-2b2c-4ff3-8d02-cbb044779692 930G 194G 734G - - 0% 20.9% - ONLINE
dedup - - - - - - - - -
47ce25eb-de6e-47a4-b8ef-ed921f5d64b6 13.4G 1.15G 11.8G - - 0% 8.88% - ONLINE
boot-pool 31G 2.65G 28.3G - - 0% 8% 1.00x ONLINE -
sda3 31.5G 2.65G 28.3G - - 0% 8.54% - ONLINE

So it seems the block size increase has helped the situation quite a bit. The 2.03x ratio is better than expected
Dedup use is much less (1.15GB) if I read correctly.

It is certainly an improvement, but still not good enough.

The purpose of all this is to have multiple ISCSI devices for games storage. All will be pretty much the same data. I want to have maybe 8-12 TB. This will be on my main server. It will hopefully be

When formatiing in windows, the maximum allowed is 2MB block size.
I want to create multiple dedup zvols of 2MB block size, share then trhough ISCSI and format with 2MB block sizes in ntfs. I expect turning on verify with deduplication would be a good idea there.

The maximum allowed in the GUI is 128k. This is a big problem.
How is this remedied?

I wish to create them, copy data over again & test the deduplication.
I expect the dedup data would be vastly reduced.


Is there some tweak in truenas scale allowing me to do this.
Another option could be to create the zvol though the cli with root access. it is probably not recommended
I could speculation with something like: zfs create mypool/newzvol -b 2M ........


Deduplication is the only reason I am looking to contirue to use truenas.
If it works with the hardware I have, plan to replace proxmox as a hypervisor.
There are lots of posts about how people dislike it. I wish to use it with the sutiable configuration if I can


So iscsi & deduplication are what I need to work with the hardware I have. If not, have no reason to use truenas at all.
If it means having to buy more storage for game storage, would not be using it with truenas obviously.

To sum up what I need is a way of forcing 2MB block sizes in a zvol.

Useful info is appreciated.

Thanks

 

[sretalla]

To sum up what I need is a way of forcing 2MB block sizes in a zvol.

That would be the job of the consumer...

The OpenZFS create command says this:
'volblocksize' must be power of 2 from 512B to 1M

 

[Arwen]

Some suggested reading about ZFS De-Dup. This is not to stop you, but help anyone trying to use De-Dup feature of ZFS.

Resource - ZFS de-Duplication - Or why you shouldn't use de-dup

The TrueNAS forums occasionally have people who come across ZFS de-duplicate, and want to investigate its use. Or think it is a good idea, and want to implement it. Here are some suggested configuration details: Understand that you need CPU...

www.truenas.com

 

[ThisTruenasUser]

That would be the job of the consumer...

The OpenZFS create command says this:
'volblocksize' must be power of 2 from 512B to 1M

Thank you.
Could you let me know the site this is from.
Asuuming this is true, how do you creat a zvol with 1MB block size.
In the gui, it is 128k max.
If it means forcing the entire pool to use 1MB block sizes, that is fine also.

 

[ThisTruenasUser]

Some suggested reading about ZFS De-Dup. This is not to stop you, but help anyone trying to use De-Dup feature of ZFS.

Resource - ZFS de-Duplication - Or why you shouldn't use de-dup

The TrueNAS forums occasionally have people who come across ZFS de-duplicate, and want to investigate its use. Or think it is a good idea, and want to implement it. Here are some suggested configuration details: Understand that you need CPU...

www.truenas.com

Thank you.
I have read through that already. I am aware of how resource intensive it can be if mis configured.

So in short, the longer the block size, the less dedup data it seems to use.
Here is my little table which is useful to me:
Unique Data ISCSI Block size Dedup Use Projected for 8TB
80GB 16k 4.32GB 442GB
196GB 128k 1.15GB 48GB

So going to 1MB block sizes (apparently the maximum allowed) , maybe 4 x less, which is about 12GB
It should be suitable for the 16GB intel optane drives if the pattern continues.


When shared , will be ISCSI and formatted to 1MB blocks.

I need to know how to create the 1MB block sizes in a zvol. The maximum is apparently 128k.
If it means forcing a pool with 1MB block sizes, is fine by be.

How is this done?

If not it means purchasing storage will get another hard drive for games, and will have no use for truenas at all.

thank you.

 

[ThisTruenasUser]

Oh and I need to meantion, I meant to 16, 128k and 1MB 'record' sizes in a zvol.
I think the block and record size are technically different, but are unsure how.

So I would like a 1MB record size choice when creating a zvol.

 

[Patrick M. Hausen]

zfs create -V <volumesize> -o volblocksize=<volblocksize> <pool>/path/to/zvol

 

[Arwen]

Hmmm, my reading of the ZFS manual page zfsprops had this to say about the subject;

volblocksize For volumes, specifies the block size of the volume. The blocksize cannot be changed once the volume has been written, so it should be set at volume creation time. The default blocksize for volumes is 8 Kbytes. Any power of 2 from 512 bytes to 128 Kbytes is valid. This property can also be referred to by its shortened column name, volblock.
This is from ZFS version 2.1.9 on my Linux desktop.

That states nothing larger than 128KB is available in a zVol. The values greater than 128KB appear to relate only to ZFS Datasets, and NOT zVois.

And yes, I checked the pool feature flag feature@large_blocks, it only applies to Datasets, from zpool-features manual page;

large_blocks GUID org.open-zfs:large_blocks DEPENDENCIES extensible_dataset READ-ONLY COMPATIBLE no This feature allows the record size on a dataset to be set larger than 128kB. This feature becomes active once a dataset contains a file with a block size larger than 128kB, and will return to being enabled once all filesystems that have ever had their recordsize larger than 128kB are destroyed.
Note the reference to "record size" and NOT "volblocksize".


This sort of makes sense. ZFS zVols are intended to mimic disks, even though it is through software. I don't know of any disks that have 1MByte block sizes, let alone 2MByte block sizes.

Do any SAN storage arrays that supply pseudo disks support 1MByte / 2MByte block sizes?

 

[ThisTruenasUser]

zfs create -V <volumesize> -o volblocksize=<volblocksize> <pool>/path/to/zvol

Thank you very much for the info.

I wish to make it a sparse drive, deduplication & deduplication verify turned on.
The sparse option I think is adding '-s'.
The other two I am having trouble finding.
I am hoping you know of then.


I am very weary of creating it in the command line & then using the GUI to change the values, as it may break something.
Thank you

 

[Arwen]

Yes, the -s is the correct option to enable sparse zVols.

The dedup is enabled / checksum set with;
zfs create -s -V <volumesize> -o volblocksize=<volblocksize> -o dedup=DEDUP_OPTION,verify <pool>/path/to/zvol

The dedup options are defined in the zfsprops manual page:
dedup=off|on|verify|sha256[,verify]|sha512[,verify]|skein[,verify]|edonr,verify Configures deduplication for a dataset. The default value is off. The default deduplication checksum is sha256 (this may change in the future). When dedup is enabled, the checksum defined here overrides the checksum property. Setting the value to verify has the same effect as the setting sha256,verify. If set to verify, ZFS will do a byte-to-byte comparison in case of two blocks having the same signature to make sure the block contents are identical. Specifying verify is mandatory for the edonr algorithm. Unless necessary, deduplication should not be enabled on a system. See the Deduplication section of zfsconcepts(7).

 

[ThisTruenasUser]

Great thank you for all the info.

I created the zvols and tested them out. At first it was terrible write performance, but then realized I am testing with a very old & slow hard drive and have sync turned on in both zvols.
So destroyed everything & started again.

So with two ISCSI shares, copied a lot of the same & multiple data to each. The data was some games from my games machine with a windows share.
Both block size 1MB and formatted in 1MB block sizes in the windows 11 virtual machine.

So:
admin@truenas[~]$ sudo zpool list -v
NAME SIZE ALLOC FREE CKPOINT EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
boot-pool 31G 2.65G 28.4G - - 0% 8% 1.00x ONLINE -
sdc3 31.5G 2.65G 28.4G - - 0% 8.54% - ONLINE
main 941G 299G 642G - - 0% 31% 2.47x ONLINE /mnt
5d7a406e-dc52-49f4-9978-5bfee96d22aa 930G 298G 630G - - 0% 32.1% - ONLINE
dedup - - - - - - - - -
99b55c4d-a62f-4621-bf1c-930a8816b583 13.4G 223M 12.8G - - 8% 1.67% - ONLINE

So about 300GB of unique data it seems, needing 223MB of deduplication storage. that is about the ratio of XGB dedup for each XTB storage, as hoped.
The dedup ratio is 2.47 is seems.


Good result.
This is using a 16GB intel optane drive for dedup.
They are very cheap on ebay. I may get a couple more for spares.
It seems I may be able to have multiple sparse iscsi drives dor games up to 12TB.

Thanks again for all the info.