TrueNAS Scale Dedup, Backup and Performance

[b1ackwid0w]

Hi, I've been using TrueNAS for my file server for years and recently the old file server is almost 80% full. Thus, I'm planning to upgrade that to a new system.

The new system is running on an esxi hypervisor with network adapter and HBA passthrough with 64 cores and 128 GB RAM. The pool consists of 8x16TB drives in a Z2 vdev and a mirror 900G nvme dedup vdev. Is this pool layout correct?

I also tried a pure SSD 6drive Z2 vdev with dedup on and no dedup vdev. Both HDD and SSD pool report same performance at around 4.6GiB read and 2.5GiB write. How on earth my HDD pool so fast?

I did find my inventory allows me to build another backup server with budget of another 64 cores and 128 GB RAM in a different server chassis with 16 bays. I'm planning to use that as a active backup to my 8 drives pool. since it has 16 bays and I just have 8 16TB drives for the back up. I decide to make a 8 drive Z2 vdev and upgrade to a strip of 8 drive Z2 vdev in the future. Does the backup must have dedup turned on?

Thanks

 

[sretalla]

128 GB RAM

8x16TB drives

I'm fairly sure that's not enough RAM for a Dedupe setup to work properly in the long term.

I also tried a pure SSD 6drive Z2 vdev with dedup on and no dedup vdev. Both HDD and SSD pool report same performance at around 4.6GiB read and 2.5GiB write. How on earth my HDD pool so fast?

With that much RAM (not yet busy doing dedupe), are you sure you're seeing disk performance? you would need to test with very large copies.

Does the backup must have dedup turned on?

As I understand it, it will depend on how you do that backup... rsync, not necessary to have dedupe on the backup side... replication task, I think it will inherit the dedupe from the source.

 

[Etorix]

I'm fairly sure that's not enough RAM for a Dedupe setup to work properly in the long term.

At the typical rule-of-thumb of 5 GB RAM per duplicated data TB, no. But with the dedup table hosted on a special vdev, it may be fine. @b1ackwid0w should watch out for ARC hit rate and check that the dedup table is not evicting useful data in ARC.

As I understand it, it will depend on how you do that backup... rsync, not necessary to have dedupe on the backup side... replication task, I think it will inherit the dedupe from the source.

I can confirm: The replicated dataset automatically inherits dedup.

I did find my inventory allows me to build another backup server with budget of another 64 cores and 128 GB RAM in a different server chassis with 16 bays.

It must be a nice situation to have 64-core CPUs sitting in inventory, waiting to be thrown in some random project…

 

[b1ackwid0w]

I'm fairly sure that's not enough RAM for a Dedupe setup to work properly in the long term.

I do have another 256 GB RAM coming soon but still might not be large enough. Maybe 768? That's the largest amount of ram my server can handle with 32GB RDIMM.

With that much RAM (not yet busy doing dedupe), are you sure you're seeing disk performance? you would need to test with very large copies.

The test was carried out with just 32GB allocated to the TrueNAS VM and the DD test was around 80GB I tested this many time and always get the same result. And I tried this with 10GBE and both pools saturated this connection. I'm planning to upgrade to 25G uplink very soon.

It must be a nice situation to have 64-core CPUs sitting in inventory, waiting to be thrown in some random project…

Intel E5 2696 V4 is so cheap right now. The price I paid for this is just around 150USD each and the backup machine I believe has E5 2683 V4which is just $50 each. I did setup a windows VM to run simulation. Didn't have any time to test this yet.

At the typical rule-of-thumb of 5 GB RAM per duplicated data TB, no. But with the dedup table hosted on a special vdev, it may be fine. @b1ackwid0w should watch out for ARC hit rate and check that the dedup table is not evicting useful data in ARC.

Since the DDT was 64K block I really don't want this to be stored on HDD array. TrueNAS official documents says dedup on a full flash array should be OK without any special vdev. But still, that large amount of small block data need optane at least or nvdimm.


My current setup is just 8x16TB HDD Z2 data vdev. and a mirrored 960GB NVME dedup vdev. I don't see any necessity for a cache or log vdev. Since I can get at least 256GB RAM for this VM. And for now. My main NAS has around 40TB non-deduped data. I'm planning to migrate into my new server

Btw the drive I’m planning to use for the backup server is currently in use in another project. Thus the layout for the backup server isn’t confirmed yet. But it will have 8x16TB z2 vdev. And will expand to a striped 8x16TB z2 vdevs in the future.
Any other suggestions for this layout?