Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.
Resource icon

My experiments in building a home server capable of handling fast + consistent deduplication

Stilez

Senior Member
Joined
Apr 8, 2016
Messages
506
Stilez submitted a new resource:

My experiments in building a home server capable of handling fast + consistent deduplication

AIM:

To help people looking at deduplication on TrueNAS 12+, what I've found on the way making it work on mine.

On sustained mixed loads, such as 50GB+ file copies and multiple transfers, using TrueNAS 12 with a deduped pool and default config, I now get almost totally consistent and reliable ~300 MB/sec client-server, server-client, and server-server, right now, and couldn't get close to that before, on 11.3 and earlier.

BACKGROUND.... WHY WOULD I WANT DEDUP ANYWAY...
Read more about this resource...
 

morganL

Captain Morgan
Administrator
Moderator
iXsystems
Joined
Mar 10, 2018
Messages
470
Looking forward to reading the next installments.. don't leave us hanging.
 

Stilez

Senior Member
Joined
Apr 8, 2016
Messages
506
Its moving the pool over the next few days, while I rebuild my workstation. Upcoming stuff I want to try, then write up......

Tunables that help performance
Testing out iSCSI as an alternative to a local disk
Seeing how snapshot deletion speed compares.

I spent years trying to get good performance, with dedup. Now I can, I want to put what I've learned about doing it, up there.

But its got a bunch of replication and disk resilvering to do before then.

Omens are good. A mass recursive delete of 957 snapshots, across 17 datasets (16269 snaps total) just took 48 minutes. Thats a dedup snap destroy rate down from tens of seconds per snap, before special vdev+Optane, to 1/6 of a second each after moving to 12-BETA and adding them.
 
Last edited:
Top