I've been deep-diving into Truenas and zfs recently, and I'm seeing some perplexing behavior specifically with dedupe.
With dedupe disabled, performance easily saturates the 1gbe connection in read and write for a 50gbe transfer, processer flutters up to 10-20%.
-deleted and Rebuilt dataset enabling dedupe-
Performance is good for much of the transfer, but intermittently drops to almost nothing for 30+ seconds on write, while hard drives thrash wildly. Memory and processor load appear unchanged.
-Added cheap 16gig ngff drive on usb as a dedicated ddt vdev-
Performance is back up to non-dedupe levels, with no more apparent spikes and hitches, hard drives are much happier.
I'm under the impression that the dedupe tables should be stored in ram when no dedicated vdev is available, However experience would indicate that this is not the case.
I feel like there is some part of the dedupe implementation that I'm fundamentally misunderstanding.
Any insight would be appreciated.
Thanks!
Tangentially: I'm hoping to scavenge a couple 10gig nics for further performance testing. 1gig is quite obviously my main bottleneck at the moment.
Hardware:
Dell r620, E5-2630(v1), 32gb memory, 3x mirrored vdevs of 2 10k rpm sas drives (6 total), zstd compression. +/- one cheap Kingston rbu-sns4151s3/16g on a usb adapter.
Disclaimer: This is a purely test system. I slapped some old hardware together specifically for experimenting.
With dedupe disabled, performance easily saturates the 1gbe connection in read and write for a 50gbe transfer, processer flutters up to 10-20%.
-deleted and Rebuilt dataset enabling dedupe-
Performance is good for much of the transfer, but intermittently drops to almost nothing for 30+ seconds on write, while hard drives thrash wildly. Memory and processor load appear unchanged.
-Added cheap 16gig ngff drive on usb as a dedicated ddt vdev-
Performance is back up to non-dedupe levels, with no more apparent spikes and hitches, hard drives are much happier.
I'm under the impression that the dedupe tables should be stored in ram when no dedicated vdev is available, However experience would indicate that this is not the case.
I feel like there is some part of the dedupe implementation that I'm fundamentally misunderstanding.
Any insight would be appreciated.
Thanks!
Tangentially: I'm hoping to scavenge a couple 10gig nics for further performance testing. 1gig is quite obviously my main bottleneck at the moment.
Hardware:
Dell r620, E5-2630(v1), 32gb memory, 3x mirrored vdevs of 2 10k rpm sas drives (6 total), zstd compression. +/- one cheap Kingston rbu-sns4151s3/16g on a usb adapter.
Disclaimer: This is a purely test system. I slapped some old hardware together specifically for experimenting.