zdb command not giving expected results

titan_rw · Mar 18, 2016

stilgars said:
Except the standard freenas-boot one, zdb doesn't seem to find my two pools. I did not use the GUI to create them but the command line (I know my SLOG is not mirrored, this is temporary)

Have you added -U to zdb to tell it where the cache file is?

Code:

zdb

Shows only the boot pool.

Code:

zdb -U /data/zfs/zpool.cache

Shows all pools except boot.

Casey Duplantis · Apr 16, 2016

Code:

zdb

Shows only the boot pool.

Code:

zdb -U /data/zfs/zpool.cache

Shows all pools except boot.[/QUOTE]

I can confirm the same behavior for my imported pool which was created on FreeBSD, imported and upgraded.

Code:

# zpool get version 

NAME          PROPERTY  VALUE    SOURCE

freenas-boot  version   -        default

tank          version   -        default

Gashy · Aug 11, 2016

Add the -r switch to your argument: zdb -e -

Jared Smith said:
I'm new to FreeNAS. Decided to doing some benchmarking. I rsynced about 375 GB data to a dataset without dedup enabled (using NFS share). Then did the same to a data set with dedup enabled. I expected to run some zdb commands to get some finer grained statistics but when I do it does not recognize my zpool.

[root@freenas] ~# zpool list
NAME SIZE ALLOC FREE EXPANDSZ FRAG CAP DEDUP HEALTH ALTROOT
freenas-boot 14.9G 482M 14.4G - - 3% 1.00x ONLINE -
storage 5.44T 753G 4.70T - 10% 13% 1.79x ONLINE /mnt
[root@freenas] ~# zdb -D storage
zdb: can't open 'storage': No such file or directory
[root@freenas] ~#

I found a similar report in a bug report

https://bugs.freenas.org/issues/7877

Should I be able to see output from the zdb command for my zpool "storage"?

zdb commands work just fine for zpool "freenas-boot"

Any thoughts?

Add the -e switch to your argument: zdb -e -D storage

~# zdb -D TANK
zdb: can't open 'TANK': No such file or directory

~# zdb -e -D TANK
All DDTs are empty

Robert Trevellyan · Aug 11, 2016

What is the output of zpool status (please use CODE tags)?

maurertodd · Apr 4, 2017

Helpful thread. I was getting the same error for my pool netbackup1.

The command below worked for me:

zdb -U /data/zfs/zpool.cache -S netbackup1

devnullius · Oct 1, 2018

I'm stuck in the same boat :/

zdb -U /data/zfs/zpool.cache -S ZFS_8x_3TB_RAIDz2_pool - just hangs

zdb -U /data/zfs/zpool.cache gives:

Code:

VOLU10TB:
	version: 5000
	name: 'VOLU10TB'
	state: 0
	txg: 2123163
	pool_guid: 16365684983299924960
	hostid: 3350735314
	hostname: ''
	com.delphix:has_per_vdev_zaps
	vdev_children: 1
	vdev_tree:
		type: 'root'
		id: 0
		guid: 16365684983299924960
		children[0]:
			type: 'disk'
			id: 0
			guid: 6974793590789398571
			path: '/dev/gptid/6baca59b-0553-11e8-a1db-0025901159d4'
			whole_disk: 1
			metaslab_array: 38
			metaslab_shift: 36
			ashift: 12
			asize: 9998678884352
			is_log: 0
			DTL: 438
			create_txg: 4
			com.delphix:vdev_zap_leaf: 36
			com.delphix:vdev_zap_top: 37
	features_for_read:
		com.delphix:hole_birth
		com.delphix:embedded_data
ZFS_8x_3TB_RAIDz2_pool:
	version: 5000
	name: 'ZFS_8x_3TB_RAIDz2_pool'
	state: 0
	txg: 2170877
	pool_guid: 4057754529204707565
	hostid: 3350735314
	hostname: 'Freenas.Workgroup'
	com.delphix:has_per_vdev_zaps
	vdev_children: 2
	vdev_tree:
		type: 'root'
		id: 0
		guid: 4057754529204707565
		children[0]:
			type: 'raidz'
			id: 0
			guid: 6541928767790556413
			nparity: 1
			metaslab_array: 49
			metaslab_shift: 36
			ashift: 12
			asize: 11993762234368
			is_log: 0
			create_txg: 4
			com.delphix:vdev_zap_top: 36
			children[0]:
				type: 'disk'
				id: 0
				guid: 1092407030297009407
				path: '/dev/gptid/7a2fc1c9-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 439
				create_txg: 4
				com.delphix:vdev_zap_leaf: 37
			children[1]:
				type: 'disk'
				id: 1
				guid: 11554200803400165353
				path: '/dev/gptid/7b297d44-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 438
				create_txg: 4
				com.delphix:vdev_zap_leaf: 38
			children[2]:
				type: 'disk'
				id: 2
				guid: 16833066816084926306
				path: '/dev/gptid/7c382ed2-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 437
				create_txg: 4
				com.delphix:vdev_zap_leaf: 39
			children[3]:
				type: 'disk'
				id: 3
				guid: 11437423961006186802
				path: '/dev/gptid/7eade0a6-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 436
				create_txg: 4
				com.delphix:vdev_zap_leaf: 40
		children[1]:
			type: 'raidz'
			id: 1
			guid: 335511987914886298
			nparity: 1
			metaslab_array: 46
			metaslab_shift: 36
			ashift: 12
			asize: 11993762234368
			is_log: 0
			create_txg: 4
			com.delphix:vdev_zap_top: 41
			children[0]:
				type: 'disk'
				id: 0
				guid: 10804527141880872542
				path: '/dev/gptid/85693c1a-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 435
				create_txg: 4
				com.delphix:vdev_zap_leaf: 42
			children[1]:
				type: 'disk'
				id: 1
				guid: 1041627678558872719
				path: '/dev/gptid/888ff6c8-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 434
				create_txg: 4
				com.delphix:vdev_zap_leaf: 43
			children[2]:
				type: 'disk'
				id: 2
				guid: 421127857251552155
				path: '/dev/gptid/8bb7c249-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 433
				create_txg: 4
				com.delphix:vdev_zap_leaf: 44
			children[3]:
				type: 'disk'
				id: 3
				guid: 14799165109861233241
				path: '/dev/gptid/8ec07bd1-08ef-11e8-ab60-0025901159d4.eli'
				whole_disk: 1
				DTL: 431
				create_txg: 4
				com.delphix:vdev_zap_leaf: 45
	features_for_read:
		com.delphix:hole_birth
		com.delphix:embedded_data

zdb -b ZFS_8x_3TB_RAIDz2_pool gives:

Code:

zdb: can't open 'ZFS_8x_3TB_RAIDz2_pool': No such file or directory

(same error for VOLU10TB).

Like others, I'm trying to determine the amount of RAM I need for deduplication: I'm backuping up my windows image backups and well, the current situation with zip compression (gzip-9) doesn't seem to recognize there's a lot of duplicate data... :)

What do to here?

HoneyBadger · Oct 1, 2018

devnullius said:
zdb -U /data/zfs/zpool.cache -S ZFS_8x_3TB_RAIDz2_pool - just hangs

Dedup simulation can take a long time especially with a lot of data in the pool.

If you know what your workload is and are confident that your recordsize will stay consistent, you can make an estimate by multiplying the total estimated number of records by 320 bytes.

devnullius · Oct 1, 2018

Thanks - I did not know what I was doing so I missed the part I should be patient :)

By now (the volume was completely full) I removed all the incremental windows image backups from my local Windows server.

There's still data in the pool, and I got these results for 'zdb -u -s'):

Code:

@Freenas:~ # zdb -U /data/zfs/zpool.cache -S ZFS_8x_3TB_RAIDz2_pool
Simulated DDT histogram:
bucket			  allocated					   referenced
______   ______________________________   ______________________________
refcnt   blocks   LSIZE   PSIZE   DSIZE   blocks   LSIZE   PSIZE   DSIZE
------   ------   -----   -----   -----   ------   -----   -----   -----
	 1	44.7M   5.56T   4.80T   4.80T	44.7M   5.56T   4.80T   4.80T
	 2	14.0M   1.74T   1.25T   1.24T	29.1M   3.61T   2.58T   2.57T
	 4	 419K   50.9G   29.6G   29.7G	1.84M	229G	131G	132G
	 8	 183K   22.5G   11.9G   11.9G	2.16M	273G	150G	151G
	16	 163K   20.3G   8.42G   8.49G	3.01M	383G	159G	160G
	32	52.9K   6.58G   3.02G   3.04G	2.18M	277G	126G	127G
	64	40.8K   5.06G   1.26G   1.32G	3.51M	446G	108G	113G
   128	 102K   12.8G   5.69G   5.72G	18.9M   2.36T   1.05T   1.06T
   256	1.22K	154M   64.6M   65.2M	 438K   54.2G   22.8G   23.0G
   512	   99   12.4M   2.90M   3.01M	62.2K   7.77G   1.77G   1.84G
	1K	   25   3.00M	508K	552K	36.1K   4.38G	688M	750M
	2K		7	772K	184K	192K	21.2K   2.19G	656M	677M
	4K		6	768K	304K	308K	41.7K   5.21G   1.89G   1.92G
	8K		1	128K	128K	128K	10.5K   1.31G   1.31G   1.31G
   16K		2	256K	132K	134K	42.3K   5.29G   2.29G   2.33G
   32K		1	128K	  4K   5.81K	49.1K   6.14G	197M	286M
  256K		1	128K	  4K   5.81K	 291K   36.3G   1.14G   1.65G
 Total	59.6M   7.41T   6.11T   6.10T	 106M   13.2T   9.13T   9.13T
dedup = 1.50, compress = 1.45, copies = 1.00, dedup * compress / copies = 2.17

When I take 59.6M * 320 bytes, that would be a little less than 20GB. I have 40GB, which means I can roughly double my amount of data. So I'll start making incremental backups again and run this command a few more times while the days progress. See if my initial 40GB will be enough, it might be - barely :)
From Google, 106M --> 106.000.000 * 320 bytes = ? GB; gives <34GB max.

Next step: figuring out if enabling dedupe is as simple as ticking a box :)

kdragon75 · Oct 1, 2018

HoneyBadger said:
Dedup simulation can take a long time especially with a lot of data in the pool.

If you know what your workload is and are confident that your recordsize will stay consistent, you can make an estimate by multiplying the total estimated number of records by 320 bytes.

Can you reference a source for this information? I would like to look further into this myself.

HoneyBadger · Oct 1, 2018

kdragon75 said:
Can you reference a source for this information? I would like to look further into this myself.

http://open-zfs.org/wiki/Performance_tuning#Deduplication

https://www.oracle.com/technetwork/...age-admin/o11-113-size-zfs-dedup-1354231.html

Total data size, divided by recordsize, times 320 bytes, equals your estimated DDT size.

320 bytes per record is a rough estimate, but because recordsize is a maximum ( ashift determines your minimum) ZFS could write smaller blocks than you expect, so it's crucial that you have confidence. A backup job spooling to an NFS share, for example, is pretty confidently going to be full-sized records every time. ZVOLs backing VM disks? You could have records as small as your ashift will allow (usually 4K) which could mean you have significantly more blocks than you counted on; and therefore a significantly bigger DDT than you budgeted RAM for.

HoneyBadger · Oct 1, 2018

devnullius said:
Thanks - I did not know what I was doing so I missed the part I should be patient :)

By now (the volume was completely full) I removed all the incremental windows image backups from my local Windows server.

There's still data in the pool, and I got these results for 'zdb -u -s'):

Code:
Simulated DDT histogram: bucket allocated referenced ______ ______________________________ ______________________________ refcnt blocks LSIZE PSIZE DSIZE blocks LSIZE PSIZE DSIZE ------ ------ ----- ----- ----- ------ ----- ----- ----- 1 44.7M 5.56T 4.80T 4.80T 44.7M 5.56T 4.80T 4.80T Total 59.6M 7.41T 6.11T 6.10T 106M 13.2T 9.13T 9.13T dedup = 1.50, compress = 1.45, copies = 1.00, dedup * compress / copies = 2.17

When I take 59.6M * 320 bytes, that would be a little less than 20GB. I have 40GB, which means I can roughly double my amount of data.

Not if you want that system to be usable at all.

If you have 40GB in the entire system, your arc_max is probably around 36GB. Metadata - such as the deduplication table - is limited by default to 1/4 of your arc_max. In your case, that's 9.6GB.

The DDT needs to live entirely in RAM for the best performance; if you're okay with "acceptable" performance, you could have it spill over onto (fast) L2ARC. If it exceeds arc_meta_max and falls onto disk, this is the point where your pool grinds to a screeching halt.

Can you tune this value (arc_meta_max) to be higher? Absolutely; you could have your entire ARC be consumed by metadata if you so chose. But you do this at the cost of your ARC's ability to hold actual data, and retrieve it quickly without having to go to spindle. This will significantly hamper your performance.

So I'll start making incremental backups again and run this command a few more times while the days progress. See if my initial 40GB will be enough, it might be - barely :)
From Google, 106M --> 106.000.000 * 320 bytes = ? GB; gives <34GB max.

Next step: figuring out if enabling dedupe is as simple as ticking a box :)

It is that simple; and that's what makes it so deadly.

Going back to your simulated histogram, look at the line with refcnt 1 - that means "there is only one reference to this data, it won't deduplicate."

44.7M of your 59.6M blocks are unique, comprising 4.80T of your 6.10T of data; that won't be reduced. But you'll be paying a cost of about 14GB of RAM just to index it.

Important Announcement for the TrueNAS Community.

zdb command not giving expected results

titan_rw

Guru

Casey Duplantis

Cadet

Gashy

Cadet

Robert Trevellyan

Pony Wrangler

maurertodd

Dabbler

devnullius

Patron

HoneyBadger

actually does care

devnullius

Patron

kdragon75

Wizard

HoneyBadger

actually does care

HoneyBadger

actually does care

Similar threads