Questions on vdev construction after pool re-creation, and performance

Elliot Dierksen · Nov 1, 2018

@Ender117 and @HoneyBadger - Moving discussion to a new thread.

I was maxing out at 4G NFS writes with my pool constructed as follows:

Code:

root@freenas2:/nonexistent # zpool list -v RAIDZ2-I
NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
RAIDZ2-I								14.5T  5.75T  8.75T		 -	32%	39%  1.00x  ONLINE  /mnt
  raidz2								7.25T  5.52T  1.73T		 -	60%	76%
	gptid/bd041ac6-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/bdef2899-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/bed51d90-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/bfb76075-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/c09c704a-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/c1922b7c-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/c276eb75-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/c3724eeb-9e63-11e7-a091-e4c722848f30	  -	  -	  -		 -	  -	  -
  raidz2								7.25T   240G  7.02T		 -	 5%	 3%
	gptid/a1b7ef4b-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/a2eb419f-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/a41758d7-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/a5444dfb-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/a6dcd16f-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/a80cd73c-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/a94711a5-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/aaa6631d-3c2a-11e8-978a-e4c722848f30	  -	  -	  -		 -	  -	  -
log										 -	  -	  -		 -	  -	  -
  nvd0p1								15.9G  5.65M  15.9G		 -	 0%	 0%
cache									   -	  -	  -		 -	  -	  -
  nvd0p4								 213G  17.3G   196G		 -	 0%	 8%
spare									   -	  -	  -		 -	  -	  -
  gptid/4abff125-23a2-11e8-a466-e4c722848f30	  -	  -	  -		 -	  -	  -

I noticed the fragmentation, and figured that was part of that part of the issue. This caused me to move the data off and re-create the pool. Now it looks like this:

Code:

root@freenas2:/nonexistent # zpool list -v RAIDZ2-I
NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
RAIDZ2-I								14.5T   158G  14.3T		 -	 0%	 1%  1.00x  ONLINE  /mnt
  raidz2								3.62T  39.6G  3.59T		 -	 0%	 1%
	gptid/70852685-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/71686954-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/724e4021-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/73554422-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
  raidz2								3.62T  39.6G  3.59T		 -	 0%	 1%
	gptid/746dafd4-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/75626368-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/764f85ad-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/779e221d-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
  raidz2								3.62T  39.6G  3.59T		 -	 0%	 1%
	gptid/ce00e493-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/ceee50a2-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/cfe1d0e6-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/d0d66ee5-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
  raidz2								3.62T  39.6G  3.59T		 -	 0%	 1%
	gptid/d450f2fb-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/d55539a0-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/d64a2170-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/d7709adf-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
log										 -	  -	  -		 -	  -	  -
  nvd0p1								15.9G	  0  15.9G		 -	 0%	 0%
cache									   -	  -	  -		 -	  -	  -
  nvd0p4								 213G  17.5G   195G		 -	 0%	 8%
spare									   -	  -	  -		 -	  -	  -
  gptid/f3fa63e8-dddc-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -

What puzzles me is a that I added 8 disks at a time as RAIDZ2. I was expecting each of those additions to create a single vdev, but I ended up with 4x2 (it did state that when I was creating them in the GUI). What gives? I know more vdevs = more performance, but I am confused by the construction of the vdevs. I did get about the amount of available storage space that I was expecting (12 disks out of 16). Strange things are afoot at the Circle-K....

Chris Moore · Nov 1, 2018

Elliot Dierksen said:
What puzzles me is a that I added 8 disks at a time as RAIDZ2. I was expecting each of those additions to create a single vdev, but I ended up with 4x2 (it did state that when I was creating them in the GUI). What gives?

The GUI can be a little tricky. I can't get you any screen grabs while I am at work, but I will try to get some illustrations for you tonight. Don't put any data in there yet, because you probably want to blow that away and build another pool with the layout you want.

HoneyBadger · Nov 1, 2018

You're definitely not getting the same actual, usable space. The SIZE value in zpool list shows raw space (eg: doesn't account for the Z2 parity)

Do zfs list RAIDZ2-I and you'll see the AVAIL space be much smaller (expected to be 8/16 drives)

When you create the pool, you select the vdev type from the drop-down, and the "width" and "height" of the blocks on the right determine the vdev configuration. Just drag the little ball on the lower-right to change it.

Smaller vdevs in Z2 (4x2)

Bigger vdevs in Z2 (8x1)

Elliot Dierksen · Nov 1, 2018

Thanks to @Chris Moore and @HoneyBadger for the instructions. I haven't actually had to recreate a pool in a while, so I did find the GUI kind of confusing. Now it is looking more like I expected.

Code:

root@freenas2:/ # zpool list -v RAIDZ2-I
NAME									 SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
RAIDZ2-I								14.5T  4.54G  14.5T		 -	 0%	 0%  1.00x  ONLINE  /mnt
  raidz2								7.25T  2.27G  7.25T		 -	 0%	 0%
	gptid/860ff564-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/87452195-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/88608e29-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/8958ea60-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/8a922989-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/8b9b3fad-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/8c864e27-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/8da9d97b-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
  raidz2								7.25T  2.27G  7.25T		 -	 0%	 0%
	gptid/b8cb5dae-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/b9c49a18-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/bab86ebe-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/bba9b791-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/bcc23de7-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/be03aff5-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/befc8529-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
	gptid/c04a8d78-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -
log										 -	  -	  -		 -	  -	  -
  nvd0p1								15.9G	  0  15.9G		 -	 0%	 0%
cache									   -	  -	  -		 -	  -	  -
  nvd0p4								 213G	  0   213G		 -	 0%	 0%
spare									   -	  -	  -		 -	  -	  -
  gptid/e499dac6-de02-11e8-adca-e4c722848f30	  -	  -	  -		 -	  -	  -

Chris Moore · Nov 1, 2018

@HoneyBadger Great illustrations, I had something like that in mind.

HoneyBadger · Nov 1, 2018

Now we're cooking. With your shiny, 0% FRAG pool, you can disable compression (it's easier that way) and blast it with some benchmarks.

Got/want those dtrace scripts?

Elliot Dierksen · Nov 1, 2018

HoneyBadger said:
Now we're cooking. With your shiny, 0% FRAG pool, you can disable compression (it's easier that way) and blast it with some benchmarks.

Got/want those dtrace scripts?

Sure. I don't have the scripts, but I am certainly curious about running them. I was going to load my data up with max compression (since most of it is archived), and then set the compression off for testing.

Chris Moore · Nov 1, 2018

[QUOTE="Elliot Dierksen, post: 490682, member: 45941"I was going to load my data up with max compression (since most of it is archived)[/QUOTE]I did that on one of the new servers at work. It is a good idea, I am not saying don't do it, but be prepared for your CPU to be slammed for as long as the transfer takes and it will slow the transfer rate. Mind you, I was transferring 290TB, but the volume of data shouldn't make as much difference. Dual 8 core Xeon system (32 threads) and they were all pegged for the duration and I filled my swap space.

HoneyBadger · Nov 1, 2018

Two most useful ones are determining the amount of dirty data, and the duration of a txg commit.

Source: http://dtrace.org/blogs/ahl/2014/08/31/openzfs-tuning/ if you want to go down the rabbit hole, but some of the scripts are Solaris-specific.

Run these two in extra windows while you're beating up the array, and keep an eye on how much dirty data is there and how long it's taking to commit it out.

dirty.d

Code:

txg-syncing
{
		this->dp = (dsl_pool_t *)arg0;
}

txg-syncing
/this->dp->dp_spa->spa_name == $$1/
{
		printf("%4dMB of %4dMB used", this->dp->dp_dirty_total / 1024 / 1024,
			`zfs_dirty_data_max / 1024 / 1024);
}

duration.d

Code:

txg-syncing
/((dsl_pool_t *)arg0)->dp_spa->spa_name == $$1/
{
		start = timestamp;
}

txg-synced
/start && ((dsl_pool_t *)arg0)->dp_spa->spa_name == $$1/
{
		this->d = timestamp - start;
		printf("sync took %d.%02d seconds", this->d / 1000000000,
			this->d / 10000000 % 100);
}

Elliot Dierksen · Nov 1, 2018

Chris Moore said:
I did that on one of the new servers at work. It is a good idea, I am not saying don't do it, but be prepared for your CPU to be slammed for as long as the transfer takes and it will slow the transfer rate.

Yes, I am expecting that. I was also interested to see how much the CPU temperatures went up while I was doing that.

The drop around 14:20 is while I wasn't doing anything. A 20-30 degree difference. Wow!

Elliot Dierksen · Nov 1, 2018

@HoneyBadger - Here are some dtrace results.

Code:

dtrace: script 'dirty.d' matched 2 probes
CPU	 ID					FUNCTION:NAME
  9  63083				 none:txg-syncing 3824MB of 4096MB used
  9  63083				 none:txg-syncing 3825MB of 4096MB used
  9  63083				 none:txg-syncing 3822MB of 4096MB used
 14  63083				 none:txg-syncing 3872MB of 4096MB used
 14  63083				 none:txg-syncing 3873MB of 4096MB used
 10  63083				 none:txg-syncing 3897MB of 4096MB used

So that looks like I am only allowing 4G of dirty data. Am I interpreting that correctly?

Here is the duration result:

Code:

dtrace: script 'duration.d' matched 2 probes
CPU	 ID					FUNCTION:NAME
 14  63084				  none:txg-synced sync took 22.82 seconds
 14  63084				  none:txg-synced sync took 27.80 seconds
 10  63084				  none:txg-synced sync took 26.63 seconds
 10  63084				  none:txg-synced sync took 0.00 seconds
  3  63084				  none:txg-synced sync took 26.00 seconds
  9  63084				  none:txg-synced sync took 23.33 seconds
  9  63084				  none:txg-synced sync took 0.00 seconds
 13  63084				  none:txg-synced sync took 18.40 seconds

I am not sure what the duration info tells me.

HoneyBadger · Nov 1, 2018

Elliot Dierksen said:
So that looks like I am only allowing 4G of dirty data. Am I interpreting that correctly?

Yep, that's the default max for OpenZFS - 10% of RAM or 4GB, whichever is lower.

I assume this is a local test, which is why there's no "network bottleneck" - you're just flooding it with dd or iozone or similar?

Elliot Dierksen said:
I am not sure what the duration info tells me.

You're maxing out the array, that's for sure - 20+ seconds to flush the transactions.

What do you normally run off this system - NFS exports hosting VMs?

Elliot Dierksen · Nov 1, 2018

HoneyBadger said:
Yep, that's the default max for OpenZFS - 10% of RAM or 4GB, whichever is lower.

I assume this is a local test, which is why there's no "network bottleneck" - you're just flooding it with dd or iozone or similar?

I have my secondary FreeNAS mounted via NFS and I am copying the files back to the primary using cpio. I am only getting about 2G throughput now, but some of that could be because I have max compression on. I know the other NAS is capable of pushing 6-8G throughput out via NFS.

HoneyBadger said:
You're maxing out the array, that's for sure - 20+ seconds to flush the transactions.

What do you normally run off this system - NFS exports hosting VMs?

Yes, normally just 3-4 NFS exported ESXi VM's. I do archive some stuff to it via CIFS, but the 10G interfaces are on a storage VLAN where only the FreeNAS boxes and the 10G interfaces of the ESXi hosts live.

HoneyBadger · Nov 1, 2018

Max compression (gzip-9) is definitely your bottleneck here. You've got roughly 3.8GB in the backlog waiting to be queued out, and I imagine ZFS is write-throttling the living daylights out of the open transaction groups trying to keep up with the molasses-slow compression and writing.

Since you have a "clean slate" I'd suggest comparing the results in terms of compressratio with lz4 vs gzip9 and see if it's really worth the hurt, since you're going to pay a price to decompress this data as well. Obviously nowhere near as brutal, but still.

Elliot Dierksen said:
Yes, normally just 3-4 NFS exported ESXi VM's. I do archive some stuff to it via CIFS, but the 10G interfaces are on a storage VLAN where only the FreeNAS boxes and the 10G interfaces of the ESXi hosts live.

Are the VMs the combination of "small enough" and "important enough" to consider dedicating a few spindles to them in a separate pool? If not, make sure you're using a smaller-than-default-128K recordsize on that dataset. RAIDZ2 will hurt your IOPS enough, you don't need to add read-modify-write into the mix.

Elliot Dierksen · Nov 1, 2018

HoneyBadger said:
Since you have a "clean slate" I'd suggest comparing the results in terms of compressratio with lz4 vs gzip9

I may do a compare on lz4 versus gzip9. A bunch of the data there is just for archival purposes, so I don't think that will hurt me.

HoneyBadger said:
Are the VMs the combination of "small enough" and "important enough" to consider dedicating a few spindles to them in a separate pool?

The performance needs of those VM's is not high enough to really justify that, or at least not to me right now.

HoneyBadger said:
If not, make sure you're using a smaller-than-default-128K recordsize on that dataset

What would you suggest for the VM's? Those are in a separate dataset, so I could easily change that there.

HoneyBadger · Nov 1, 2018

Elliot Dierksen said:
I may do a compare on lz4 versus gzip9. A bunch of the data there is just for archival purposes, so I don't think that will hurt me.

Spoiler: LZ4 will be way faster than gzip9 with much less CPU usage. ;)

The performance needs of those VM's is not high enough to really justify that, or at least not to me right now.

What would you suggest for the VM's? Those are in a separate dataset, so I could easily change that there.

I'd suggest recordsize=16K as a nice balance between small-block latency and sequential data - you'll want to set this before you copy the VMs back though, or the svMotion will write a bunch of 128K blocks. That's the default for ZVOLs under FreeNAS, and it's pretty good. Basically the smaller recordsize gives you better latency numbers (really good ones if you match your filesystem's block or database record size!) but results in slower speeds for big sequential access (like the inital migration or copy)

Elliot Dierksen · Nov 1, 2018

HoneyBadger said:
I imagine ZFS is write-throttling the living daylights out of the open transaction groups trying to keep up with the molasses-slow compression and writing.

Indeed, check out the differences in the dtrace results.

Code:

dtrace: script 'dirty.d' matched 2 probes
CPU	 ID					FUNCTION:NAME
 14  63083				 none:txg-syncing   64MB of 4096MB used
  8  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
  1  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
  5  63083				 none:txg-syncing   64MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
  8  63083				 none:txg-syncing   64MB of 4096MB used
  6  63083				 none:txg-syncing   64MB of 4096MB used
  6  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
  1  63083				 none:txg-syncing   64MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
  7  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
  7  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
  9  63083				 none:txg-syncing   64MB of 4096MB used
  9  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
  1  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   65MB of 4096MB used
 13  63083				 none:txg-syncing   55MB of 4096MB used
  9  63083				 none:txg-syncing   83MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
  5  63083				 none:txg-syncing   67MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   62MB of 4096MB used
 11  63083				 none:txg-syncing   92MB of 4096MB used
  4  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
  4  63083				 none:txg-syncing   64MB of 4096MB used
  6  63083				 none:txg-syncing   76MB of 4096MB used
 13  63083				 none:txg-syncing   77MB of 4096MB used
  9  63083				 none:txg-syncing   69MB of 4096MB used
 10  63083				 none:txg-syncing   77MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
  8  63083				 none:txg-syncing   64MB of 4096MB used
  9  63083				 none:txg-syncing   64MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
  4  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   66MB of 4096MB used
 15  63083				 none:txg-syncing  101MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   65MB of 4096MB used
  0  63083				 none:txg-syncing   76MB of 4096MB used
  7  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   65MB of 4096MB used

Code:

dtrace: script 'duration.d' matched 2 probes
CPU	 ID					FUNCTION:NAME
  7  63084				  none:txg-synced sync took 0.23 seconds
 14  63084				  none:txg-synced sync took 0.26 seconds
 12  63084				  none:txg-synced sync took 0.26 seconds
  8  63084				  none:txg-synced sync took 0.24 seconds
  2  63084				  none:txg-synced sync took 0.24 seconds
  9  63084				  none:txg-synced sync took 0.18 seconds
 14  63084				  none:txg-synced sync took 0.21 seconds
  6  63084				  none:txg-synced sync took 0.27 seconds
 12  63084				  none:txg-synced sync took 0.24 seconds
 10  63084				  none:txg-synced sync took 0.21 seconds
 11  63084				  none:txg-synced sync took 0.11 seconds
 14  63084				  none:txg-synced sync took 0.20 seconds
 14  63084				  none:txg-synced sync took 0.19 seconds
 12  63084				  none:txg-synced sync took 0.16 seconds
  8  63084				  none:txg-synced sync took 0.14 seconds
 11  63084				  none:txg-synced sync took 0.15 seconds
  7  63084				  none:txg-synced sync took 0.18 seconds
 10  63084				  none:txg-synced sync took 0.21 seconds
 12  63084				  none:txg-synced sync took 0.20 seconds
 12  63084				  none:txg-synced sync took 0.19 seconds
 14  63084				  none:txg-synced sync took 0.25 seconds
 15  63084				  none:txg-synced sync took 0.21 seconds
 11  63084				  none:txg-synced sync took 0.19 seconds
  0  63084				  none:txg-synced sync took 0.17 seconds
  4  63084				  none:txg-synced sync took 0.11 seconds
  9  63084				  none:txg-synced sync took 0.23 seconds
 10  63084				  none:txg-synced sync took 0.19 seconds
  9  63084				  none:txg-synced sync took 0.11 seconds
  3  63084				  none:txg-synced sync took 0.27 seconds
  8  63084				  none:txg-synced sync took 0.19 seconds
 14  63084				  none:txg-synced sync took 0.13 seconds
 12  63084				  none:txg-synced sync took 0.21 seconds
  3  63084				  none:txg-synced sync took 0.17 seconds
  0  63084				  none:txg-synced sync took 0.11 seconds
  6  63084				  none:txg-synced sync took 0.25 seconds
 13  63084				  none:txg-synced sync took 0.25 seconds
  9  63084				  none:txg-synced sync took 0.24 seconds
 10  63084				  none:txg-synced sync took 0.22 seconds
  5  63084				  none:txg-synced sync took 0.22 seconds
  1  63084				  none:txg-synced sync took 0.19 seconds
 14  63084				  none:txg-synced sync took 0.10 seconds
 14  63084				  none:txg-synced sync took 0.24 seconds
  6  63084				  none:txg-synced sync took 0.22 seconds
 14  63084				  none:txg-synced sync took 0.22 seconds
 13  63084				  none:txg-synced sync took 0.23 seconds
 11  63084				  none:txg-synced sync took 0.16 seconds
  7  63084				  none:txg-synced sync took 0.18 seconds
  9  63084				  none:txg-synced sync took 0.24 seconds
 15  63084				  none:txg-synced sync took 0.22 seconds
  6  63084				  none:txg-synced sync took 0.21 seconds
 13  63084				  none:txg-synced sync took 0.16 seconds
 10  63084				  none:txg-synced sync took 0.25 seconds
 11  63084				  none:txg-synced sync took 0.11 seconds

I'll have to see what the network throughput is with lz4 as compared to the throttled results with gzip9. Anything else you would suggest since I have a green field at the moment? I did set the VMWare dataset to 16K record size.

Ender117 · Nov 1, 2018

For pure benchmark purpose I would start with compression=off and go from there. But I am interested in the impact of compression as well.

OP, for your system I feel you could set dirty_data_max_max to 8g. Though I believe once defraged your bottleneck (CPU and network aside) would be 900p instead of the HDDs.

Ender117 · Nov 1, 2018

Elliot Dierksen said:

Indeed, check out the differences in the dtrace results.

Code:

dtrace: script 'dirty.d' matched 2 probes
CPU	 ID					FUNCTION:NAME
 14  63083				 none:txg-syncing   64MB of 4096MB used
  8  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
  1  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
  5  63083				 none:txg-syncing   64MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
  8  63083				 none:txg-syncing   64MB of 4096MB used
  6  63083				 none:txg-syncing   64MB of 4096MB used
  6  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
  1  63083				 none:txg-syncing   64MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
  7  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
  7  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
  9  63083				 none:txg-syncing   64MB of 4096MB used
  9  63083				 none:txg-syncing   64MB of 4096MB used
 10  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
  1  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
 15  63083				 none:txg-syncing   65MB of 4096MB used
 13  63083				 none:txg-syncing   55MB of 4096MB used
  9  63083				 none:txg-syncing   83MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
  5  63083				 none:txg-syncing   67MB of 4096MB used
 15  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   62MB of 4096MB used
 11  63083				 none:txg-syncing   92MB of 4096MB used
  4  63083				 none:txg-syncing   64MB of 4096MB used
  2  63083				 none:txg-syncing   64MB of 4096MB used
  4  63083				 none:txg-syncing   64MB of 4096MB used
  6  63083				 none:txg-syncing   76MB of 4096MB used
 13  63083				 none:txg-syncing   77MB of 4096MB used
  9  63083				 none:txg-syncing   69MB of 4096MB used
 10  63083				 none:txg-syncing   77MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
 13  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
  8  63083				 none:txg-syncing   64MB of 4096MB used
  9  63083				 none:txg-syncing   64MB of 4096MB used
  3  63083				 none:txg-syncing   64MB of 4096MB used
  4  63083				 none:txg-syncing   64MB of 4096MB used
 12  63083				 none:txg-syncing   66MB of 4096MB used
 15  63083				 none:txg-syncing  101MB of 4096MB used
 11  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   64MB of 4096MB used
  0  63083				 none:txg-syncing   65MB of 4096MB used
  0  63083				 none:txg-syncing   76MB of 4096MB used
  7  63083				 none:txg-syncing   64MB of 4096MB used
 14  63083				 none:txg-syncing   65MB of 4096MB used

Code:

dtrace: script 'duration.d' matched 2 probes
CPU	 ID					FUNCTION:NAME
  7  63084				  none:txg-synced sync took 0.23 seconds
 14  63084				  none:txg-synced sync took 0.26 seconds
 12  63084				  none:txg-synced sync took 0.26 seconds
  8  63084				  none:txg-synced sync took 0.24 seconds
  2  63084				  none:txg-synced sync took 0.24 seconds
  9  63084				  none:txg-synced sync took 0.18 seconds
 14  63084				  none:txg-synced sync took 0.21 seconds
  6  63084				  none:txg-synced sync took 0.27 seconds
 12  63084				  none:txg-synced sync took 0.24 seconds
 10  63084				  none:txg-synced sync took 0.21 seconds
 11  63084				  none:txg-synced sync took 0.11 seconds
 14  63084				  none:txg-synced sync took 0.20 seconds
 14  63084				  none:txg-synced sync took 0.19 seconds
 12  63084				  none:txg-synced sync took 0.16 seconds
  8  63084				  none:txg-synced sync took 0.14 seconds
 11  63084				  none:txg-synced sync took 0.15 seconds
  7  63084				  none:txg-synced sync took 0.18 seconds
 10  63084				  none:txg-synced sync took 0.21 seconds
 12  63084				  none:txg-synced sync took 0.20 seconds
 12  63084				  none:txg-synced sync took 0.19 seconds
 14  63084				  none:txg-synced sync took 0.25 seconds
 15  63084				  none:txg-synced sync took 0.21 seconds
 11  63084				  none:txg-synced sync took 0.19 seconds
  0  63084				  none:txg-synced sync took 0.17 seconds
  4  63084				  none:txg-synced sync took 0.11 seconds
  9  63084				  none:txg-synced sync took 0.23 seconds
 10  63084				  none:txg-synced sync took 0.19 seconds
  9  63084				  none:txg-synced sync took 0.11 seconds
  3  63084				  none:txg-synced sync took 0.27 seconds
  8  63084				  none:txg-synced sync took 0.19 seconds
 14  63084				  none:txg-synced sync took 0.13 seconds
 12  63084				  none:txg-synced sync took 0.21 seconds
  3  63084				  none:txg-synced sync took 0.17 seconds
  0  63084				  none:txg-synced sync took 0.11 seconds
  6  63084				  none:txg-synced sync took 0.25 seconds
 13  63084				  none:txg-synced sync took 0.25 seconds
  9  63084				  none:txg-synced sync took 0.24 seconds
 10  63084				  none:txg-synced sync took 0.22 seconds
  5  63084				  none:txg-synced sync took 0.22 seconds
  1  63084				  none:txg-synced sync took 0.19 seconds
 14  63084				  none:txg-synced sync took 0.10 seconds
 14  63084				  none:txg-synced sync took 0.24 seconds
  6  63084				  none:txg-synced sync took 0.22 seconds
 14  63084				  none:txg-synced sync took 0.22 seconds
 13  63084				  none:txg-synced sync took 0.23 seconds
 11  63084				  none:txg-synced sync took 0.16 seconds
  7  63084				  none:txg-synced sync took 0.18 seconds
  9  63084				  none:txg-synced sync took 0.24 seconds
 15  63084				  none:txg-synced sync took 0.22 seconds
  6  63084				  none:txg-synced sync took 0.21 seconds
 13  63084				  none:txg-synced sync took 0.16 seconds
 10  63084				  none:txg-synced sync took 0.25 seconds
 11  63084				  none:txg-synced sync took 0.11 seconds

I'll have to see what the network throughput is with lz4 as compared to the throttled results with gzip9. Anything else you would suggest since I have a green field at the moment? I did set the VMWare dataset to 16K record size.

This translates to ~15Gb raw speed of the 2 HDD vdevs combined, not bad at all.

Could you try iozone -a -s 512M -O on a compression=off, recordsize=16K, sync=always, backed by 900p dataset ? This should give you the IOPS number of your pool, and show how well 900p works as an SLOG. On my system the number are like 1/4 of diskinfo -wS result and has been bothering me a lot.

Elliot Dierksen · Nov 1, 2018

HoneyBadger said:
Spoiler: LZ4 will be way faster than gzip9 with much less CPU usage. ;)

Check out the difference in CPU load. It is more than I would have thought. The data I had copied to that point was only getting 1.10 compression. Even if lz4 is less, it doesn't look like it is worth the hit with the data I have in this pool.

@Ender117 - I will run the iozone tests when my copy completes so I don't skew the results.

Important Announcement for the TrueNAS Community.

Questions on vdev construction after pool re-creation, and performance

Guru

Hall of Famer

actually does care

Guru

Hall of Famer

actually does care

Guru

Hall of Famer

actually does care

Guru

Guru

actually does care

Guru

actually does care

Guru

actually does care

Guru

Patron

Patron

Guru

Similar threads