Interpreting zpool iostat

[Chris Moore]

I have a server at work that I am making a copy of the entire pool onto another set of disks and I am trying to figure out how long it will take to copy. I think I understand but I wanted someone to check my work. Here is the iostat output:

Code:

			  capacity	 operations	bandwidth
pool		alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
SmallPond	878G   231T	 24  1.52K   103K  78.2M
BigPond	  161T   165T  1.39K	122  70.9M   512K
----------  -----  -----  -----  -----  -----  -----

The way I figure it, it will take between 12 and 14 days for the copy to finish. Any comments?

 

[Chris Moore]

Just an update. The numbers changed a bit and I am still hopeful that someone will take a moment to look at this.

Code:

			  capacity	 operations	bandwidth
pool		alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
SmallPond   1.55T   230T	 16  1.76K  68.4M  92.3M
BigPond	  161T   165T  1.66K	 84  86.4M   350K
----------  -----  -----  -----  -----  -----  -----

 

[MatthewSteinhoff]

Using this file transfer calculator, 80MB/s and 160TB to move, I'm getting more than twenty days. If I increase the transfer rate to 120MB/s - closer to what I'd expect to see on a single VDEV, big RAIDZ2 pool - I get 17 days. If you have multiple VDEVs making up your pool, your throughput would be higher. So, without doing much math myself, yeah, 12-14 is somewhere between reasonable and optimistic.

If either of the pools scrub during the copy, I imagine transfer performance will crawl so that may add a non-trivial amount of time to the transfer.

Cheers,
Matt

 

[Chris Moore]

The transfer has been running for about three days now and has copied 44 TB of data from one pool to the other. That looks like about 590 GB an hour on average since I started this on Friday (7 Sep 2018) morning.
Just 117 TB to go and it is looking like it might be done in about 8 more days if the transfer speed stays where it is, but it may slow as the target pool gets full.

 

[HoneyBadger]

it may slow as the target pool gets full.

If you're not doing much with SmallPond other than populating it, you should actually get a fairly steady ingest rate as your writes will be mostly sequential and you won't fragment free space as badly.

I do wonder how you're not even breaking 100MB/s on a local copy though. I'd think that based on the size of your pools (over 200TB each) there's enough spindles in play that you should have much higher speeds unless there's major contention for reads on BigPond or you've got sync=always set and are bottlenecking at the SLOG/ZIL.

 

[Chris Moore]

I do wonder how you're not even breaking 100MB/s on a local copy though. I'd think that based on the size of your pools (over 200TB each) there's enough spindles in play that you should have much higher speeds unless there's major contention for reads on BigPond or you've got sync=always set and are bottlenecking at the SLOG/ZIL.

I was not working here when the server was ordered. I had no hand in the hardware selection. It is a "45Drives", "Storinator" and they used two Rocket 750 cards to run the 60 drives in that system. I blame the general slow performance of the system on the drive controllers. The system I am transferring the data to has the drives on a LSI 9207-8i with four 16 drive expander backplanes, two from each port. I don't know which system is slower, but they are both 'old' and the whole reason for the backup is that I will be migrating to a 'new' system and I certainly hope the new one is faster. We will see.

 

[Chris Moore]

Here is the update:

Code:

# zpool iostat
			  capacity	 operations	bandwidth
pool		alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
SmallPond   72.7T   159T	  0  2.22K  3.48K   118M
BigPond	  161T   165T  2.18K	  3   118M  22.6K
----------  -----  -----  -----  -----  -----  -----

Code:

# zpool list
NAME	   SIZE  ALLOC   FREE  EXPANDSZ   FRAG	CAP  DEDUP  HEALTH  ALTROOT
SmallPond  232T  72.7T   159T		 -	15%	31%  1.00x  ONLINE  -
BigPond	326T   161T   165T		 -	27%	49%  1.00x  ONLINE  -

I was thinking that the reason the speed appears slow could be the compression factor. These pools are both using GZIP-9 and the CPU utilization is high.

 

[HoneyBadger]

Yeah, gzip9 would certainly clobber performance especially under CPU constraint. I hope the compressratio is really, really good.

 

[Chris Moore]

I have 280 TB of data in 161 TB of disk space. Is that good? I have the new server on the bench and I will use lz4 with it. It has 10 TB drives.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[HoneyBadger]

I have 280 TB of data in 161 TB of disk space. Is that good?

Depends on the data; have you tried loading any of it into an lz4 dataset to see how that handles it?

I'm getting around 1.6x on my "general Windows server" LUNs using lz4 - I won't compare VDI ones, that's just cheating.

 

[Chris Moore]

have you tried loading any of it into an lz4 dataset to see how that handles it?

Not yet, but that is on the way. I want to do some burn-in testing on the new server before I start loading data into it. I just got the drives unboxed and dropped in the slots on Tuesday.

 

[MatthewSteinhoff]

When bulk loading data, we set compression to gzip9. Then, once loaded, we change the compression to lz4. This gives us maximum compression for static, usually archive, data while giving us the performance boost of lz4 for live data.

In some cases, the difference between gzip9 and lz4 is negligible except for CPU brutality. In others, the initial load starts us at 3x compressed and we know the sending machine was only seeing 1.7x using lz4. It's a mixed bag but not much downside if you have the time.

Cheers,
Matt

 

[Chris Moore]

I had a little surprise today. I thought that the copy would be done and when I checked, this is what I found:

Code:

# zpool iostat
			   capacity	 operations	bandwidth
pool		alloc   free   read  write   read  write
----------  -----  -----  -----  -----  -----  -----
SmallPond	168T  64.1T	  2  2.24K  11.6K   115M
BigPond	  161T   165T  2.20K	 14   115M   124K
----------  -----  -----  -----  -----  -----  -----

Somehow, SmallPond has more allocated storage than BigPond. So I took a look at this:

Code:

# zfs get used,compressratio,compression,logicalused SmallPond
NAME	   PROPERTY	   VALUE	 SOURCE
SmallPond  used		   119T	  -
SmallPond  compressratio  2.41x	 -
SmallPond  compression	gzip-9	local
SmallPond  logicalused	275T	  -

# zfs get used,compressratio,compression,logicalused BigPond
NAME	 PROPERTY	   VALUE	 SOURCE
BigPond  used		   132T	  -
BigPond  compressratio  2.43x	 -
BigPond  compression	gzip	  local
BigPond  logicalused	299T	  -

I am not sure I understand why BigPond compression is just 'gzip' and SmallPond is 'gzip-9' and the compression ratio is better on BigPond so it is taking up less physical space on disk. It is curious.