Really slow random read speeds

[vicmarto]

Hello.

Please, why I'm getting only around 18 MB/s when copying folders with a lot of small files (28,000+) from the zpool to the local client SSD drive (using NFS or SSH, makes no difference)?

This is using a 1Gbps connection. The pool is configured as mirror (trying vainly to maximize the read speed!):

Code:

  NAME		  STATE	 READ WRITE CKSUM
	zpool		 ONLINE	   0	 0	 0
	  mirror-0	ONLINE	   0	 0	 0
		ada0	  ONLINE	   0	 0	 0
		ada1	  ONLINE	   0	 0	 0
	  mirror-1	ONLINE	   0	 0	 0
		ada2	  ONLINE	   0	 0	 0
		ada3	  ONLINE	   0	 0	 0
	  mirror-2	ONLINE	   0	 0	 0
		ada4	  ONLINE	   0	 0	 0
		ada5	  ONLINE	   0	 0	 0

Some relevant properties:

Code:

recordsize			128K
compression		   lz4
atime				 off
dedup				 off

But, instead:

1. Write speed is OK: When copying this same folder from the local client SSD drive to the zpool, I'm saturating the 1Gbps!
2. Sequential read speed is OK: When copying big files (30GB+) from the zpool to the local client SSD drive, I'm saturating the 1Gbps!

Please help, where can be the problem?

 

[nightshade00013]

The answer is simply you are writing TONS of small files. There is the transfer and then a point where some housekeeping is done and then transfer starts again. A single large file is mostly transfer and a little housekeeping. Small files are a tiny amount of transfer and then some system housekeeping for each file.

Think of it this way, you are moving out of where you live. Now you have to empty the kitchen cabinets, is it faster to put the stuff in a box and then take the box to the car or carry each individual item. And no cheating by taking multiple items at once. It's a single item each time or a bunch in a box. The COW system takes time to do everything and it is not just a read or write operation but a read/write operation which is even slower.

 

[garm]

This is why we have tar ;)

 

[vicmarto]

Thanks all for your answers. That is exactly what I though, but maybe expected a (now understod unrealistic) better performance.

Today I want to share some (real data) test I have been doing, because the results I'm getting are not the expected. I can't explain why, maybe you can and we all learn something new.

The system is an Asrock E3C224D2I, an i3-4130T (2.90GHz), 16 GB ECC RAM (the max supported) and 6 (consumer grade) 4TB HDDs.

The test try to determine what zfs raid level is the most fastest.

Was done destroying and rebuilding the pool each time, and consist on:
1) Duplicate (locally by ssh) one big file with a weigh of 21,46GB, compute the time needed, and calculate the MB/s.
2) Duplicate (locally by ssh) one big folder full of little files (28211) with a total weigh of 22,45GB, compute the time needed, and calculate the MB/s.


The expected result was: STIPED > 3x MIRROR > 2x RAIDZ > RAIDZ2

BUT, the reality was:

Code:

STIPED
# zpool create -O compression=lz4 -O atime=off zpool /dev/ada0 /dev/ada1 mirror /dev/ada2 /dev/ada3 mirror /dev/ada4 /dev/ada5
(Transfer a 1 file of 21,46GB by SSH)
# time cp 21,46GB 21,46GB_copy
0.038u 11.118s 2:16.11 8.1%	25+170k 163631+163625io 0pf+0w --------> 161,45 MB/s
(Transfer a 1 directory with 28211 files for a total of 22,45GB by SSH)
time cp -r 28211/ 28211_copy
0.224u 32.921s 12:30.21 4.4%	25+165k 171589+379145io 5386167pf+0w -----------------> 30,64 MB/s

Code:

3x MIRROR
# zpool create -O compression=lz4 -O atime=off zpool mirror /dev/ada0 /dev/ada1 mirror /dev/ada2 /dev/ada3 mirror /dev/ada4 /dev/ada5
(Transfer a 1 file of 21,46GB by SSH)
# time cp 21,46GB 21,46GB_copy
0.031u 10.819s 2:48.05 6.4%	25+168k 163631+163625io 0pf+0w --------> 130,76 MB/s
(Transfer a 1 directory with 28211 files for a total of 22,45GB by SSH)
time cp -r 28211/ 28211_copy
0.167u 31.566s 13:03.37 4.0%	25+165k 171152+379145io 5386167pf+0w ----------------->  29,35 MB/s

Code:

2x RAIDZ
# zpool create -O compression=lz4 -O atime=off zpool raidz /dev/ada0 /dev/ada1 /dev/ada2 raidz /dev/ada3 /dev/ada4 /dev/ada5
(Transfer a 1 file of 21,46GB by SSH)
# time cp 21,46GB 21,46GB_copy
0.039u 11.041s 2:36.40 7.0%	25+167k 163630+163625io 0pf+0w --------> 140,87 MB/s
(Transfer a 1 directory with 28211 files for a total of 22,45GB by SSH)
# time cp -r 28211/ 28211_copy
0.267u 30.315s 13:56.94 3.6%	25+167k 171467+379145io 5386167pf+0w -----------------> 27,47 MB/s

Code:

RAIDZ2
# zpool create -O compression=lz4 -O atime=off zpool raidz2 /dev/ada0 /dev/ada1 /dev/ada2 /dev/ada3 /dev/ada4 /dev/ada5
(Transfer a 1 file of 21,46GB by SSH)
# time cp 21,46GB 21,46GB_copy
0.053u 10.719s 2:31.99 7.0%	26+172k 163630+163625io 0pf+0w --------> 144,58 MB/s
(Transfer a 1 directory with 28211 files for a total of 22,45GB by SSH)
# time cp -r 28211/ 28211_copy
0.267u 30.143s 11:24.09 4.4%	25+167k 172788+379145io 5386167pf+0w -----------------> 33,60 MB/s

What's you opinion? Why RAIDZ2 is more fast that 3x MIRROR?



P/D: (Every copy was done two times to confirm the times)