Very slow directory traversal

[wsmith67]

I have a Mini 3.0 X+ with 64GB of RAM and 5 Ironwolf Pro 18TB drives in a raidz2 pool. The dataset in question has record size 128k, lz4 compression, and AES256 encryption. The test is running by itself; the system has no other load.

There is a directory with 42,000 subdirectories, each of which has just a few file entries. This structure was created all at the same time with a recursive `cp`. Using a minimal file walker written in Go (to avoid any excess overhead of `find | wc` or the like) it takes many minutes to traverse this structure and count the 68,000 total files. It is processing fewer than 30 directories per second. The disks are seeking like crazy. (If I run the file count again (so using cached metadata) it executes instantly, as one would expect.)

Simple question: this seems ridiculously slow. Is this behavior expected? Thanks for any wisdom/advice.

 

[jgreco]

42000 subdirectories? You could be needing to fetch a lot of metadata. A RAIDZ vdev has IOPS that are closely related to the underlying devices, and if you are only running a single thread, and are only able to sustain 300 IOPS-ish, you could easily be needing several minutes to traverse this structure. I have some stuff that automatically rsync's between data centers and it is really slow if you don't set up something like L2ARC caching of metadata.

 

[wsmith67]

42000 subdirectories? You could be needing to fetch a lot of metadata. A RAIDZ vdev has IOPS that are closely related to the underlying devices, and if you are only running a single thread, and are only able to sustain 300 IOPS-ish, you could easily be needing several minutes to traverse this structure. I have some stuff that automatically rsync's between data centers and it is really slow if you don't set up something like L2ARC caching of metadata.

It is an outlier. I guess what's surprising me is that given the directories are so small, and the entire tree was written at once, I expected more metadata locality than I'm (literally) hearing. Also, 300 IOPS is about right, but that implies 10 IOPS per directory (with 5 files in it) which seems like a lot. I expect it to be slow but not THIS slow.

 

[winnielinnie]

(If I run the file count again (so using cached metadata) it executes instantly, as one would expect.

Chances are if you don't use/access this cache'd metadata within an hour (or even shorter), it'll be evicted from the cache. Thus, the issue starts all over again.

See this thread.

And starting from post #9 is the solution and follow-up discussion.

 

[Constantin]

I second setting up a persistent, metadata-only L2ARC to help speed traversals. See my thread on L2ARC performance impacts, especially as it pertains to heavy directory traversals, as is common with rsync backups.

The more effective, but riskier approach to speeding up things is a special VDEV, but I'd shy away from that unless you know what you are doing. While a dead L2ARC will have zero impact on your data integrity, a completely blown SVDEV will take your entire pool down.

If the data in these directories is largely dormant, I'd also consider using a tool like Apple Disk Image to create sparsebundles and pack in the data that way. That allows for much faster rsync, especially of many small files.