Even if you add higher parity levels, the computations get more complex and eventually you will cross a threshold where rebuilding a disk takes longer than the average time between disk failures in large arrays. I've seen documented systems where this is nearly the case with RAIDZ3 already and moving to "RAIDZ4"parity would surely push it past the limit in even more systems. This is obviously a problem. You would always be resilvering and would suffer from a pool that is always operating at degraded performance which is simply not acceptable and certainly not optimal.
The problem here is more that RAIDZn isn't supposed to be used on pools with heavy workloads, as the vdev performance is similar to one of the component devices. You're getting an archival quality storage vdev.
I too have seen where "rebuilding a disk takes longer than the average time between disk failures" but this is invariably from a misdesigned system that is expecting ongoing high performance out of such a pool.
The first simplest solution is to make more pools rather than single large pools.
Huh??? You could just add more vdevs to a single large pool for the same effect.
Another answer that some have turned to is to use mirrors which scale a lot better. Also the main benefit to mirrors are that rebuilds are extremely fast. It's essentially just a copy operation from the other mirror. Meanwhile, all the other mirrored sets can operate at full capacity and serve up all the IOPS that the customer needs during these mirror resilvers.
http://jrs-s.net/2015/02/06/zfs-you-should-use-mirror-vdevs-not-raidz/
Except that in practice, that's not what happens. If you've got a system that is expecting high performance out of a set of disks, you suddenly develop a massive hot spot on the surviving mirror disk when one of the disks dies. This is a major impediment to rebuild.
While ZFS has no easy way to deal with that, other than to provision your mirrors at N+1 (or more) width, I can tell you that there ARE ways to cope with that problem from a CS point of view. For example, consider a 24 disk system that is designed to have 12 disks and then another 12 disks "mirroring" the data of the first 12. Again, remember we're not talking ZFS here... but rather a concept. On the first set of disks, you use a hash function to determine on which disk to store a given unit of data. On the second set of disks, you use a different hash function to determine on which disk to store that same unit of data. The data is stored on both sets of disks, but if a disk fails, because you've used a different hash to distribute the data on the other set of disks, the missing data from the failed disk is stored evenly across the entire set of disks in the opposing mirror.
This is extremely cool, because each drive in the opposing mirror only sees a 1/12 increase in its workload due to the lost disk - severely reducing the "hot spot" effect. And when you're rebuilding the failed disk, the data is also pulled evenly from all those other drives, meaning your rebuild probably progresses at nearly the write speed of the replaced drive...!
So anyways this isn't theoretical. This is the technique I developed more than a decade ago for Usenet service providers to provide hotspot-resistant redundancy at a server level (the hash controls distribution of articles to storage servers). It works very well. The whole thing ended up functioning as a primitive pseudo-distributed-file-system that took advantage of some various statistical properties of the traffic.
Getting back to the issue at hand: The usual problem with mirrors is that you often get to a point where you're reliant on that performance from each component device, because most often that's just two devices. If you can commit to keeping at least N+1 (and preferably N+2) width, where N is the number of drives (minimum 2) you need in a vdev to sustain your required performance, then this isn't much of an issue. But I don't usually see that happening. Most people get all crazy eyed when they think about all the "wasted disk space."
But that's what mirrors are for. We have RAIDZn for mega space and poor performance. We have mirrors for modest space and high performance.
For even larger systems companies are moving to distributed filesystems.
See this video to explain what companies who have outgrown traditional parity RAID solutions are now using:
https://www.youtube.com/watch?v=x54s9cjMjPU
Now parity itself is not dead, but as RAID it is.
The linked video is essentially just discussing the next evolutionary step for RAID parity, which isn't parity. It's a more clever strategy - great, we do need that. The fact that it's being stored on various servers is just a necessary side effect of scale, because there's no way to attach 100PB of storage to a single server and still be able to make use of its performance potential. So, yes, distributed filesystem for that, but we already knew we needed multiple storage servers for massive scale storage.
The problem is, moving to multiple servers isn't a comprehensive answer. You have to be able to also scale up an individual machine. Drives will continue to get bigger, and we've got to figure out how to cope with this. We need a little of both.
