Drive size and resilvering times

[feenberg]

Is resilvering time a valid reason to avoid very large drives? I was thinking that 4GB disks would resilver in half the time of 8GB disks, but since there would be twice as many expected failures, is anything really gained? You might think that 4GB drives would have half the failure rate, but vendors don't seem to claim that. Resilvering times are measured in weeks for our largest (14TB) drives, and even with double parity, it is nervous making.

 

[blanchet]

If you have very large hard drives, you should avoid large vdev, otherwise you may lack of IOPS and the resilvering time may become very long.

ZFS: resilver performance of various RAID schemas

What is your vdev layout ?
How busy is your pool ?

 

[c77dk]

and when you write weeks: which drives are you using?

 

[sretalla]

What might be more interesting to know is the width (number of drives) of your VDEVs/pool.

Very wide (probably starting from 10, but anything over 12) VDEVs take inordinately long to resilver more-or less regardless of the drive size (but it of course adds a further multiplication of the problem).

 

[ChrisRJ]

Resilvering times are measured in weeks for our largest (14TB) drives, and even with double parity, it is nervous making.

What do you mean with "are measured in weeks"? I have 16 TB drives and seem to remember that resilvering took about a day (with about 25-30% of the space in use). If you have experienced a duration of several weeks, I would think that something is wrong elsewhere.

 

[jgreco]

Resilvering time is dependent on many factors. During the first percent or two of a resilver, it is VERY common for some outlandishly long resilver time to be projected, because the drive is working through metadata blocks, which require lots of seeking. However, if you have lots of small files or other fragmented data, that number may never drop and it may actually take a week or three, for realz. Also, as noted, extra-wide RAIDZ vdevs can also cause this.

 

[anowosad]

Are there any statistics how long it takes to resilver a drive depending on size vdev pool number pool utilization

 

[sretalla]

I recently resilvered 3 disks in one VDEV which consisted of 10TB and 12TB disks (last 3 were 12Tb relacing the last 10TB ones).

That VDEV (nominal total storage of 87.2TB) was 10% full.

The resilver took 04:32:05 for the last disk and about the same for the other 2.

In that same pool, I have another VDEV where I have all 10TB disks.

That VDEV (nominal total storage of 72.5TB) is 94.4% full.

I didn't record the time taken for the resilver, but it took well over 24 hours.

edit... (thanks to @jgreco 's excellent point): those VDEVs are 0% and 3% fragmented respectively. Also, both VDEVs are RAIDZ2, 8-wide.

 

[jgreco]

how long it takes to resilver a drive depending on size vdev pool number pool utilization

No, because the biggest driver by far is going to be actual fragmentation and the number of seeks that need to be performed. Resilver times can be catastrophically large (> weeks) on a 4TB drive pool with huge amounts of fragmentation, but less than a day on 14TB drives used for large file storage.

 

[anowosad]

I recently resilvered 3 disks in one VDEV which consisted of 10TB and 12TB disks (last 3 were 12Tb relacing the last 10TB ones).

That VDEV (nominal total storage of 87.2TB) was 10% full.

The resilver took 04:32:05 for the last disk and about the same for the other 2.

In that same pool, I have another VDEV where I have all 10TB disks.

That VDEV (nominal total storage of 72.5TB) is 94.4% full.

I didn't record the time taken for the resilver, but it took well over 24 hours.

edit... (thanks to @jgreco 's excellent point): those VDEVs are 0% and 3% fragmented respectively. Also, both VDEVs are RAIDZ2, 8-wide.

Thank You!
This information is very valuable!
I have found some article from 2016 for zfs resilvering performance. Does something like this exist for TrueNAS Core/Scale resilvering performance?

ZFS: resilver performance of various RAID schemas

From this web page above. Details on that site if you are interesting.

 

[jgreco]

2016 for zfs resilvering performance. Does something like this exist for TrueNAS Core/Scale resilvering performance?

It's the same.

However, you need to understand that this can only ever be a comparative setup. If you are expecting the MINUTES listed in these graphs, you need to be aware that this is complete bovine excrement, because this assumes a fresh pool and minimal fragmentation. This is rarely the case in reality.

 

[anowosad]

It's the same.

However, you need to understand that this can only ever be a comparative setup. If you are expecting the MINUTES listed in these graphs, you need to be aware that this is complete bovine excrement, because this assumes a fresh pool and minimal fragmentation. This is rarely the case in reality.

Can you provide any other real case scenario stats/graphs?

 

[jgreco]

Can you provide any other real case scenario stats/graphs?

To what purpose? It's going to depend on the state of the pool. As I said, I can demonstrate whatever you would like; if you would like to see it taking weeks for a pool of a dozen 4TB drives in RAIDZ2 to complete, all this realy takes is heavy fragmentation.

I can show you a car that can't break 40 miles per hour, and another one that CAN break 200 miles per hour. Neither of these is a good indicator of the performance YOUR car will experience.

 

[Ericloewe]

Basically, everyone's situation is different. We can give you tips on how to avoid the nasty cases, but it's hard to guarantee something like "resilver in 300 minutes or your pizza is free".

 

[pschatz100]

Umm... free pizza got my attention

I have had to resilver the pool on my system two times. When I first built the system, I created a RaidZ2 pool with 4x2TB disks. Two of the disks were very old. After running the system for about three months, one of the old disks failed and was replaced with a new disk. At the time, the pool was about 20% full and I recall that the resilver took roughly 2 hours. Two years later, the other old disk failed. At that time, the pool was 80% full and the resilver took overnight (I don't remember exactly how ling - but at least 8 hours.)

 

[john60]

Is the resilver processing time linear with the size of the disk if all other factors are the same?

I have qty 6 of 4TB disks in a Z2 that is 80% full of very small files. I replaced the first disk with a 16TB, and it took about 14 hours.
My plan is to replace all disks with 16TB drives so that once I am done, I'll have 4x capacity.
When the 16TB disk are 80% full with the same small files, resilver time will be 4x14=56 hours if linear with respect to disk size.
Is this correct?

My system has tons of memory, 50GiB free. I would have assumed the resilver would try to read large chunks of data to increase odds of in-memory hits when it does the parity calculations. Also tones of available CPU during the resilver process. mean idle CPU is 92%. So the processing time is all about speaking specific blocks without any attempts to pre-cache some of this data?

is the re-silver algorithm described somewhere?

 

[Ericloewe]

Is the resilver processing time linear with the size of the disk if all other factors are the same?

Not really, it depends on the volume of data stored, first and foremost.

When the 16TB disk are 80% full with the same small files, resilver time will be 4x14=56 hours if linear with respect to disk size.
Is this correct?

There's far too much going on to make a yes/no statement, but if IOPS stay the same, fragmentation stays the same, metadata continues in a similar manner, etc... Then conceptually, yes.

I would have assumed the resilver would try to read large chunks of data to increase odds of in-memory hits when it does the parity calculations. A

Scrub/resilver must traverse the entire tree of blocks in order, so reading metadata is always going to be slow. However, reads of the actual data referenced by the block pointers are held back and issued in large blocks whenever possible (hence the dual counters in the zpool status output while scrubbing). This represented a meaningful speedup for most applications when it was introduced a few years back.

Also tones of available CPU during the resilver process. mean idle CPU is 92%.

Well, it's no secret that mechanical hard drives have not kept up with improvements in other areas of computing, in terms of speed.

is the re-silver algorithm described somewhere?

Beyond traversing all block pointers one at a time, which is probably rather boring code, the sequential scrub/resilver stuff is described in ZFS dev summit presentations from a few years back. Not to be confused with sequential resilver, which merely copies disks over, then runs a traditional scrub when it's done.

 

[jgreco]

Is the resilver processing time linear with the size of the disk if all other factors are the same?

No. It is a metadata tree traversal with some optimizations, so it is noticeably worse on systems with lots of small files, fragmentation, etc., because it is seeking around to follow the metadata. This allows ZFS to manage a filesystem vastly greater than what NTFS, FFS, or EXT3 could manage.