Resilvering takes forever

[ByteNick]

TrueNAS-12.0-U1, RAIDZ2, 8x8Tb Pool, plus 3 spares.
After upgrading to TrueNAS 12 and upgrading the pool, it became degraded and started to resilver...

Estimation is more than 18 days...

Any idea how I can stop this and making work again, without waiting so long? I am asking because nothing really works as it should. Plaxe doesn't play anything, Nextcloud does not really serve files and so on.

 

[joeschmuck]

I wish I had a suggestion for you, maybe someone here with more knowledge could offer advice. A person would think that a U1 release would be rock solid, I'm not that confident although I just upgraded to U1 myself just about 30 minutes ago. I did not upgrade the ZFS version, that won't happen until I know I'm not going back to FreeNAS 11.3-U5, and then it could be another year or more. There are no added features I need from the change. I am surprised that your resilvering becasue of it.

Question, when was the last time a SMART Long test was done and a Scrub?

18 days, you must have a lot of data.

Thinking outside the box here... If you have a full backup of your NAS, you could delete all your data and then the resilvering should be fast (I hope), then copy back all your data. That has got to be faster than 18 days. Also if you wanted to blow your pool away for a different configuration, this gives you that opportunity. I didn't say it was a great idea, just an outside the box idea.

 

[Heracles]

Estimation is more than 18 days...

What is known to cause this is when you have SMR drives...

What exact model your HD are ?

 

[ByteNick]

All of the HDD's are Seagate Ironwolf 8Tb.

 

[jgreco]

All of the HDD's are Seagate Ironwolf 8Tb.

That's a marketing name, not a model number. A model number is like "Seagate ST150176LW".

 

[ByteNick]

That's a marketing name, not a model number. A model number is like "Seagate ST150176LW".

ST8000VN0022-2EL

 

[joeschmuck]

ST8000VN0022-2EL

I wasn't thinking SMR drives at 8TB but they do in fact exist. The model provided appears to be CMR according to the IronWolf website.

 

[ByteNick]

@joeschmuck: This is what Seagate says, but Resilvering still takes forever, SMR like.

 

[ByteNick]

If there are some parameters to change, and you can tell me which ones, that would help a lot.

 

[ExR90]

Go look at the metrics in the reports along disk categories like Latency and IO mainly. If Latency on any of the disks is about about 20 ms, then your disks are the problem. If not, check CPU and see if it's pegged, maybe you have compression on or dedup? Could be your pool is too full (not full, but near full and that kills performance too).

Depending on your controller, it may be the limiting factor too if you have a single SATA lane with SATA-150 or SATA-300 and it's trying to use that single path touching all the disks.

 

[ExR90]

If you have old hardware, especially in the disk controller area and/or CPU, with raidz2 you are going to get punished on an 8 wide pool. That's pretty much at the limit of recommended pool width. Assuming you have somewhat old gear, i'd break that into 2 vdev's (though you'd have to rebuild the pool to do so) at minimum. I personally I use 2 disk mirrors for my vdevs. My main pool has 4 of those mirror vdevs, and 1 hot spare. That scales well and has very good performance. Resilvers on 14TB disks only take about 8 hours roughly.