Disk resilvering takes forever

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
Hi,
One of my hard drive in raid1 failed. So I replaced it with the same model (wd 4tb red model). The replication started 3 days ago and it's still not finished... That sounds like a long time to transfer around 3.5 TB of data from one hard drive to another on the same machine...
I run FreeNAS 9.3.
Anyone had similar issues before?
 
Last edited by a moderator:

wblock

Documentation Engineer
Joined
Nov 14, 2014
Messages
1,506
That is not replication, but resilvering. Is your 4TB drive really that full? Because overfull drives get a lot slower.
 

rs225

Guru
Joined
Jun 28, 2014
Messages
878
If you can run the command zpool status you should see how much time is estimated for completion of the resilver.
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
Thanks for correcting my title! I will try this command line tonight and see what's happening ;)
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
It really takes a long time, it is still going since last Saturday. according to the zpool command i am 76% done ...
 
Last edited by a moderator:

nojohnny101

Wizard
Joined
Dec 3, 2015
Messages
1,477
Please post a screenshot or picture of the zpool status screen. What is the rate of resilver?
 

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
You haven't answered the question above... how full is that pool? The output of zpool status and zpool list would help. Resilvering will suck if the pool is nearly full.
 
Last edited by a moderator:

RegularJoe

Patron
Joined
Aug 19, 2013
Messages
330
is one a 512k sector drive and the new one is a 4k sector drive? Does the new drive have a firmware bug? have you done a diskinfo -citv /dev/xx to see if both perform the same? have you run a smart test short and long on the new drive?
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
Ok im not sure what is going on, it seem to resilver the disk in loop, restarting from scratch all the time.
Screen_Shot_2018_01_27_at_10_27_13_AM.png

as you can see it went back to 63% here....


Screen_Shot_2018_01_27_at_10_28_02_AM.png


this is my zpool list:
videos 3.62T 2.89T 749G - 30% 79% 1.00x DEGRADED /mnt


What do you think I should do? How do I stop the resilvering ?

regular Joe, to answer your questions:
is one a 512k sector drive and the new one is a 4k sector drive?
iam not sure, but both of my hard drive are the same model : WD40EFRX

Does the new drive have a firmware bug?
I don't know

have you done a diskinfo -citv /dev/xx to see if both perform the same?
no

have you run a smart test short and long on the new drive?
no
 
Last edited by a moderator:

tvsjr

Guru
Joined
Aug 29, 2015
Messages
959
The drive that you're using as a replacement is throwing MASSIVE numbers of checksum errors. And, you have errors on your other drive as well... so, no matter what, you've lost data. The cause of the checksum errors could be a bad drive (this is why we encourage doing burn-in tests before placing a drive in service) or hardware problems - like a bad HBA/SATA controller, bad or undersized PSU, cable issue, etc.
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
The drive that you're using as a replacement is throwing MASSIVE numbers of checksum errors. And, you have errors on your other drive as well... so, no matter what, you've lost data. The cause of the checksum errors could be a bad drive (this is why we encourage doing burn-in tests before placing a drive in service) or hardware problems - like a bad HBA/SATA controller, bad or undersized PSU, cable issue, etc.
Ok I see... What should I do next? How do I stop the resilvering?
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
Should I send those hard drive back to wd?
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
I had a similar problem when replacing a disk, and the errors were caused by bad power to one of my drives. I was using a molex to sata power adapter for one of the drives and it was bad.

Make certain that all your connectors are good and properly connected - both sata and power.
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
Thanks for your answer Pschatz. I will try this too.
But i need to stop the resilvering safely first. It seem that it's not possible ... I've looked into the forum, but no one have a real answer for this.
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
My guess is i am gonna have to force stop it ?
 

rogerh

Guru
Joined
Apr 18, 2014
Messages
1,111
My guess is i am gonna have to force stop it ?
It might be best to select the new disk in the GUI disk view and offline it. If it won't do that because of the errors and running a scrub does not help then I agree it will be necessary to shutdown FreeNAS and physically remove it. Then a scrub might recover as much as possible from the remaining disk and allow a new one to be added.
 

pschatz100

Guru
Joined
Mar 30, 2014
Messages
1,184
In my situation, I never forced the resilver to stop. I did a system shut down and then checked all my cables and connections. I replaced the molex adapter with another one I happened to have on hand. When I powered up the system again, the resilvering restarted. I checked the disks under the volume status tab and the write errors were gone. Resilvering progress seemed to be better, so I let it run to completion. Then I ran a scrub on the volume.

At the end of the day, I don't know whether or not I actually lost any data. I haven't noticed any, but I do keep good backups - just in case.
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
Thanks!
yes so I did force shut down the server and then unplugged the new hard drive. Then when I restarted I wanted to scrub the volume. But I can't, FreeNAS still think it resilvering...
Code:
[root@freenas] ~# zpool scrub -s videos
cannot cancel scrubbing videos: currently resilvering


Now I am in the process of saving all my data so I can recreate a healthy volume from scratch...
what should i do with the new hard drive? should I return it?
 
Last edited by a moderator:

rogerh

Guru
Joined
Apr 18, 2014
Messages
1,111
Test it in another computer (SMART long test, badblocks if you are feeling thorough). If bad, return. If good, look into cable or power problems. Or possibly MB or memory if you still have problems with the FreeNAS machine.
 

nicosalto

Dabbler
Joined
Nov 25, 2016
Messages
40
Thanks Rogerh,
I will test them with SMART long test. Since my data is now all offline, I was thinking to take the time to expand my pool with 2 extra 4T hard drive and switch to RAID 5 or 6. And also update to FreeNAS Corral as well.
I hope I wont have any more bad sectors issues!
 
Last edited by a moderator:
Top