Replace Disk triggered resilver with an expected runtime of 580-620 hours

[HashManz]

Hi, I mostly use my freenas set-up as a media server.
First time post, so I'll try and follow the posting rules as best I can.
Situation: FreeNas GUI reported that a pool was degraded. This also coincided with the pool being too full, so I had a disk handy that I could add (which was recognized as ada4) . Via the gui, I did the following:
->Clicked on the Storage Tab
->Selected my volume (myprimarystorage)
->Clicked 'View Status' at the bottom of the page
->Selected the degraded disk (8103502573423647997)
->Clicked on the 'Replace' option at the bottom of the page
->Selected the new disk (ada4) and clicked on 'Replace Disk' to start the process.



The process seemed quite slow, so I checked around on the forums and ran the following command to find out more:
zpool status -v
To get the following output:

Code:

  pool: freenas-boot																												
state: ONLINE																													 
  scan: scrub repaired 0 in 0h0m with 0 errors on Thu Mar 23 03:45:58 2017														 
config:																															 
																																	
		NAME										  STATE	 READ WRITE CKSUM													
		freenas-boot								  ONLINE	   0	 0	 0													
		  gptid/a26c1b01-b16d-11e4-a4a2-bcaec543a0ab  ONLINE	   0	 0	 0													
																																	
errors: No known data errors																										
																																	
  pool: myprimarystorage																											
state: DEGRADED																													
status: One or more devices is currently being resilvered.  The pool will														   
		continue to function, possibly in a degraded state.																		 
action: Wait for the resilver to complete.																						 
  scan: resilver in progress since Tue Apr 18 11:39:22 2017																		 
		51.5G scanned out of 21.8T at 10.4M/s, 607h53m to go																		
		11.0G resilvered, 0.23% done																								
config:																															 
																																	
		NAME											  STATE	 READ WRITE CKSUM												
		myprimarystorage								  DEGRADED	 0	 0	 0												
		  raidz1-0										DEGRADED	 0	 0	 0												
			gptid/fbb02c3d-b16f-11e4-9823-bcaec543a0ab	ONLINE	   0	 0	 0												
			replacing-1								   UNAVAIL	  0	 0	 0												
			  8103502573423647997						 UNAVAIL	  0	 0	 0  was /dev/gptid/fccbf3d0-b16f-11e4-9823-bcaec543a0ab																															   
			  gptid/d8786603-23d7-11e7-a9f9-bcaec543a0ab  ONLINE	   0	 0	 0  (resilvering)								 
			gptid/fd2c3e81-b16f-11e4-9823-bcaec543a0ab	ONLINE	   0	 0	 0												
			gptid/fd9598aa-b16f-11e4-9823-bcaec543a0ab	ONLINE	   0	 0	 0												
		  gptid/7deafcdf-1ed5-11e5-8938-bcaec543a0ab	  ONLINE	   0	 0	 0																																								 

So here are my questions:
1. Is it ok to shutdown the server while this process is occuring? I'd use either the command line GUI shutdown option or the web interface GUI button, not a hard power-off.
2. If I can shutdown, can I just remove the bad drive, restart the server, add the new drive to the pool and let it set itself up again (i've ordered another drive just in case)
3. Not storage related - Can I update to FreeNas 9.10 without losing the settings/zpools/jails that I have currently set up?

Additional details regarding my set-up:
Motherboard: Asus Sabertooth X58
CPU: Intel(R) Core(TM) i7 CPU 960 @ 3.20GHz
RAM: 8GB
Storage:

1. 8GB USB Stick as the OS - FreeNAS-9.3-STABLE-201502070132
2. 6xSeagate ST5000DM000 5TB HDD in RAIDZ1

Thanks for any help that can be provided. Also would appreciate any help/tips on the general setup as well (e.g - should go raidz2 - I would be keen but I think im all out of sata slots atm, may need to get a pci-e sata controller if there is such a thing)

 

[nojohnny101]

The pool is currently re-silvering itself, if you shut it down this will interrupt the process. As far as the speed you are seeing for the re-silvering process, it is usually slow starting out and then speeds up. After the re-silvering has been running for about 30 minutes, run the command again and you should see much higher speeds.

Yes you can upgrade to 9.10 and your jails and plug-in should be fine although no guarantees. I haven't read of any widespread problems. 9.10.2-U2 is very stable.

EDIT:
Just want to make sure that you know running raidz1 is highly discouraged and puts your data at risk in the very situation you are in right now. Sure hope your pool doesn't crap out on you and one of your remaining disks don't fail during the resilver.

 

[SweetAndLow]

How full is your pool? Anything over 80% should be addressed.

Sent from my Nexus 5X using Tapatalk

 

[HashManz]

The pool is currently re-silvering itself, if you shut it down this will interrupt the process. As far as the speed you are seeing for the re-silvering process, it is usually slow starting out and then speeds up. After the re-silvering has been running for about 30 minutes, run the command again and you should see much higher speeds.

Yes you can upgrade to 9.10 and your jails and plug-in should be fine although no guarantees. I haven't read of any widespread problems. 9.10.2-U2 is very stable.

EDIT:
Just want to make sure that you know running raidz1 is highly discouraged and puts your data at risk in the very situation you are in right now. Sure hope your pool doesn't crap out on you and one of your remaining disks don't fail during the resilver.

Thanks! it seems to be speeding up now. I was worried since it stayed at around 10 MB/s for 3 hours or so.

How full is your pool? Anything over 80% should be addressed.

Sent from my Nexus 5X using Tapatalk

Yeah, the pool was getting quite overfull (96%!) so I had ordered a disk. sadly one of the disk degraded so I was only able to replace a disk in the pool instead of adding one. I have ordered another disk to replaced the damaged one as well, but that's still in transit. I think that's it for my mobo sata slots, so next stop is a pci-e sata card and 2 more disks to bump up to raidz2. Question though: Can I convert the current raidz1 pool to raidz2 by adding drives and not losing data, or will I have to backup my data first before I change the pool from raidz1 to raidz2?

 

[SweetAndLow]

You can't add a single disk safely and you can't switch to raidz2 without backing up, rebuilding and restoring your data.

Sent from my Nexus 5X using Tapatalk

 

[Vito Reiter]

Just a note to add: Over my years in this industry and seeing all the 'ETAs', I managed to figure out a lot of them don't calculate using current speed vs. amount of data left. A lot of them use time it took to get to x% and multiply it by the y% remaining. So if it took an hour to get to 10%, it would say 9 hours remaining, even if it sped up to 1Gbps from the 10Mbps it was at during 0%-10%. Weird stuff... Really just gotta go by eye with how far along it is and how much is left most of the time, unfortunately.

 

[HashManz]

Just a note to add: Over my years in this industry and seeing all the 'ETAs', I managed to figure out a lot of them don't calculate using current speed vs. amount of data left. A lot of them use time it took to get to x% and multiply it by the y% remaining. So if it took an hour to get to 10%, it would say 9 hours remaining, even if it sped up to 1Gbps from the 10Mbps it was at during 0%-10%. Weird stuff... Really just gotta go by eye with how far along it is and how much is left most of the time, unfortunately.

Yeah, the eta's were off since it uses total time taken vs amount resilvered as opposed to sampling the rate over the last 1/2/5/10 seconds. I did a simple calculation and it seems to be chugging along at 60 MBps which is typically the speed at which files transfer on and off the system when doing a network copy. I hoped it would be faster, but 4 days is better than 25 days.

 

[Arwen]

There are several things related to ETA of RAID disk replacements. In the bad old days, before the wonders
of ZFS , partitions or disks were fully re-synced, regardless of the actual used data. Those do have good
ETA numbers, since it's based on straight forward copying of data. That said, those speeds can fluctuate based
on higher or lower load, like a backup starts or finishes.

ZFS has several quirky things. It will only re-silver used data. At present, it does this by walking the tree,
regardless if it requires massive disk seeks.


There are several improvements coming, one of which is sequential disk re-silvers. That should speed up disk
replacements, (and adding a mirror). If I remember correctly, this may also speed up scrubs.

Another change is the potential use of MetaData Cache devices, (which should be mirrored). If using spinning
disks for data, and using a mirrored pair of SSDs for the MetaData, this reduces the activity to the spinning disks
for re-silvers.