<div class="bbWrapper">At the rhythm my <a href="http://forums.freenas.org/showthread.php?1310-My-new-NAS-box-running-on-8-0-2" target="_blank" class="link link--external" rel="nofollow ugc noopener">7TB RaidZ2 array</a> is getting filled, I'm looking to upgrade the disks within 6 months. I don't plan to build a separate system but rather replace one disk at the time and resilver everything. Basically, I plan to move from WD 2TB disks to <a href="http://www.storagereview.com/hitachi_deskstar_7k4000_review" target="_blank" class="link link--external" rel="nofollow ugc noopener">Hitachi Deskstar 4TB disks</a> so that should expand my current array size to 14TB. The X7SPA-HF-D525-O motherboard handles fine 4TB drives, based on what <a href="https://www.google.com/search?q=X7SPA-HF-D525-O+Hitachi+Deskstar+4TB" target="_blank" class="link link--external" rel="nofollow ugc noopener">Google says</a>. Obviously, I will not use the disks at their full speed capacity but I'm happy with a 80MB/sec transfer rate I get now.<br />
<br />
Now, can you please advice what is the proper way to replace and resilver the array disks? I'm wondering how long it could take to resilver each disk on a 5TB volume.</div>
<div class="bbWrapper">Hi TECK,<br />
<br />
I believe the current documentation has the correct procedure for doing what you ask. Thanks for posting about the 4TB disks, I think I am going to do the same, although I'd like to convert it to a 6 disk z2 array instead of the 5 disk array I currently have.<br />
<br />
The documentation says "failed drive", but it is the same to replace with a larger drive.<br />
<br />
<a href="http://doc.freenas.org/index.php/Volumes#Replacing_a_Failed_Drive_or_Zil_Device" target="_blank" class="link link--external" rel="nofollow ugc noopener">http://doc.freenas.org/index.php/Volumes#Replacing_a_Failed_Drive_or_Zil_Device</a></div>
<div class="bbWrapper">Thanks for the link, Proto. Based in your experience, how long you think it will take to resilver a disk? I'm curious how you will increase the array size, I guess you will copy your data elsewhere in order to define a proper RaidZ2 volume?</div>
<div class="bbWrapper">On our systems, I think the last time I replaced a disk it took 10 hours. Since the array doesn't expand until all the disks are replaced, the resilver shouldn't be any longer replacing a 2TB with a 4TB. After the array is expanded and those disks get more data added, THEN it could take AWHILE...</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="TECK" data-source="post: 37593"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37593"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37593">TECK said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
I'm wondering how long it could take to resilver each disk on a 5TB volume.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>Depends on a number of things. The amount of data in the <i>zpool</i>, load on the array and lest we forget the base FreeBSD ZFS code, there are changes 8.2 vs 8.3.<br />
<br />
<blockquote data-attributes="" data-quote="TECK" data-source="post: 37593"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37593"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37593">TECK said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
Now, can you please advice what is the proper way to replace and resilver the array disks?
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>The proper way is what <b>protosd</b> already said.<br />
<br />
Of course if you are feeling adventurous there is an alternative that is faster and places less strain on all the drives. Ran across a post by <a href="http://mail.opensolaris.org/pipermail/zfs-discuss/2011-June/048883.html" target="_blank" class="link link--external" rel="nofollow ugc noopener">PJD@</a> earlier today and it also works under <a href="http://mail.opensolaris.org/pipermail/zfs-discuss/2011-June/048905.html" target="_blank" class="link link--external" rel="nofollow ugc noopener">Solaris</a> as well.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="paleoN" data-source="post: 37604"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37604"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37604">paleoN said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
Of course if you are feeling adventurous there is an alternative that is faster and places less strain on all the drives. Ran across a post by <a href="http://mail.opensolaris.org/pipermail/zfs-discuss/2011-June/048883.html" target="_blank" class="link link--external" rel="nofollow ugc noopener">PJD@</a> earlier today and it also works under <a href="http://mail.opensolaris.org/pipermail/zfs-discuss/2011-June/048905.html" target="_blank" class="link link--external" rel="nofollow ugc noopener">Solaris</a> as well.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote><br />
Hmmm, interesting. I would think just to be safe it would be good to do a scrub after this method, but I guess that defeats the purpose of doing it this way.<br />
<br />
I would also highly recommend using <b>ddrescue</b> versus dd. It allows you to resume the copy exactly where it was interrupted, and is just nice if you want to pause for some reason and resume later. Don't forget to include the log option if you do use ddrescue. Also, there are 2 versions of ddrescue, I prefer not to use the GNU version.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="protosd" data-source="post: 37605"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37605"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37605">protosd said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
I would also highly recommend using <b>ddrescue</b> versus dd.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>Wasn't thinking about that, good idea.<br />
<br />
Given that you are increasing the size of the drives, at the least you will 'lose' the two redundant ZFS labels at the end of the drive. They will end up somewhere in the middle. I believe they will be recreated when you <b>online</b> the disk. You can always check with:<div class="bbCodeBlock bbCodeCode"><div class="type">Code:</div><pre>zdb -l /dev/adaXpX</pre></div>Or whatever you used to make the <i>zpool</i> with.<br />
<br />
<br />
Now if you had all the replacement disks on hand to begin with this is much safer.<br />
<br />
<ol>
<li data-xf-list-type="ol">Backup your FreeNAS config<br />
<br />
</li>
<li data-xf-list-type="ol">Export pool<br />
<br />
</li>
<li data-xf-list-type="ol">Match serials between old & new disks<br />
<br />
</li>
<li data-xf-list-type="ol">In a separate box, if you have one, carefully <b>ddrescue</b> old disk/partition to new disk/partition.<br />
You can do this at the same time for as many pairs as you have SATA ports. :D<br />
<br />
</li>
<li data-xf-list-type="ol">Install new <i>dd</i>'ed disks into the NAS and power it on.<br />
<br />
</li>
<li data-xf-list-type="ol">Import pool</li>
</ol>If all goes as planned the pool should autoexpand after import and the ZFS labels should be fixed, I <i>think</i>.<br />
<br />
Instead if it blows up in your face, wipe the metadata from the new drives, remove the new drives, restore your old config, put back the old drives and do it the proper way.</div>
<div class="bbWrapper">Thank you for the useful info, guys. Unfortunately, the data I have on my NAS does not allow me to "play" with alternative solutions. If someone tested the ddresque procedure, I will gladly do it if it works 100%. Otherwise, I would rather stick with the longer but safe alternative.<br />
<br />
I will definitely order all disks at once. I have a spare box running on CentOS 6 but I'm not comfortable fiddling with partitions and changing disk serials. I don't even know where the disk serials are changed, obviously it has to be on some partition data. Anyways, I cannot take the chance to "hope" things will work, I need some confirmation from anyone else who did it successfully and increased their array safely with the method you mentioned.</div>
<div class="bbWrapper">There is something unclear on the FreeNAS disk replacement procedure:<br />
"<i>Once the disk is showing as OFFLINE, click the disk's Replace button. Select the replacement disk from the drop-down menu and click the Replace Disk button. If the disk is being added to a ZFS pool, it will start to resilver.</i>"<br />
<br />
In theory, if I swap the disks, the new disk should retain the same ID (i.e. ada1) and will be inserted into array as ada1p2. I hope FreeNAS will not see the new disk with a incremental value ada7 and add it to array as ada7p2?</div>
<div class="bbWrapper">The disk retaining the same ID (I think) is based on the hardware and how the driver works. Ultimately it really doesn't matter if it's assigned ata152. When you reboot the numbers will be reset from 0 and it'll all still work. ;)<br />
<br />
I did an upgrade on one of my FreeNAS servers by resilvering each disk. I had 2 vdevs so I did 1 disk in each VDEV at the same time. What I did:<br />
<br />
1. Shutdown the server<br />
2. Pulled out 1 drive from each vdev(Both were RAIDZ2)<br />
3. Inserted 2 new disks.<br />
4. Booted up the server.<br />
5. Used the "replace" button and selected a new disk.(resilvering would start after this step)<br />
6. Used the "replace" button for the other disk and selected the other new disk.(resilvering would re-start after this step)<br />
7. Waiting for resilvering to complete.<br />
8. Go to step 1.<br />
<br />
I had zero issues except having to wait for the resilvering.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="TECK" data-source="post: 37645"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37645"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37645">TECK said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
In theory, if I swap the disks, the new disk should retain the same ID (i.e. ada1) and will be inserted into array as ada1p2. I hope FreeNAS will not see the new disk with a incremental value ada7 and add it to array as ada7p2?
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>Do you have an empty bay available? From what I've read it's considered better to replace a drive while the original is still around. Given that you are using double-parity it's not as much of a concern.<br />
<br />
Assuming you have the new disk plugged into the same SATA port, I would <b>not</b> expect the disks to be enumerated any differently. If you are concerned about it you can always <i>wire down</i> the device names to a particular port.<br />
<br />
<blockquote data-attributes="" data-quote="TECK" data-source="post: 37643"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37643"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37643">TECK said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
I have a spare box running on CentOS 6 but I'm not comfortable fiddling with partitions and changing disk serials.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>The <i>match serials</i> step was for keeping track of which disks are which. On a piece of paper record the new disk serial <i>#NEW***xx</i> and beside it the old disk serial <i>#OLD***xx</i>. Then install the new disk in the correct spot, not that it will matter much to ZFS. You don't, not to mention can't, change the disk serial number using the method above.<br />
<br />
No partition fiddling necessary. You would clone the contents of the partition if you used partitons. If you used whole disks you don't have partitions and don't have to think about them.<br />
<br />
<blockquote data-attributes="" data-quote="TECK" data-source="post: 37643"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37643"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37643">TECK said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
Anyways, I cannot take the chance to "hope" things will work, I need some confirmation from anyone else who did it successfully and increased their array safely with the method you mentioned.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>The above method is safe, as safe as anything is with <i>dd/ddrescue</i> commands. If you are uncomfortable with them, which is understandable, then you should do it the proper way. As far as confirmation, a scrub will confirm it worked. No hope needed.<br />
<br />
When i first posted the above, I don't think I would have tried it with my own data. I still wouldn't if I didn't have all the disks at the start. I would try it myself, but I still need to expand the number of drives I have.</div>
<div class="bbWrapper">The "proper way" assuming 10 hours per resliver with 6 disks is <b>60 hours</b>.<br />
<br />
The <i>adventurous way</i> is quite a bit less. You have at least 7 SATA ports available which means you can do up to 3 pairs of disks at once. Assuming a speed of 100 MB/s with 2TB to clone will take about 5.8 hours per disk. You can run 3 clones in parallel. For a grand total of 11.6, let's say an even 12 hours.<br />
<br />
That's <b>2 full days</b>, 48 hours, less!<br />
<br />
<b>TECK</b>, given that you aren't doing this for another 6 months perhaps someone else will come along and try it first.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="paleoN" data-source="post: 37688"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37688"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37688">paleoN said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
Do you have an empty bay available?
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>I don't, all SATA ports are full.<br />
<br />
<blockquote data-attributes="" data-quote="paleoN" data-source="post: 37691"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37691"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37691">paleoN said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
The "proper way" assuming 10 hours per resliver with 6 disks is <b>60 hours</b>.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>That is not an issue for me. I plan to get the 6 disks (I expect to spend about $1,500) and do 2 disks/day. One in the morning before I leave for work and one at night when I arrive at home. It should take me 3 days to finish everything and will have a lot of spare data once completed.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="TECK" data-source="post: 37713"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37713"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37713">TECK said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
It should take me 3 days to finish everything and will have a lot of spare data once completed.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>So, about 72 hours. Your procedure is sound, more than reasonable and shouldn't have much impact on you as you have noted.<br />
<br />
<br />
For those interested in the <i>adventurous way</i>, I have successfully tested it in a FreeNAS-8.2.0-RELEASE-p1-x64 VM with a 3 x <i>raidz1</i> array.<br />
<br />
The original disks were 10GB each, 8GB partitions. The new disks are 20GB each, 18GB partitions. Following the procedure I outlined in [post=37607]post #7[/post] all worked as I thought.<br />
<br />
The new disks only 'had' the first 2 ZFS labels as the other two were in the middle of the partition and not where they belong. What's more they refer to the old disk names. ZFS, <i>zpool import</i>, takes all this in stride for us. :) On import the ZFS labels were updated and the pool autoexpanded to 36GB. Ran a scrub and it came back clean.<br />
<br />
Advantages:<ul>
<li data-xf-list-type="ul">Orders of magnitude faster depending on the number of drives & pool configuration</li>
<li data-xf-list-type="ul">A pristine, untouched "backup" of your pool, it's the complete original pool, on the old drives</li>
<li data-xf-list-type="ul">Very low stress on all the drives involved<br />
One long continuous streaming read/write is among the easiest workloads for a hard drive</li>
</ul><br />
Disadvantages:<ul>
<li data-xf-list-type="ul">Shall we say <i>somewhat</i> technical :p</li>
<li data-xf-list-type="ul">Error prone & unforgiving<br />
Know with absolute certainly which disks are which and what <b>ddrescue</b> commands you are running</li>
<li data-xf-list-type="ul">Others I haven't thought about?</li>
</ul><br />
I should be nuking my mirror next week or even this week in the course of some testing. I think I will simulate this by manually partitioning them to a 1TB partition and later growing them to their full size.</div>
<div class="bbWrapper">Here's some advice.. we found out after we were 3/4 of the way through. Our drives were setup as individual disks on a RAID controller. By enabling disk writes and disabling all read caching on the controller we shaved off almost 35% from the resilvering time. This seems logical since ZFS has it's own read cache that is much smarter than the controller. The write cache seemed to optimize the write pattern to the disks being resilvered.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="noobsauce80" data-source="post: 37846"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37846"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37846">noobsauce80 said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
This seems logical since ZFS has it's own read cache that is much smarter than the controller. The write cache seemed to optimize the write pattern to the disks being resilvered.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>Yes, I've read as much. In fact depending on the number of disks behind the controller and the workload you may get even greater performance by turning off the write cache. A controller flashed to IT mode, if possible, can also see an additional increase in performance.<br />
<br />
AFAIK, a resilver <i>should</i> be limited by, among other things, the random write performance of the resilvering disk(s). A heavily fragmented pool will make this workload even worse. Assuming you are trying to avoid using the pool during the resilver I would be surprised if turning off the write cache would be any faster.<br />
<br />
If I'm not mistaken <b>TECK</b> has a HBA anyway and not a RAID card.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="paleoN" data-source="post: 37852"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37852"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37852">paleoN said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
A heavily fragmented pool will make this workload even worse.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>Will a scrub help before I perform the upgrade? Is there a better method to defrag the volumes?<br />
<br />
<blockquote data-attributes="" data-quote="paleoN" data-source="post: 37852"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37852"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37852">paleoN said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
If I'm not mistaken <b>TECK</b> has a HBA anyway and not a RAID card.
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>Correct, I use the 6 SATA ports available on Supermicro motherboard.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="TECK" data-source="post: 37866"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37866"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37866">TECK said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
Will a scrub help before I perform the upgrade? Is there a better method to defrag the volumes?
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>There is currently only one way to defrag ZFS. Backup the entire pool, <i>zfs send</i>/<i>rsync</i>, destroy the original pool, recreate the pool and restore from backup.<br />
<br />
What makes you think your pool is heavily fragmented? Two of the more significant things that cause fragmentation are creating & deleting lots & lots of snapshots and using up all the free space in your pool.</div>
<div class="bbWrapper"><blockquote data-attributes="" data-quote="paleoN" data-source="post: 37874"
class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch">
<div class="bbCodeBlock-title">
<a href="/community/goto/post?id=37874"
class="bbCodeBlock-sourceJump"
rel="nofollow"
data-xf-click="attribution"
data-content-selector="#post-37874">paleoN said:</a>
</div>
<div class="bbCodeBlock-content">
<div class="bbCodeBlock-expandContent js-expandContent ">
What makes you think your pool is heavily fragmented?
</div>
<div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div>
</div>
</blockquote>Is not, I have only few fragments. I was just curious what is the proper way to defrag a volume.</div>
This site uses cookies to help personalise content, tailor your experience and to keep you logged in if you register.
By continuing to use this site, you are consenting to our use of cookies.
