How do I make a degraded 2 drive RAIDZ?

[Skrenes]

<div class="bbWrapper">Hello everyone. I have a workstation machine running Debian stretch with 24 GB of RAM + 8TB WD Red with an exact setup in another city (at my parents home) and they mirror each other with a nightly rsync script. I'm running out of storage (83% occupied). I've decided to give FreeNAS a shot and bought two more 8TB WD Reds. My goal is to setup the three 8TB WD Reds as RAID5/RAIDZ.<br /> <br /> My question is how to migrate from my occupied drive to the RAIDZ setup. Is it possible to setup the two empty drives as a degraded RAIDZ volume, copy the data over, then repair the RAID using the source drive?</div>

 

[m0nkey_]

<div class="bbWrapper">No, this is not possible within FreeNAS. You will need to backup your data and build the pool with all three drives attached.</div>

 

[Chris Moore]

<div class="bbWrapper"><blockquote data-attributes="member: 89329" data-quote="Skrenes" data-source="post: 482307" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=482307" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-482307">Skrenes said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> My question is how to migrate from my occupied drive to the RAIDZ setup. Is it possible to setup the two empty drives as a degraded RAIDZ volume, copy the data over, then repair the RAID using the source drive? </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>I would say no, but I have been shown to be wrong before, so there may be a more experienced person that knows some trickery from the command line. I still think it is a flat no using the GUI.<br /> <br /> The thing you need to do first though, you need to read some of the documentation to guide you into good choices because it sounds like you are making some less good choices based on incomplete information. Have a look at these resources:<br /> <br /> Slideshow explaining VDev, zpool, ZIL and L2ARC<br /> <a href="https://forums.freenas.org/index.php?threads/slideshow-explaining-vdev-zpool-zil-and-l2arc-for-noobs.7775/" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/</a><br /> <br /> Terminology and Abbreviations Primer<br /> <a href="https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/</a><br /> <br /> FreeNAS® Quick Hardware Guide<br /> <a href="https://forums.freenas.org/index.php?resources/freenas%C2%AE-quick-hardware-guide.7/" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/</a><br /> <br /> Hardware Recommendations Guide Rev 1e) 2017-05-06<br /> <a href="https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/</a><br /> <br /> There are many more useful links under the button in my signature.</div>

 

[HoneyBadger]

<div class="bbWrapper">Possible yes, recommended no. Your RAIDZ pool would be vulnerable to degradation or corrupted data until it is fully resilvered, and 8TB drives are beyond most people's comfort level for use with Z1.</div>

 

[Skrenes]

<div class="bbWrapper">Thanks everyone for your prompt and honest feedback! I spent more than an hour reading the presentation and links provided by Chris. I think I will stick with my Debian setup (which I forgot to mention, uses a boot partition and a LUKS AES-XTS 512bit encrypted partition). I'll use mdadm to expand my one drive setup to RAID 5 and continue mirroring to my remote NAS.<br /> <br /> When I get more time, I will purchase a used 8 hot-swap LFF server with 64GB+ of ECC memory and use FreeNAS.</div>

 

[Chris Moore]

<div class="bbWrapper"><blockquote data-attributes="member: 89329" data-quote="Skrenes" data-source="post: 482332" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=482332" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-482332">Skrenes said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> When I get more time, I will purchase a used 8 hot-swap LFF server with 64GB+ of ECC memory and use FreeNAS. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Closer to the end of the calendar year, or early part of the new year, there is usually a good supply of retired data-center gear on eBay. If you come back with a budget, we can help you find something that will be a good value and fully compatible. There are some hardware requirements that are specific to BSD (the base OS) and ZFS (the file system) so the hardware does need to be chosen with care for the sake of reliability.</div>

 

[danb35]

<div class="bbWrapper">Yes, it's possible, but as <a href="https://www.truenas.com/community/members/36971/" class="username" data-xf-init="member-tooltip" data-user-id="36971" data-username="@HoneyBadger">@HoneyBadger</a> says, strongly discouraged. In general, the process would work like this:<br /> <ul> <li data-xf-list-type="ul"><code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd><s>dd if=/dev/zero of=/root/sparsefile bs=1m count=8m</s></kbd></code> <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>truncate -s 8T /root/sparsefile</kbd></code><br /> </li> <li data-xf-list-type="ul"><code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>zpool create tank raid1 ada0 ada1 /root/sparsefile</kbd></code></li> <li data-xf-list-type="ul"><code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>zpool offline tank /root/sparsefile</kbd></code></li> <li data-xf-list-type="ul">(optional) <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>rm /root/sparsefile</kbd></code><br /> </li> <li data-xf-list-type="ul">Set up your shares, copy your data, etc.</li> <li data-xf-list-type="ul">Through the GUI, do a disk replacement to bring the third disk online and write the appropriate data.</li> </ul>The sparsefile creation works because zfs has compression enabled by default, and zeroes are highly compressible. So, despite being a 8 TB file, it takes very little space on your boot device.<br /> <br /> Now, this procedure doesn't partition your disks and create the swap partition. Nor does it create the pool using gptids. Both of those are typically done by FreeNAS, and are discussed elsewhere in the forums--those details are left as an exercise for the reader.<br /> <br /> Of course, until the disk replacement is complete, you have (potentially) several TB of data on a pool with no redundancy. A disk failure means complete data loss, while a read error will likely mean corrupted data. So, not really a good idea.</div>

 

[Arwen]

<div class="bbWrapper"><a href="https://www.truenas.com/community/members/3713/" class="username" data-xf-init="member-tooltip" data-user-id="3713" data-username="@danb35">@danb35</a>, we probably should have a Resource on how to do that. It comes up every few months. And with a resource, we can make sure the wording has &quot;strongly discouraged&quot; as part of the first sentence.<br /> <br /> Then we can get the details of the correct partitioning and device pathing documented.</div>

 

[Allan Jude]

<div class="bbWrapper"><blockquote data-attributes="member: 3713" data-quote="danb35" data-source="post: 482436" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=482436" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-482436">danb35 said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>dd if=/dev/zero of=/root/sparsefile bs=1m count=8g</kbd></code> </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>You can just use <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>truncate -s 8g /root/sparsefile</kbd></code> instead. It won't take nearly as long as writing out 8gb of zeros (even if ZFS compressed them away)<br /> <br /> You can also use 'gzero', a special device in FreeBSD that makes a huge device that ignores all writes, and returns 0s for all reads. Then just offline it immediately.</div>

 

[danb35]

<div class="bbWrapper"><blockquote data-attributes="member: 36994" data-quote="Allan Jude" data-source="post: 482620" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=482620" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-482620">Allan Jude said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> You can also use 'gzero', a special device in FreeBSD that makes a huge device that ignores all writes </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Sounds like an interesting form of write-only memory. So the syntax in that case would be <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>zpool create tank raidz1 ada0 ada1 gzero</kbd></code>?</div>

 

[HoneyBadger]

<div class="bbWrapper"><blockquote data-attributes="member: 36994" data-quote="Allan Jude" data-source="post: 482620" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=482620" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-482620">Allan Jude said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> You can also use 'gzero', a special device in FreeBSD that makes a huge device that ignores all writes, and returns 0s for all reads. Then just offline it immediately. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Curious as to whether or not that would even successfully create the pool, since even creating the partitions would probably return a fail since it would just return 0's when trying to set things up.<br /> <br /> Edit: Yep, it works. Just do <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>geom zero load</kbd></code> and you'll get the gzero device available. You need to force pool creation with <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>-f</kbd></code> due to mismatched device size but it works. Trying to mirror the FreeNAS-specific partition setup might make it fail but passing the whole device appears to work long enough to fool it. I'll take a stab at documenting it later on with the caveat suggested by <a href="https://www.truenas.com/community/members/40216/" class="username" data-xf-init="member-tooltip" data-user-id="40216" data-username="@Arwen">@Arwen</a> of &quot;you really REALLY shouldn't do this even if you think it's a good idea&quot;<br /> <br /> <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>truncate</kbd></code> should do the trick too, you'll want to specify <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>8T</kbd></code> to match the other drives (and then offline it immediately) - although again, this would result in there being zero redundancy for any blocks written to the Z1 in this configuration.<br /> <br /> And in this case, the OP has signalled their intentions to wait up to get an 8-drive setup, and not play Russian Roulette with their data.</div>

 

[kdragon75]

<div class="bbWrapper"><blockquote data-attributes="member: 36994" data-quote="Allan Jude" data-source="post: 482620" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=482620" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-482620">Allan Jude said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> You can just use <code style="background-color: #dee3e7; padding: .2px; margin: .1em; display: inline-block;"> <kbd>truncate -s 8g /root/sparsefile</kbd></code> instead. It won't take nearly as long as writing out 8gb of zeros (even if ZFS compressed them away)<br /> <br /> You can also use 'gzero', a special device in FreeBSD that makes a huge device that ignores all writes, and returns 0s for all reads. Then just offline it immediately. </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Oh snaps, we got some real devs in here now!</div>

 

[Allan Jude]

<div class="bbWrapper">Yeah, definitely don't run this for long. When I used gzero it was a RAID-Z2, and the extra drive (DOA replacement) was coming within 48 hours. This is definitely a 'footgun'</div>

 

[danb35]

<div class="bbWrapper"><blockquote data-attributes="member: 36971" data-quote="HoneyBadger" data-source="post: 482624" class="bbCodeBlock bbCodeBlock--expandable bbCodeBlock--quote js-expandWatch"> <div class="bbCodeBlock-title"> <a href="/community/goto/post?id=482624" class="bbCodeBlock-sourceJump" rel="nofollow" data-xf-click="attribution" data-content-selector="#post-482624">HoneyBadger said:</a> </div> <div class="bbCodeBlock-content"> <div class="bbCodeBlock-expandContent js-expandContent "> &quot;you really REALLY shouldn't do this even if you think it's a good idea&quot; </div> <div class="bbCodeBlock-expandLink js-expandLink"><a role="button" tabindex="0">Click to expand...</a></div> </div> </blockquote>Or even &quot;<b>especially</b> if you think it's a good idea&quot;...<br /> <br /> Although it gives you, in effect, the ability to turn a two-disk stripe into a three-disk RAIDZ1 at some future time. (Edit: or turn an <i>n</i>-disk RAIDZ1 into an <i>n</i>+1-disk RAIDZ2). If you look at it that way, maybe it isn't such a terribly bad idea--not obviously worse than using striped disks, anyway...</div>

 

[Skrenes]

<div class="bbWrapper">You're all awesome! What a response! This definitely makes me look forward to using FreeNAS in the future. Before I even pursued this upgrade path, I did do a bit of a sanity check on RAID reliabilities with conservative/shoddy MTBF estimates for my drives using <a href="https://www.servethehome.com/raid-calculator/raid-reliability-calculator-simple-mttdl-model/" target="_blank" class="link link--external" rel="nofollow ugc noopener">this nifty resource</a>. It definitely had me nervous using the degraded setup with mdadm, which is why in addition to my source drive, I have the cloned off-site NAS, and a portable external drive, all mirrored and checksummed for good measure. I'd have to have the worst luck in the world to have 3 consecutive drive failures or URE's during this process.<br /> <br /> I'll definitely check back before I purchase a used server to ensure a good fit with FreeBSD and ZFS.</div>

 

[danb35]

<div class="bbWrapper">OK, here's the resource: <a href="https://forums.freenas.org/index.php?resources/creating-a-degraded-pool.100/" target="_blank" class="link link--external" rel="nofollow ugc noopener">https://forums.freenas.org/index.php?resources/creating-a-degraded-pool.100/</a></div>