How to Add Spare to Volume? Is this correct?

[ScottNZ]

Hi All,

I'm relatively new to the forum. Please if this is not the correct place for this question, or it has been answered before, direct me elsewhere.

Background
I've got a 5-drive RAIDZ1 Volume to which I'm wanting to add a "spare", sixth drive, ada1 (yes I overlooked a SATA port when adding the first five disks so the "spare" ended up in ada1. 'Doh!):

The spare-to-be is installed and connected, and I want to make sure I configure it properly. I've looked at the docs, and it seems like I should do this:

1. Storage -> ZFS Volume Manager (see below screen shot image)
2. Volume To Extend: Choose RAIDZVol1
3. Available Disks (lists 1 Drive); Click [+]
4. Volume Layout "(Estimated capacity 0 B)": Choose "Spare"
5. Lists "1 x 1 x 1.0TB; Capacity 0B"
6. Drag and Drop to resize: seems disabled

Questions:

1. Does the above look correct?
2. Is it correct that the "Drag and drop this to resize seems disabled" and "Estimated capacity" reads 0B?
3. At this point should I click <Add Extra Device> OR <Extend Volume> or both, one after the other?
4. Is it true that the "spare" disk does not need any prior initialisation/formatting/etc?
5. When will this spare be used? If a drive fails, do I need to activate the spare through configuration or will it be automatically enlisted in RAIDZVol1?

Many, many thanks for your help--
Regards,
Scott

 

[cyberjock]

Well, spares don't go into use automatically. So I don't use spares and I don't recommend people use them as a general rule. If you can't handle shutting down the server to do a disk replacement and are okay with spinning a disk for months or years and potentially wearing *that* disk out while it does nothing, feel free to do it.

But keep in mind that a spare still requires human intervention to go into use.

 

[solarisguy]

Backup your data and recreate the volume (pool) as RAID-Z2

 

[ScottNZ]

Well, spares don't go into use automatically. So I don't use spares and I don't recommend people use them as a general rule. If you can't handle shutting down the server to do a disk replacement and are okay with spinning a disk for months or years and potentially wearing *that* disk out while it does nothing, feel free to do it.

But keep in mind that a spare still requires human intervention to go into use.

Hi cyberjock,

Hmm, I see what you mean: I was assuming there was some benefit to having the extra, fail-over drive powered up in "spare" status. So I might as well unplug the drive and leave it in the case or reformat with RAID-Z2 as solarisguy mentioned? Or staying with RAID-Z1, if a drive fails, I must expect to down the server and manually replace failed drive, yeah?

I need to research the benefits of RAID-Z2 versus -Z1 with respect to dealing with drive failures. I don't know enough about the Z2 advantages at this point. Would I not still need human intervention to replace a failed drive with Z2?

Thanks,
Scott

 

[Ericloewe]

Hi cyberjock,

Hmm, I see what you mean: I was assuming there was some benefit to having the extra, fail-over drive powered up in "spare" status. So I might as well unplug the drive and leave it in the case or reformat with RAID-Z2 as solarisguy mentioned? Or staying with RAID-Z1, if a drive fails, I must expect to down the server and manually replace failed drive, yeah?

I need to research the benefits of RAID-Z2 versus -Z1 with respect to dealing with drive failures. I don't know enough about the Z2 advantages at this point. Would I not still need human intervention to replace a failed drive with Z2?

Thanks,
Scott

The advantage is quite clear. If one disk fails, you still have enough room for another one to fail. In fact, it's very likely that a RAIDZ1 will fail during a rebuild (the why has been discussed to death).

 

[solarisguy]

You can think of RAID-Z2 as if it were a RAID-Z1 volume with an active spare, that is a spare that automagically goes in after a disk failure.

If you are using your FreeNAS only through a single Gigabit Ethernet network connection, given your CPU, you will not be able to observe the performance difference between RAID-Z1 and RAID-Z2 configurations.

 

[joeschmuck]

In fact, it's very likely that a RAIDZ1 will fail during a rebuild (the why has been discussed to death).

I doubt that statement is based on actual data at all. I could believe neglect and ignorance could cause a pool to be lost while resilvering or someone using a bunch of crappy old drives that were on thier last leg even before building the FreeNAS server, but to say a drive will more than likely fail just because you are resilvering a drive, I'm not buying it.

@ScottNZ, depending on your expected use you should use a RAIDZ1, RAIDZ2, or a RAIDZ3. Most people go for a RAIDZ2 to provide them a safety margin of two failed drives and you still have your data, but a third failed drive means complete loss. If this is for a business then you have built your system incorrectly as you are not using ECC RAM. I suspect this is for home use so I'd recommend a RAIDZ2.

Since you are not using ECC RAM which is taboo for a ZFS file system, make sure you have tested out your RAM very well, possibly for several days to ensure you have zero errors. Use MemTest for this. Run a CPU stress tester for about 2 hours (don't over do it, it can harm a CPU if done too long) and ensure no issues there as well.

 

[Ericloewe]

I doubt that statement is based on actual data at all. I could believe neglect and ignorance could cause a pool to be lost while resilvering or someone using a bunch of crappy old drives that were on thier last leg even before building the FreeNAS server, but to say a drive will more than likely fail just because you are resilvering a drive, I'm not buying it.

The logic goes something like "Every single written sector will be read to rebuild the array. This can cause a drive that is just about to fail to go over the edge. Additionally, any single unreadable sector will cause the whole process to fail."

The first part is the dubious one, as it would amount to a freak accident. The second part is a lot more realistic, especially if regular scrubs weren't done properly. It also relies on statistics - since per bit failure rates haven't gone down (at least according to manufacturers' specs) and drives are larger, each drive will, on average, have more errors.

Of course, you're absolutely right - there's very little data available for these scenarios. The closest thing to hard numbers is Serve The Home's Mean Time To Data Loss calculator, which is relatively opaque as to some of its assumptions.

Honestly, in cases like these, the pessimist within me takes over and I tend to overestimate the risk of some of these scenarios.

 

[ser_rhaegar]

I doubt that statement is based on actual data at all. I could believe neglect and ignorance could cause a pool to be lost while resilvering or someone using a bunch of crappy old drives that were on thier last leg even before building the FreeNAS server, but to say a drive will more than likely fail just because you are resilvering a drive, I'm not buying it.

Consumer drives have a URE of 1E14. This pretty much guarantees 1 URE per 12TB read/written. So if you rebuild an array that is over 12TB (1E14) in size, you're going to hit a URE. If you're rebuilding a RAID1/RAIDZ1, game over.

Enterprise drives have 1E15 and 1E16 URE ratings depending on the brand/model. You're safer using an enterprise drive in a RAID1/RAIDZ1 than a consumer drive for this reason.

 

[joeschmuck]

Consumer drives have a URE of 1E14. This pretty much guarantees 1 URE per 12TB read/written. So if you rebuild an array that is over 12TB (1E14) in size, you're going to hit a URE. If you're rebuilding a RAID1/RAIDZ1, game over.

Enterprise drives have 1E15 and 1E16 URE ratings depending on the brand/model. You're safer using an enterprise drive in a RAID1/RAIDZ1 than a consumer drive for this reason.

So those are statistics from the manufacturer and I agree that you "could" run into that situation however typically when you run into one URE, you are likely to run into many more very soon for that drive and it will fail. Based on the statistics alone I should have at least hit one URE for one of my six drives by now but I haven't. I can believe it will happen just as they start failing. This is also one of the reasons to purchase drives from different lots if you can, just to stagger any possibility of all your drives failing within a week or so of each other.

As for 1E15 and 1E16, from what I've read [and this is not factual data that I am aware of] the ratings may be higher because the drives are smaller in capacity and thus not using the entire platter so this ends up raising the URE value. This makes sense to me but as I indicated, I didn't find anything to state this is factual. One thing I will not do is flat out lie just to make a point, however I could just be plain wrong and I can accept that if it's true.

Just to make sure that I was clear, I fully support RAIDZ2 and I feel it's a smart choice and worth every dollar I spent on drives for it.

 

[solarisguy]

Just had enough time to look at non-recoverable read errors per bits read (the same as URE in this thread) for 4TB drives from Western Digital:

• 1 in 10^13 for WD AV, WD Purple
• 1 in 10^14 for MegaScale DC 4000.B, WD Black, WD Green, WD Red and WD Se
• 1 in 10^15 for Ultrastar 7K4000, WD Re

If you want better values of that parameter, then 1.2TB is the maximum available:

• 1 in 10^16 for Ultrastar C10K1200 (SAS only 10,000 rpm, 1.2TB)
• 1 in 10^16 for WD Xe (enterprise performance series 10,000 rpm, 900GB)

Conclusions? With drives that have 1 in 10^15, I would be more likely to deploy RAID-Z2, than RAID-Z3. So suddenly 6TB Ultrastar He6 does not look that expensive, since it also has 1 in 10^15...

 

[ScottNZ]

Hi All,

Great discussion--thanks. I've done more research as well (inlcuding cyberjock's .pptx Guide), and now wish I had gone with RAIDZ2. So, looks like a backup and re-install. My system is a home system, but I'm a contractor and it's going to house my contract work as well as family media so it may behove me to consider it more of a business system. I'll re-evaluate my hardware as well, specifically the ECC RAM.

What's the most efficient way to "start over" with RAIDZ2? Can I do a full-install (versus upgrade) over the top of my existing setup or do I need to drop volumes/reformat/etc?

Thanks,
Scott

 

[joeschmuck]

You unfortunately need to recreate your pool meaning destroying your current one. Backup all the data you can before destroying your pool. As for your system not having ECC RAM, you can relocate the hard drives and the bootable USB flash drive to another computer that does support ECC RAM and it will work fine, no need to reload your data. So this allows you time to build up a good computer for this use.

Last thing and this is probably the most important one... Backup you important data to CD/DVD or some other media. You never know when it will save your life, especially if you have work product to protect.

 

[Ericloewe]

Last thing and this is probably the most important one... Backup you important data to CD/DVD or some other media. You never know when it will save your life, especially if you have work product to protect.

Definitely keep several backups (offsite even), but don't trust CDs/DVDs for long-term storage - recordable ones don't last long, unfortunately.

 

[ScottNZ]

OK thanks. I'm OK with destroying the existing pool (it stings a little bit, but it's a learning experience I won't forget). If I do a full install from a CD, will that overwrite my current pool and allow me to re-initialise everything, create a new pool, etc. I've come to terms with the fact I'll have to recreate everything else, datasets, shares, cron jobs, and so on. Any "gotchas" like the installer detecting the existing pool and offering to reuse it?

I'm going to try it, but I'd like to know if that's the correct approach. Luckily I don't have much on the system yet, so it's no big loss except for configuration time.

Thanks,
Scott

 

[solarisguy]

If you select Detach Volume, and then on the first screen select Mark the disks as new (destroy data) afterwards you have blank disks, as far as FreeNAS is concerned.

 

[ScottNZ]

@solarisguy: I think that's exactly what I need--thanks. I'm going to go with your suggestion and rebuild all six drives with RAIDZ2 instead of five Z1's plus one "spare". I may be able to save an hour or two of service and script configuration, etc., if I don't have to reinstall from scratch.

Cheers,
Scott

 

[joeschmuck]

Yea, you don't need to reinstall FreeNAS at all, just destroy your current pool and recreate a new pool as RAIDZ2. As for script configurations... where do you have those stored? If they are on your pool, copy them off first. Now I am assuming the "script configuration" is something other than the FreeNAS config file.

 

[ScottNZ]

Thanks everyone--it worked smooth as a Brazilian.

I Detached the 5-disk RAIDZ1 volume, marked the disks as new/destroy data, and created a new RAIDZ2 volume using all six disks. Sweet! All my users, groups, cron jobs, SMART tests, and other misc. settings remained. I did delete and later recreate all the shares, but that was minor. If there is an even simpler way, I don't want to know about it at this point haha.

Now I need to determine if/how well my mobo supports ECC RAM. Thanks very much for all your help.

Scott.

 

[Ericloewe]

Now I need to determine if/how well my mobo supports ECC RAM. Thanks very much for all your help.

Scott.

I'll save you the trouble. It doesn't. Zxx, Xxx, Qxx, Pxx and Hxx chipsets do not support ECC (yes, it's stupid since the memory controller is in the CPU, but it's how Intel made the things). You'd need a C22x chipset.
Your processor doesn't support ECC either. Only Xeons and some i3s and Celerons do.