SOLVED All disks online but array shows degraded.

[some_guy_named_ed]

So here's the back story -

I woke up this morning to find my array in a degraded state. Without thinking too much, I identified the faulty drive and pulled it from the chassis. I replaced it with another drive. When I noticed it didn't re-silver right away, I started poking around. I used the gui option to extend the array with the new disk and added it as a spare.

Immediately it started resilvering and eventually completed. I thought to myself "Yay! I didn't break it" but lo and behold it's still in a degraded state.

I'm sure it's actually fine and needs some sort of metadata cleanup or something because I botched a proper procedure somewhere. All shares are up and speed seems nominal. Can anybody advise me on what to do next to clean things up? I've attached a screengrab at what I'm looking at in the GUI. The offline disk is the one I removed and for some reason, the new one (da15p2) is in two places now.

Thanks for any help on this.

Ed

 

[PhiloEpisteme]

I'm sure it's actually fine and needs some sort of metadata cleanup or something because I botched a proper procedure somewhere.

In general, it is recommended that for any procedure which affects drives you check the User Guide and double-check your steps. In this case, you pulled a drive but had hot spares configured. The drive you pulled was then replaced with a hot spare, this is why it started resilvering.

When a hot spare replaces a drive due to failure it is considered temporary until you either replace the originally failed drive, in which case the hot spare goes back to being a hot spare, or formally remove the failed disk from the pool, in which case the hot spare is promoted to a permanent member of the pool. Have you attempted to click the ... for /dev/gptid/ . . . disk and removing it from the pool? Once that drive is removed from the pool, da15p2 should be promoted to permanent member and the degraded status removed.

 

[some_guy_named_ed]

So if I'm reading what you're saying correctly - My array is in fact good for the moment. The spare has taken over for the dead drive but is designated as a 'spare in use' rather then a a proper array member. I hope I understand that correctly.

In order to re-designate the drive, I have to remove the dead drive. Unfortunately, in the '...' menu I have no option to remove it. Only edit (which does nothing), online, or replace.

Also, my Freenas version is "FreeNAS-11.3-U3.2" if that helps. Do I need an update or is there a console way of removing the dead drive?

Thanks

 

[some_guy_named_ed]

I think I found the issue. Could this bug be the problem?

https://jira.ixsystems.com/browse/NAS-106518

I'm going to update and test and close this out if it works.

 

[PhiloEpisteme]

So if I'm reading what you're saying correctly - My array is in fact good for the moment. The spare has taken over for the dead drive but is designated as a 'spare in use' rather then a a proper array member. I hope I understand that correctly.

More or less, yes. If you were running zfs from the command line you'd just remove the "failed" or removed drive from the pool and that should fix it.

The bug you listed certainly seems like it could be related.

I also wanted to mention a few bits about using hot spares. They are great if a drive outright fails or is removed. In these cases, as has happened here, the spare steps in and replaces the bad drive. However, they only do that when a full failure has occurred. If you set up proper reporting (you may have already done this) and replace a drive as soon as it starts to show issues such as SMART errors, you can typically replace it before it outright fails. In these latter cases the spare is never used. IMO the most typical value to be gotten out of hot spares is in situations where you do not have ready access to the server to replace failed disk.

 

[some_guy_named_ed]

Thanks for all that info. Just wanted to say that the update was definitely needed. Once done, the detach option appeared and I was able to remove the bad drive. In my bumbling around I'd have probably done that sooner as it would have made more sense to me and avoided this whole ordeal. Whether you want to say it was user error or the bug, I learned something and managed not to lose any data in the process. So I'm happy.

Thanks again.