FreeNAS 11.1 system, unreadable sectors popup

[Member]

Currently on FreeNAS 11.1-RELEASE
i3 4170, 16GB ECC, 4x4TB WD RED, RAIDZ2
I have had this machine since 2016 Summer, have never seen this message.

I just received this message in my Freenas Alerts:

So, I followed instructions on how to fix some of the errors https://forums.freenas.org/index.ph...-1-currently-unreadable-pending-sectors.9824/

1. I performed long tests on both ada0 and ada1:

2. Tried to write to sector using dd ex:

Code:

sysctl kern.geom.debugflags=16
dd if=/dev/zero of=/dev/ada2 bs=4096 count=1 seek=385724 conv=noerror,sync

Still received the same error:
ada0

ada1

3. Then I ran a scrub on my volume
4. Still receiving same error and another message

Questions:
1. Did I run the dd command incorrectly and to the wrong sectors?
2. The auto scrubs are setup like this: (do i need to run more scrubs?)

3. What did I do incorrectly to receive all these error messages? will it affect data integrity??
4. What else can I try to fix these problems?

Thank you!

 

[tvsjr]

Please post the output of "zpool status". It's a near certainty that ada0 and ada1 are both failing, meaning you're operating right now with zero redundancy. I would suggest a full backup of the pool immediately.

 

[Member]

Please post the output of "zpool status". It's a near certainty that ada0 and ada1 are both failing, meaning you're operating right now with zero redundancy. I would suggest a full backup of the pool immediately.

Here's the zpool status: (just unplugged freenas box yesterday and rebooted today)

 

[Jailer]

Both are failing smart test and need to be replaced asap. Ada1 looks to be the worst of the 2 so I would replace that disk first.

 

[Member]

Both are failing smart test and need to be replaced asap. Ada1 looks to be the worst of the 2 so I would replace that disk first.

Thanks. How do I proceed? Find serial number of drives to identify drives then pull out of chassis?
I think I can RMA the WD reds. But once I get new drives, what do I need to configure?

Do you know why both drives are failing? I have been running scrubs and the system is only 1.5 years old...

 

[SweetAndLow]

You don't fix this. You replace the drives. Rma them if they are still under warranty.

 

[Member]

You don't fix this. You replace the drives. Rma them if they are still under warranty.

Ok, WD REDS have 3 years warranty in North America. I have only had this system for 1.5 years, do you know what went wrong?

 

[SweetAndLow]

Ok, WD REDS have 3 years warranty in North America. I have only had this system for 1.5 years, do you know what went wrong?

there is 1 thing i know to be 100% correct and that is hdd's fail. So nothing went wrong your drives are just failing. Run a smart test to be sure. Pending sectors are not terrible but it's a sign of things to cone.

 

[Member]

there is 1 thing i know to be 100% correct and that is hdd's fail. So nothing went wrong your drives are just failing. Run a smart test to be sure. Pending sectors are not terrible but it's a sign of things to cone.

I'll just RMA them, maybe just got a bad batch of drives...
Are there any freenas guides on how prepare and rebuild my pool when I get new drives?

 

[Jailer]

I'll just RMA them, maybe just got a bad batch of drives...
Are there any freenas guides on how prepare and rebuild my pool when I get new drives?

Yes, the manual linked at the top of this forum and in your FreeNAS gui.

 

[Member]

Yes, the manual linked at the top of this forum and in your FreeNAS gui.

Currently I'm running Raidz2 , but going to RMA two drives, can I have my other two working drives be the main drives and the new RMA drives become parity drives?

How do I ensure my data is on my two working drives before sending out the failed drives?
http://doc.freenas.org/11/storage.html#replacing-a-failed-drive

 

[Ericloewe]

The Resources section is your friend:
https://forums.freenas.org/index.php?resources/
https://forums.freenas.org/index.php?resources/replacing-a-failed-failing-disk.75/

1. Did I run the dd command incorrectly and to the wrong sectors?

Yes, since there was no good reason to even run it in the first place.

Seriously, trying to overwrite a bad sector is an exercise in futility.
If it really is bad, the drive will remap it elsewhere and you're wasting your time.
If it was a random error, the drive will happily play along and you're wasting your time.
If it's wonky, the drive may or may not remap it, maybe even back and forth - you don't know because you didn't write the firmware and so you're wasting your time.

In other words, use the interfaces provided for you! That means assuming that every LBA is good until there is proof to the contrary. Once there is, ZFS will solve it for you at the data level and the drive will "solve" it for you at its level. Once it becomes clear that the drive cannot fulfill its promise of safeguarding all the LBAs and their contents, replace it.

 

[tvsjr]

That's not how RAIDZ2 works. RAID-Z distributes parity across multiple drives, so you can't "choose" which drive is parity. Further, part of the file system's guarantee is that every byte of data will survive if you remove up to two drives.

 

[Ericloewe]

Currently I'm running Raidz2 , but going to RMA two drives, can I have my other two working drives be the main drives and the new RMA drives become parity drives?

How do I ensure my data is on my two working drives before sending out the failed drives?

That is not how RAIDZ works. That's not how any self-respecting RAID solution works. Yes, that means unraid is a piece of crap, it's not an accident of language. I'll take HW RAID over unraid any day of the week, which says a lot.

The illustration below, helpfully supplied by Matt Ahrens in his blog at Delphix, shows how data is distributed among disks in a RAIDZ vdev. P is a parity block, D is a data block and X is a padding block to meet certain requirements that are irrelevant to the current discussion. All disks get both parity and data.

 

[Member]

That is not how RAIDZ works. That's not how any self-respecting RAID solution works. Yes, that means unraid is a piece of crap, it's not an accident of language. I'll take HW RAID over unraid any day of the week, which says a lot.

The illustration below, helpfully supplied by Matt Ahrens in his blog at Delphix, shows how data is distributed among disks in a RAIDZ vdev. P is a parity block, D is a data block and X is a padding block to meet certain requirements that are irrelevant to the current discussion.All disks get both parity and data.

View attachment 22407

Thanks for correcting me!
"All disks get both parity and data." That's key feature.
So basically I just need to follow the guide TWICE http://doc.freenas.org/11/storage.html#replacing-a-failed-drive
And my data should be intact.

 

[Redcoat]

Thanks for correcting me!
"All disks get both parity and data." That's key feature.
So basically I just need to follow the guide TWICE http://doc.freenas.org/11/storage.html#replacing-a-failed-drive
And my data should be intact.

If I were you, I would follow https://forums.freenas.org/index.php?resources/replacing-a-failed-failing-disk.75/
which was written specially to help us avoid the most common pitfalls of the drive replacement exercise. And do note its "burned-in" recommendation.

 

[Member]

If I were you, I would follow https://forums.freenas.org/index.php?resources/replacing-a-failed-failing-disk.75/
which was written specially to help us avoid the most common pitfalls of the drive replacement exercise. And do note its "burned-in" recommendation.

Thank you for your help and for the useful link!
So I can just send both of the failing drives in for RMA?
Since two of my drives are failing, should I replace drive 1 first then wait for resilver and burn-in then replace the 2nd drive?

I also usually perform a burn-in test on another system before installing into the Freenas Box.

thx again

 

[Redcoat]

@Ericloewe provided the link you first. I just picked it out as perhaps preferable of the two as you had identified the other.

I would RMA the two drives, burn in both of the replacements (run badblocks on both simultaneously using tmux), install one and let resilver, then install the other. I do not know if two drives can be replaced at once.

 

[danb35]

So I can just send both of the failing drives in for RMA?

No, pay the $10 or so each for advance RMA service. Get the replacements, burn them in, run the replacement procedure, then send back the bad disks. And since you'd need to pay for the return shipping anyway, it really hardly costs anything.

But this question seems to come up at least weekly, and the posters never seem to show any awareness that the issue's ever been discussed before, nor any degree of understanding about what the message is saying. It really seems pretty obvious--the disk has bad sectors. That's a bad thing. Replace the disk. Clearly it isn't that obvious, but I can't see where the disconnect is.

 

[Chris Moore]

@Ericloewe provided the link you first. I just picked it out as perhaps preferable of the two as you had identified the other.

I would RMA the two drives, burn in both of the replacements (run badblocks on both simultaneously using tmux), install one and let resilver, then install the other. I do not know if two drives can be replaced at once.

You can replace two drives at once, but I suggest that you don't do it. Even though the drives are failing, just replace one at a time.

Sent from my SAMSUNG-SGH-I537 using Tapatalk