How to fix 1 unreadable sector

[ovizii]

What I have are 4 x 1TB HDs setup as Raidz2.
Getting this error: smartd[95211]: Device: /dev/ada0, 1 Currently unreadable (pending) sectors

Here is some info you can work with (I've bolded parts I think are relevant) :

Code:

[root@boxy] ~# smartctl -a /dev/ada0
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    SAMSUNG SpinPoint F3
Device Model:    SAMSUNG HD103SJ
Serial Number:    S246JD2Z919929
LU WWN Device Id: 5 0024e9 00406fa02
Firmware Version: 1AJ10001
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Thu Jul  3 11:39:15 2014 SAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 121)    The previous self-test completed having
                    the read element of the test failed.
Total time to complete Offline
data collection:        ( 9240) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (  2) minutes.
Extended self-test routine
recommended polling time:      ( 154) minutes.
SCT capabilities:            (0x003f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x002f  100  100  051    Pre-fail  Always      -      1882
  2 Throughput_Performance  0x0026  052  052  000    Old_age  Always      -      9004
  3 Spin_Up_Time            0x0023  071  069  025    Pre-fail  Always      -      8836
  4 Start_Stop_Count        0x0032  099  099  000    Old_age  Always      -      1798
  5 Reallocated_Sector_Ct  0x0033  252  252  010    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x002e  252  252  051    Old_age  Always      -      0
  8 Seek_Time_Performance  0x0024  252  252  015    Old_age  Offline      -      0
  9 Power_On_Hours          0x0032  100  100  000    Old_age  Always      -      3078
10 Spin_Retry_Count        0x0032  252  252  051    Old_age  Always      -      0
11 Calibration_Retry_Count 0x0032  252  252  000    Old_age  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      626
191 G-Sense_Error_Rate      0x0022  100  100  000    Old_age  Always      -      4
192 Power-Off_Retract_Count 0x0022  252  252  000    Old_age  Always      -      0
194 Temperature_Celsius    0x0002  064  053  000    Old_age  Always      -      25 (Min/Max 14/47)
195 Hardware_ECC_Recovered  0x003a  100  100  000    Old_age  Always      -      0
196 Reallocated_Event_Count 0x0032  252  252  000    Old_age  Always      -      0
197 Current_Pending_Sector  0x0032  100  100  000    Old_age  Always      -      1
198 Offline_Uncorrectable  0x0030  252  252  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x0036  200  200  000    Old_age  Always      -      0
200 Multi_Zone_Error_Rate  0x002a  100  100  000    Old_age  Always      -      1706
223 Load_Retry_Count        0x0032  252  252  000    Old_age  Always      -      0
225 Load_Cycle_Count        0x0032  100  100  000    Old_age  Always      -      2071
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline      Completed: read failure      90%      3078        1953523077
# 2  Extended offline    Completed: read failure      10%      3069        1953523077
# 3  Short offline      Completed without error      00%      3066        -
# 4  Short offline      Completed: read failure      70%      3043        1953523077
# 5  Extended offline    Completed without error      00%      3023        -
# 6  Short offline      Completed without error      00%      3019        -
 
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Completed_read_failure [90% left] (0-65535)
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

gpart show
=>        34  1953525101  ada0  GPT  (931G)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  1949330703    2  freebsd-zfs  (929G)

Code:

iskinfo -v ada0
ada0
    512            # sectorsize
    1000204886016    # mediasize in bytes (931G)
    1953525168      # mediasize in sectors
    0              # stripesize
    0              # stripeoffset
    1938021        # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    S246JD2Z919929    # Disk ident.

Code:

disk -t /dev/ada0
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=1938021 heads=16 sectors/track=63 (1008 blks/cyl)
 
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=1938021 heads=16 sectors/track=63 (1008 blks/cyl)
 
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 1953525167 (953869 Meg), flag 80 (active)
    beg: cyl 0/ head 0/ sector 2;
    end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

Now from what I've read seeing it is only 1 single sector I'm supposed to Write to this sector to get the disk to relocate it, right? I'm a bit lost how to do this, any help please?

 

[Ericloewe]

What I have are 4 x 1TB HDs setup as Raidz2.
Getting this error: smartd[95211]: Device: /dev/ada0, 1 Currently unreadable (pending) sectors

Here is some info you can work with (I've bolded parts I think are relevant) :

Code:

[root@boxy] ~# smartctl -a /dev/ada0
smartctl 6.2 2013-07-26 r3841 [FreeBSD 9.2-RELEASE-p4 amd64] (local build)
Copyright (C) 2002-13, Bruce Allen, Christian Franke, www.smartmontools.org
 
=== START OF INFORMATION SECTION ===
Model Family:    SAMSUNG SpinPoint F3
Device Model:    SAMSUNG HD103SJ
Serial Number:    S246JD2Z919929
LU WWN Device Id: 5 0024e9 00406fa02
Firmware Version: 1AJ10001
User Capacity:    1,000,204,886,016 bytes [1.00 TB]
Sector Size:      512 bytes logical/physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:  ATA8-ACS T13/1699-D revision 6
SATA Version is:  SATA 2.6, 3.0 Gb/s
Local Time is:    Thu Jul  3 11:39:15 2014 SAST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled
 
=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED
 
General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      ( 121)    The previous self-test completed having
                    the read element of the test failed.
Total time to complete Offline
data collection:        ( 9240) seconds.
Offline data collection
capabilities:              (0x5b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    No Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine
recommended polling time:      (  2) minutes.
Extended self-test routine
recommended polling time:      ( 154) minutes.
SCT capabilities:            (0x003f)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.
 
SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG    VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate    0x002f  100  100  051    Pre-fail  Always      -      1882
  2 Throughput_Performance  0x0026  052  052  000    Old_age  Always      -      9004
  3 Spin_Up_Time            0x0023  071  069  025    Pre-fail  Always      -      8836
  4 Start_Stop_Count        0x0032  099  099  000    Old_age  Always      -      1798
  5 Reallocated_Sector_Ct  0x0033  252  252  010    Pre-fail  Always      -      0
  7 Seek_Error_Rate        0x002e  252  252  051    Old_age  Always      -      0
  8 Seek_Time_Performance  0x0024  252  252  015    Old_age  Offline      -      0
  9 Power_On_Hours          0x0032  100  100  000    Old_age  Always      -      3078
10 Spin_Retry_Count        0x0032  252  252  051    Old_age  Always      -      0
11 Calibration_Retry_Count 0x0032  252  252  000    Old_age  Always      -      0
12 Power_Cycle_Count      0x0032  100  100  000    Old_age  Always      -      626
191 G-Sense_Error_Rate      0x0022  100  100  000    Old_age  Always      -      4
192 Power-Off_Retract_Count 0x0022  252  252  000    Old_age  Always      -      0
194 Temperature_Celsius    0x0002  064  053  000    Old_age  Always      -      25 (Min/Max 14/47)
195 Hardware_ECC_Recovered  0x003a  100  100  000    Old_age  Always      -      0
196 Reallocated_Event_Count 0x0032  252  252  000    Old_age  Always      -      0
197 Current_Pending_Sector  0x0032  100  100  000    Old_age  Always      -      1
198 Offline_Uncorrectable  0x0030  252  252  000    Old_age  Offline      -      0
199 UDMA_CRC_Error_Count    0x0036  200  200  000    Old_age  Always      -      0
200 Multi_Zone_Error_Rate  0x002a  100  100  000    Old_age  Always      -      1706
223 Load_Retry_Count        0x0032  252  252  000    Old_age  Always      -      0
225 Load_Cycle_Count        0x0032  100  100  000    Old_age  Always      -      2071
 
SMART Error Log Version: 1
No Errors Logged
 
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline      Completed: read failure      90%      3078        1953523077
# 2  Extended offline    Completed: read failure      10%      3069        1953523077
# 3  Short offline      Completed without error      00%      3066        -
# 4  Short offline      Completed: read failure      70%      3043        1953523077
# 5  Extended offline    Completed without error      00%      3023        -
# 6  Short offline      Completed without error      00%      3019        -
 
SMART Selective self-test log data structure revision number 0
Note: revision number not 1 implies that no selective self-test has ever been run
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Completed_read_failure [90% left] (0-65535)
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

Code:

gpart show
=>        34  1953525101  ada0  GPT  (931G)
          34          94        - free -  (47k)
        128    4194304    1  freebsd-swap  (2.0G)
    4194432  1949330703    2  freebsd-zfs  (929G)

Code:

iskinfo -v ada0
ada0
    512            # sectorsize
    1000204886016    # mediasize in bytes (931G)
    1953525168      # mediasize in sectors
    0              # stripesize
    0              # stripeoffset
    1938021        # Cylinders according to firmware.
    16              # Heads according to firmware.
    63              # Sectors according to firmware.
    S246JD2Z919929    # Disk ident.

Code:

disk -t /dev/ada0
******* Working on device /dev/ada0 *******
parameters extracted from in-core disklabel are:
cylinders=1938021 heads=16 sectors/track=63 (1008 blks/cyl)
 
Figures below won't work with BIOS for partitions not in cyl 1
parameters to be used for BIOS calculations are:
cylinders=1938021 heads=16 sectors/track=63 (1008 blks/cyl)
 
Media sector size is 512
Warning: BIOS sector numbering starts with sector 1
Information from DOS bootblock is:
The data for partition 1 is:
sysid 238 (0xee),(EFI GPT)
    start 1, size 1953525167 (953869 Meg), flag 80 (active)
    beg: cyl 0/ head 0/ sector 2;
    end: cyl 1023/ head 255/ sector 63
The data for partition 2 is:
<UNUSED>
The data for partition 3 is:
<UNUSED>
The data for partition 4 is:
<UNUSED>

Now from what I've read seeing it is only 1 single sector I'm supposed to Write to this sector to get the disk to relocate it, right? I'm a bit lost how to do this, any help please?

Let's start from the beginning:

Do you have scrubs and SMART tests scheduled? If so just replace the drive (it's failing!) and you won't have to worry about anything else. Follow the manual.

DO NOT mess with gpart or any other utility.

If you don't have scrubs and SMART tests scheduled, replace the drive and immediately schedule them (follow the manual).

 

[danb35]

Unfortunately, the bold doesn't work inside the code tags, but it's nonetheless clear enough. I'm seeing that the drive is failing SMART tests in addition to having the bad sector. Looks like it's time to RMA the drive.

 

[Ericloewe]

Also, the drive said it's seen 47ºC - that's a recipe for bad drives. 40ºC is the maximum ceiling we recommend around here.

 

[ovizii]

OK lets start at the beginning:

I've had these drives for quite a while but only put them into this NAS about 10 days ago.
I do have smart tests and scrubs scheduled.
The last scrub said:
Scrub
Status: Completed
Errors: 0 Repaired: 0

Everything looks healthy on that side.

After reading through the forums, I thought generally, 1 unreadable sector isn't cause for concern and was looking for instructions on how to write right onto this sore spot with dd in order for the drive to re-allocate the bad sector? Of course if there are more popping up its time to replace it.

Having said that, I'm not all that familiar with smart results, would you mind pointing out which other one are cause for alarm?

 

[Ericloewe]

OK lets start at the beginning:

I've had these drives for quite a while but only put them into this NAS about 10 days ago.
I do have smart tests and scrubs scheduled.
The last scrub said:
Scrub
Status: Completed
Errors: 0 Repaired: 0

Everything looks healthy on that side.

After reading through the forums, I thought generally, 1 unreadable sector isn't cause for concern and was looking for instructions on how to write right onto this sore spot with dd in order for the drive to re-allocate the bad sector? Of course if there are more popping up its time to replace it.

Having said that, I'm not all that familiar with smart results, would you mind pointing out which other one are cause for alarm?

You don't. Once the scrub has been run, ZFS will have corrected the error.

As for the SMART data: The multizone error rate is non-0 and so is the read error rate (which should be 0 on Samsung F3 drives and WD drives).

 

[ovizii]

Sorry, I'm a bit confused. Can you elaborate?

What do you mean by "You don't"?
When you say ZFS will have corrected the error, you're not referring to the smart test error, right?

 

[Ericloewe]

What I mean is you don't go around trying to overwrite bad sectors to get them relocated. You run a scrub and whatever damage has been done is corrected through parity.

 

[solarisguy]

Yes, as Ericloewe has suggested, ZFS scrub should be tried first.

The disk seems to be healthy otherwise. (Yes, it had reached +47°C, but it is at a healthy +25°C for now.)

There would be a problem if the failed sector was in the swap partition.

 

[danb35]

The other concern I see with your SMART data starts on line 84--your drive has failed three of the last six self-tests. If it's under warranty, RMA the thing.

 

[solarisguy]

@danb35, the tests failed on the same sector. I do not know enough about Samsung hard drives to know why it did not reallocate the failed sector yet.

 

[danb35]

True, but those repeated failures tell me (perhaps incorrectly) that the sector can't simply be reallocated.

 

[solarisguy]

True, but those repeated failures tell me (perhaps incorrectly) that the sector can't simply be reallocated.

I have no idea

 

[cyberjock]

I'd say that it is possible to reallocate the bad sector. But considering the fact its failing SMART tests just RMA the thing.

Car analogy: Would you change the oil on a car that was just in an accident and totalled? I wouldn't...

 

[ovizii]

Look guys, I'm new to this thing. From what I understand, the smart tests don't write and neither does a scrub from what I read in the freeNAS documentation => A ZFS scrub only checks used disk space. To check unused disk space, schedule a S.M.A.R.T. Test Type of Long Self-Test to run once or twice a month. (My NAS is new, only holding "test data" until I'm comfortable storing stuff on it.

So I found this: http://smartmontools.sourceforge.net/badblockhowto.html and am trying to figure out how to write to this spoecific sector.
I'm just a bit stumped as my smartd results say:

LBA_of_first_error
1953523077

BUT:
gpart show
=> 34 1953525101 ada0 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1949330703 2 freebsd-zfs (929G)

meaning that 1953523077 > 1949330703 so my bad sector is outside my partition?

 

[solarisguy]

You misread the output.
4194432 is the start of the partition, you had guessed that right.
1949330703 is the size of the partition, and not its end point.

 

[ovizii]

Ah, awesome, thanks!

So here is my progress so far but I still need some help, I'M cobbling this together as I go since nobody answered my original question how to overwrite the bad sector:


LBA of first bad sector: 1953523077

gpart show
=> 34 1953525101 ada0 GPT (931G)
34 94 - free - (47k)
128 4194304 1 freebsd-swap (2.0G)
4194432 1949330703 2 freebsd-zfs (929G)


so the sector is inside /dev/ada0p2

ada0p2 starts at 4194432 is 1949330703 long

so => the problem 1953523077 - offset 4194432 => the problem occurs at 1949328645 sectors into the partition ada0p2

I now need some help with the rest of the calculation below. Following this tutorial: http://smartmontools.sourceforge.net/badblockhowto.html
according to the tuts I found online I now need to find out the block size but all the examples I can find are for extfs:

Second Step: we need to find the block size of the file system (normally 4096 bytes for ext2):
root]# tune2fs -l /dev/hda3 | grep Block
Block count: 29119820
Block size: 4096

In this case the block size is 4096 bytes. Third Step: we need to determine which File System Block contains this LBA. The formula is:
b = (int)((L-S)*512/B)
where:
b = File System block number
B = File system block size in bytes
L = LBA of bad sector
S = Starting sector of partition as shown by fdisk -lu
and (int) denotes the integer part.

In our example, L=23421417, S=5269320, and B=4096. Hence the 'problem' LBA is in block number
b = (int)18152097*512/4096 = (int)2269012.125
so b=2269012.


Note: the fractional part of 0.125 indicates that this problem LBA is actually the second of the eight sectors that make up this file system block.
Fourth Step: we use debugfs to locate the inode stored in this block, and the file that contains that inode:

 

[solarisguy]

Have your scrub completed, yet?

ZFS does not have, known to me, tools that allow to write to a single sector or block. Given that ZFS assumes integrity of everything, I doubt that they would fit the ZFS framework.

The standard Unix way to write to a sector is to use dd

FreeNAS/ZFS would not allow you to do that on a mounted drive, so take it out to another machine. It can be a Windows system, then you would use Cygwin. It is all easy with dd. Read the entire disk to /dev/null The process would fail on the inaccessible sector (I hope), so you would have the exact sector number as dd sees it. Write only to that sector using dd. Check what S.M.A.R.T. says. Repeat as desired, since it would be a rather quick operation. If you want to run Cygwin on Windows, you only need to install the base Cygwin packages and S.M.A.R.T. (all from cygwin.com).

If the entire disk is read without an error..., I doubt, you can use math to calculate the faulty sector, read it to a file, and try to write back the same values.

P.S.
I was joking, do not read the entire disk. Start reading at a sector a little bit earlier than the fault. As the starting address use a nice address divisible let's say by 8192 or 16384.

 

[joeschmuck]

So I assume you understand that if you write data manually to this one drive, that you will likely destroy some data. Of course that shouldn't be an issue since you have a RAIDZ2 running. If you really want to take this approach I'd take the drive out and fill it with data using dd, top to bottom. I suspect that the reason the SMART tests have failed [which is very concerning BTW] is not due to a single bad sector, but many bad contiguous sectors. Typically when you have a failed sector after the drive has been running for a while (a few hundred hours), the problems are caused my the media deteriorating. Once this starts happening it generally continues to happen. There are some tools out there which will do a full destructive media test which is probably what you want to do to give you confidence about the drive condition. If the drive is under warranty then I'd RMA it.

 

[rs225]

The LBA number listed in SMART is absolute. You do not need to calculate any offset from the partition table.

As others have said, directly manipulating the drive this way is dangerous. That said, if you want to overwrite the first LBA that had a read failure in your SMART test (and which MIGHT be the same one that is pending relocation):

dd if=/dev/zero of=512-bytes-of-zero count=1
ls -l 512-bytes-of-zero (verify it is 512 bytes)
dd if=512-bytes-of-zero of=/dev/ada0 count=1 seek=1953523077

I recommend doing it this way so that you have double certainty that you aren't going to write more than 512-bytes do your drive. A runaway overwrite would be very bad. The count=1 is key to stopping the write. The count is in blocks, with default blocksize being 512-bytes.

I left out bs=512b since that is the presumed default. But you could add bs=512b.

If this was a 4K sector drive, you would need to set bs=4k, divide the LBA number by 8, and drop any fraction. I would think there wouldn't be a fraction, so if there is, something isn't right. Since 4K drives only read 4K sectors, it should be possible for the first bad LBA to be anything other than a sector divisible by 8.

If it doesn't let you write the drive, it means you probably need to export the pool or unmount any partitions first.

edit: If you ever use dd to overwrite part of a file, you must include conv=notrunc otherwise dd will trim off the rest of the file that it doesn't overwrite.