ZFS Pool Degraded

[Erates]

Hey,

I know there are multiple threads regarding this issue. But I can't find one that handles my problem.

Since a couple of days, I'm receiving an email with this data:

Code:

The volume raid0 (ZFS) state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

When I ran "zpool status -vx", I got an error on 2 files. The file's weren't important so I thought I would delete them, and the error would go away. But the next day, I received an email saying that the raid has become DEGRADED.
I immediately turned off my NAS, because I didn't know what to do. Today, I booted it up again, because I want to solve the problem. Once it was booted, I received an email:

Code:

The volume raid0 (ZFS) state is ONLINE: One or more devices has experienced an error resulting in data corruption. Applications may be affected.

Now when I did "zpool status -vx", I got this result:

pool: raid0
state: ONLINE
status: One or more devices has experienced an error resulting in data
corruption. Applications may be affected.
action: Restore the file in question if possible. Otherwise restore the
entire pool from backup.
see: http://illumos.org/msg/ZFS-8000-8A
scan: scrub repaired 0 in 17h15m with 18 errors on Wed Jan 20 02:19:05 2016
config:

NAME STATE READ WRITE CKSUM
raid0 ONLINE 0 0 0
mirror-0 ONLINE 0 0 0
gptid/bf48e1a6-6439-11e5-8d97-001bfc5a43df ONLINE 0 0 0
gptid/bfa52aa3-6439-11e5-8d97-001bfc5a43df ONLINE 0 0 0

errors: Permanent errors have been detected in the following files:

/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/com.apple.TimeMachine.Results.plist
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/c8d
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/com.apple.TimeMachine.Results.plist/<xattrdir>/org.netatalk.Metadat
a
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e59
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e59/<xattrdir>/org.netatalk.Metadata
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e3e
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e3e/<xattrdir>/org.netatalk.Metadata
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e3f
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e3f/<xattrdir>/org.netatalk.Metadata
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e40
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e40/<xattrdir>/org.netatalk.Metadata
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/e41
/mnt/raid0/sharedir/GoPro/GOPR0005.MP4
/mnt/raid0/sharedir/GoPro/GOPR0006.MP4
/mnt/raid0/sharedir/GoPro/GOPR0008.MP4
/mnt/raid0/timemachine/MacBook Pro van Rens.sparsebundle/bands/623

So it went from 2 corrupt files, to 16. And because my GoPro video's are on there, I don't want to delete them.

It's a mirror on 2 WD Red Pro's 4TB. Full spec is below:

FreeNAS: FreeNAS-9.3-STABLE-201601181840
Mobo: Asus P5k
CPU: Intel Core2Quad Q9500
Ram: 6Gb, forgot what brand, probably no ECC
Hard Drive: 2x WD Red Pro 4TB

So, what can I do?

Erates

 

[m0nkey_]

I'm going to point out the obvious here,you're not meeting minimum requirements. However, that being said, you likely have some bad hardware. Did you take the time to set-up regular SMART tests on the drives? Have you run a memtest to see if your memory is failing?

Please get a debug file and post it here, the button should be under System > Advanced > Save debug.

 

[jde]

My guess is that having less than the minimum memory or lack of ECC memory ate your data. Can you post the output of smartctl -a for each of the drives. Also, you might want to run memtest to see if your memory is flaking out. Seeing as how your memory may be flakey and non-ECC I would not recommend running a scrub as it may trash your whole pool. https://forums.freenas.org/index.php?threads/ecc-vs-non-ecc-ram-and-zfs.15449/
Wait to hear from the forum's heavy hitters before taking any unreversable action.

 

[Erates]

Thanks for the quick response!

I was running a regular scrub, so I turned it off immediately :)
I also run a short SMART check on each disk every night, and a long SMART check every week.

Below is the debug zip, the "smartctl -a", I'll do when I get home, because the web interface's shell isn't what it should be and my SSH port isn't forwarded.
http://www.filedropper.com/debug-elara-20160122143441

 

[Bidule0hm]

I'll do when I get home, because the web interface's shell isn't what it should be and my SSH port isn't forwarded.

If I read that correctly I can guess you've exposed the web GUI to the internet? if yes then it's a very very very bad idea...

 

[Mirfster]

If I read that correctly I can guess you've exposed the web GUI to the internet? if yes then it's a very very very bad idea...

If that is the case, then it would just be another item on the lists of "no-nos" done here.

 

[Erates]

Well yea, it's a home NAS, nothing special. And I know I should not expose the GUI, but I do it this way until my VPN server is up.

Here is the smartctl output:
/dev/ada0

Code:

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p31 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red Pro
Device Model:     WDC WD4001FFSX-68JNUN0
Serial Number:    ***************
LU WWN Device Id: 5 0014ee 003eb6736
Firmware Version: 81.00A81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Jan 24 11:15:10 2016 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (40200) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 435) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   153   150   021    Pre-fail  Always       -       11341
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       38
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       5689
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       28
 16 Unknown_Attribute       0x0022   020   180   000    Old_age   Always       -       351670976061
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       438
194 Temperature_Celsius     0x0022   120   099   000    Old_age   Always       -       32
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      5688         -
# 2  Short offline       Completed without error       00%      5675         -
# 3  Extended offline    Interrupted (host reset)      10%      5655         -
# 4  Short offline       Completed without error       00%      5639         -
# 5  Short offline       Completed without error       00%      5615         -
# 6  Short offline       Completed without error       00%      5591         -
# 7  Short offline       Completed without error       00%      5567         -
# 8  Short offline       Completed without error       00%      5543         -
# 9  Short offline       Completed without error       00%      5519         -
#10  Short offline       Completed without error       00%      5495         -
#11  Extended offline    Completed without error       00%      5490         -
#12  Short offline       Completed without error       00%      5471         -
#13  Short offline       Completed without error       00%      5447         -
#14  Short offline       Completed without error       00%      5423         -
#15  Short offline       Completed without error       00%      5399         -
#16  Short offline       Completed without error       00%      5375         -
#17  Short offline       Completed without error       00%      5351         -
#18  Short offline       Completed without error       00%      5327         -
#19  Extended offline    Completed without error       00%      5322         -
#20  Short offline       Completed without error       00%      5303         -
#21  Short offline       Completed without error       00%      5279         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

/dev/ada1

Code:

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p31 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red Pro
Device Model:     WDC WD4001FFSX-68JNUN0
Serial Number:    ***************
LU WWN Device Id: 5 0014ee 0594d8b5a
Firmware Version: 81.00A81
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 3.0 Gb/s)
Local Time is:    Sun Jan 24 11:18:55 2016 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00) Offline data collection activity
                                        was never started.
                                        Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (40560) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 439) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x70bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   151   148   021    Pre-fail  Always       -       11416
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       39
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   093   093   000    Old_age   Always       -       5752
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       29
 16 Unknown_Attribute       0x0022   021   179   000    Old_age   Always       -       377232727863
183 Runtime_Bad_Block       0x0032   100   100   000    Old_age   Always       -       0
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       8
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       434
194 Temperature_Celsius     0x0022   118   098   000    Old_age   Always       -       34
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%      5751         -
# 2  Short offline       Completed without error       00%      5738         -
# 3  Extended offline    Interrupted (host reset)      50%      5718         -
# 4  Short offline       Completed without error       00%      5701         -
# 5  Short offline       Completed without error       00%      5678         -
# 6  Short offline       Completed without error       00%      5654         -
# 7  Short offline       Completed without error       00%      5630         -
# 8  Short offline       Completed without error       00%      5606         -
# 9  Short offline       Completed without error       00%      5582         -
#10  Short offline       Completed without error       00%      5558         -
#11  Extended offline    Completed without error       00%      5556         -
#12  Short offline       Completed without error       00%      5534         -
#13  Short offline       Completed without error       00%      5510         -
#14  Short offline       Completed without error       00%      5486         -
#15  Short offline       Completed without error       00%      5462         -
#16  Short offline       Completed without error       00%      5438         -
#17  Short offline       Completed without error       00%      5414         -
#18  Short offline       Completed without error       00%      5390         -
#19  Extended offline    Completed without error       00%      5389         -
#20  Short offline       Completed without error       00%      5366         -
#21  Short offline       Completed without error       00%      5342         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.