One or more devices has experienced an error resulting in data corruption.

[Jonas LV]

Hi all, Can someone please help me with the following, I am even ready to hire someone to help me with this:

I have assembled my self a small NAS:

• ASRock Z87E-ITX Motherboard (Socket 1150, Z87 Express, DDR3, S-ATA 600, WiFi, Mini ITX, Haswell)
• Intel Core i5 4570T CPU (2.9GHz, 2 Core, 4 Threads, 4MB Cache, LGA1150 Socket)
• Ballistix Tactical BLT2C8G3D1608ET3LX0CEU 16GB kit (8GB x 2) DDR3 PC3-12800 Unbuffered NON-ECC 1.35V
• Seasonic SS-400FL2 F3 - 400w FANLESS '80 Plus Platinum' Modular Power Supply
• Fractal Design Node 304 Case
• 4 x WD Red NAS Hard Drive (3TB)
• 1 x 128GB Samsung SSD
• 1x Noctua NF-A14 PWM Fan (140mm)
• 2x Noctua NF-F12 PWM Fan (120mm)
• FreeNAS 9.3 STABLE 201506042008

Created RAID10 (2x2) pool and started sharing for Windows (CIF)

Transferred number of files to that share and one day after power outage I have got an alert in Web interface saying there is a critical error that some files are corrupted. I have ran the following in the console to see what the state is:

Code:

zpool status -v

  pool: freenas-boot
state: ONLINE
  scan: none requested
config:
        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
               ada4p2    ONLINE       0     0     0

errors: No known data errors

  pool: mynaspool
state: ONLINE
status: One or more devices has experienced an error resulting in data corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: none requested
config:
        NAME                                            STATE     READ WRITE CKSUM
        mynas                                           ONLINE       0     0     0
         mirror-0                                       ONLINE       0     0     0
            gptid/8fced0b9-0edf-11e5-905b-d050991bb0dd  ONLINE       0     0     0
            gptid/902dcd99-0edf-11e5-905b-d050991bb0dd  ONLINE       0     0     0
         mirror-1                                       ONLINE       0     0     0
            gptid/908f7fb9-0edf-11e5-905b-d050991bb0dd  ONLINE       0     0     0
            gptid/90ee212a-0edf-11e5-905b-d050991bb0dd  ONLINE       0     0     0

errors: Permanent errors have been detected in the following files:

/mnt/mynas/myfiles/thefile.rar
the list of files continues...

and this is for another 122 files. They account for 100GB data.
Most of them I would survive without, but I am really keen to get back at least 20 of them.

While trying to access the share through Windows all works fine, all files are displayed but the files in questions can't be touched by any means - no renaming, no copying (I/O error), just moving around in the same share.
I have extracted all the files that are not harmed, now somehow need to get out those 20 important and corrupted.

Can someone please help me with this issue? Thank you in advance...

 

[m0nkey_]

Can you provide more detail on your system specifications. Such as motherboard make/model, NIC, etc. I will point out, and as most people on the forum will also, you're running an i5 which does not support ECC. You may want to perform a memory test to ensure you RAM isn't going bad. Bad RAM will cause data corruption.

 

[Jonas LV]

Here is my current FreeNAS build:

• ASRock Z87E-ITX Motherboard (Socket 1150, Z87 Express, DDR3, S-ATA 600, WiFi, Mini ITX, Haswell)
• Intel Core i5 4570T CPU (2.9GHz, 2 Core, 4 Threads, 4MB Cache, LGA1150 Socket)
• Ballistix Tactical BLT2C8G3D1608ET3LX0CEU 16GB kit (8GB x 2) DDR3 PC3-12800 Unbuffered NON-ECC 1.35V
• Seasonic SS-400FL2 F3 - 400w FANLESS '80 Plus Platinum' Modular Power Supply
• Fractal Design Node 304 Case
• 4 x WD Red NAS Hard Drive (3TB)
• 1 x 128GB Samsung SSD
• 1x Noctua NF-A14 PWM Fan (140mm)
• 2x Noctua NF-F12 PWM Fan (120mm)
• FreeNAS 9.3 STABLE 201506042008

 

[Jonas LV]

I am mainly interested in recovering simply the data - not all just some of corrupted files. If I was to take our one HDD will the system restore from part of RAID that is still healthy? Or do I do it without interrupting from hardware perspective?

 

[Mirfster]

I would venture to say that you are not going to safely get that data recovered for the corrupted files. That is the problem of running non-ECC memory and why it is strongly advised against.

 

[Jonas LV]

How come after doing some research I was given advice to use non-ECC memory...? But this is not the topic - I still would like to recover the data.. .is there someone who could help me to figure out if any of volumes holds it...?

 

[joeschmuck]

I wouldn't expect you to be able to recover the files since the file system has stated the files are corrupt. I think you have done all that you can to copy your data off the NAS. Hopefully if that data was important, you have another copy of it elsewhere.

How come after doing some research I was given advice to use non-ECC memory...?

I highly doubt that. If so, please point out the posting here in these forums and we will make the person walk the plank. ECC RAM is a requirement due to the way the ZFS file system operates, and it's a NAS which should have ECC RAM regardless. Whoever let you believe non-ECC RAM was okay was an idiot, unless your data isn't important at all, then you just run the risk of data corruption.

But in your situation here it appears the error was not having an UPS on your NAS and configured to shut it down during a power outage. This too although not a requirement to get FreeNAS running, is a requirement if you value your system and it's data.

Cheers,
Joe

 

[Robert Trevellyan]

If I was to take our one HDD will the system restore from part of RAID that is still healthy?

Did you try a scrub?
Did you have any periodic snapshots enabled?

 

[Jonas LV]

Did you try a scrub?
Did you have any periodic snapshots enabled?

I can not see any periodic snapshots setup. (I did not know I should do it my self - not out of box)
I still do not understand SCRUB. It says I need to specify existing name, but what name it might be? In the WebUI I can see 1 scrub "name" (the same as my zfs pool. Threshold days 35, Days of Month: Everyday. - What do I do with it? - What do I type in to console?

 

[Jonas LV]

...

I see the reason why they might be an idiot. My self did not came across very smart not reading up my self about it...
What comes to UPS - I have and had one when broke my system. I accidentally switched it off......

 

[danb35]

Well, yes, you should have used ECC memory. If/when you upgrade your hardware, you'll know that, and will hopefully make better choices. For now, running a memory test is a good idea, though it still can't rule out the possibility that there was a temporary error which caused data corruption (even if it reports all clear, there are temporary failure modes for RAM).

If you never set up a scrub, you probably never set up SMART tests for your drives either. To start the scrub from the command line, run 'zpool scrub mynaspool'. Run 'zpool scrub freenas-boot' as well, just to be on the safe side. The scrub on freenas-boot should only take a few seconds, but expect the scrub on your main pool to take several hours. Once that's complete, post the output of zpool status (in code tags, please) again. Then run long SMART tests on all of your drives--'smartctl -t long /dev/ada0', repeat for ada1, ada2, ada3. Expect those also to run for several hours (though you can start them in immediate succession--you don't have to wait for ada0 to finish before starting ada1). When those are complete, post the output of 'smartctl -a' for each of your drives ('smartctl -a /dev/ada0', etc.), again in code tags to preserve the formatting.

Next, what version of FreeNAS are you actually running? You say you're running 9.1, but that can't be true, as you have a ZFS boot pool. What's the actual, exact version you're running?

Finally, do you leave this machine running 24/7? Because there should be a scrub scheduled every 6 weeks by default, on both the main and the boot pool, but it doesn't look like either of those has ever run. Or has it been running less than 6 weeks?

 

[Robert Trevellyan]

You can start a scrub from the GUI too. Select a pool in the Storage tab and three buttons will appear at the bottom of the page. One looks like a broom. Click it to start a scrub of the selected pool.

I still do not understand SCRUB.

Then you have a lot of homework to do. There's lots of great information in the form of pinned or "sticky" posts in these forums. The Wikipedia page on ZFS is also an excellent starting point.

 

[Jonas LV]

...

Thank you danb35 - you are being extremely helpful. I have got new messages...:

I ran 'zpool scrub mynaspool' and 'zpool scrub freenas-boot'. It ran very quickly both of them.
Then repeated zpool status -v and that is what I have got:

Code:

pool: freenas-boot                                                                                                         
state: ONLINE                                                                                                               
  scan: scrub repaired 0 in 0h0m with 0 errors on Sun Mar 27 18:26:56 2016                                                   
config:                                                                 
     NAME        STATE     READ WRITE CKSUM
freenas-boot  ONLINE       0     0     0                       
     ada4p2    ONLINE       0     0     0                         
errors: No known data errors                                             
                                                                                                                            
  pool: mynaspool                                                       
state: DEGRADED                                                         
status: One or more devices has experienced an error resulting in data corruption. Applications may be affected.                                       
action: Restore the file in question if possible.  Otherwise restore the entire pool from backup.                                                     
   see: http://illumos.org/msg/ZFS-8000-8A                                                                                   
  scan: scrub repaired 0 in 0h7m with 141 errors on Sun Mar 27 18:34:02 2016                                            
config:                                                                                                                       
        NAME                                            STATE     READ WRITE CKSUM                                              
        kuparas                                         DEGRADED     0     0   141                                              
          mirror-0                                      DEGRADED     0     0   132                                              
            gptid/8fced0b9-0edf-11e5-905b-d050991bb0dd  DEGRADED     0     0   132  too many errors                            
            gptid/902dcd99-0edf-11e5-905b-d050991bb0dd  DEGRADED     0     0   132  too many errors                            
          mirror-1                                      DEGRADED     0     0   150                                              
            gptid/908f7fb9-0edf-11e5-905b-d050991bb0dd  DEGRADED     0     0   150  too many errors                            
            gptid/90ee212a-0edf-11e5-905b-d050991bb0dd  DEGRADED     0     0   150  too many errors                            


errors: Permanent errors have been detected in the following files:                                                         

/mnt/mynas/myfiles/thefile.rar
the list of files continues...
                                                                                                              

Also sorry for my mistake - It's FreeNAS 9.3 I am running - corrected in previous posts as well.
So far (before errors) I was turning this machine on just to add some backups on and sync few folders. Opposite of 24/7. Max 10h a week.

SMART tests are started, will update the post as soon as they have finished.

Any thoughts on new zpool status? - Should I still have hope to restore those few files?

 

[SweetAndLow]

Like already started there is about 0% chance you will get those files back. Your pool is basically dead probably because of hardware failure.

 

[devnullius]

Sad story to read :/ Above story can only happen if more than 1 disc goes bad, right? If only 1 disk started failing, all his files should be recoverable, right? I feel your pain, Jonas. Good luck :/

Devnullius

 

[danb35]

Above story can only happen if more than 1 disc goes bad, right?

I don't think it's disk failure, as such, that's doing this (mainly because it doesn't seem likely that all four disks would die at the same time), but the SMART results will help figure that out. (Very) bad RAM could certainly have scrambled the data. Other hardware problems--bad power supply, for example--could cause widespread problems like this.

 

[danb35]

So far (before errors) I was turning this machine on just to add some backups on and sync few folders. Opposite of 24/7. Max 10h a week.

The problem with this is that system maintenance tasks (like scrubs and SMART tests) are scheduled at particular times. If the system isn't on at that time, they don't happen. The SMART tests are something that you'd schedule, so if you know that you run the system on Tuesday evenings, for example, you can schedule them for that time. The scrubs are automatically scheduled when a pool is created, but you can change that schedule to a time when you know the system will be up, if it is up on a set schedule. But in both cases, you'd need to manually work on that schedule. Other tasks aren't adjustable through the GUI, though, which is one reason it's recommended to run FreeNAS 24/7.

Any thoughts on new zpool status? - Should I still have hope to restore those few files?

Very unlikely, I'm afraid.

 

[Jonas LV]

S.M.A.R.T. Results:

ada0:

Code:

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)                                                  
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                
                                                                                                                             
=== START OF INFORMATION SECTION ===                                                                                          
Model Family:     Western Digital Red (AF)                                                                                    
Device Model:     WDC WD30EFRX-68EUZN0                                                                                        
Serial Number:    WD-WCC4N4EZ78UE                                                                                            
LU WWN Device Id: 5 0014ee 20ba9f44a                                                                                          
Firmware Version: 82.00A82                                                                                                    
User Capacity:    3,000,592,982,016 bytes [3.00 TB]                                                                          
Sector Sizes:     512 bytes logical, 4096 bytes physical                                                                      
Rotation Rate:    5400 rpm                                                                                                    
Device is:        In smartctl database [for details use: -P show]                                                            
ATA Version is:   ACS-2 (minor revision not indicated)                                                                        
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)                                                                      
Local Time is:    Mon Mar 28 12:47:41 2016 BST                                                                                
SMART support is: Available - device has SMART capability.                                                                    
SMART support is: Enabled                                                                                                    
                                                                                                                             
=== START OF READ SMART DATA SECTION ===                                                                                      
SMART overall-health self-assessment test result: PASSED                                                                      
                                                                                                                             
General SMART Values:                                                                                                        
Offline data collection status:  (0x00) Offline data collection activity was never started.                                                                                                           Auto Offline Data Collection: Disabled.                                              
Self-test execution status:      (   0) The previous self-test routine completed without error or no self-test has ever been run.                                                                
Total time to complete Offline                                                                                                
data collection:                (38940) seconds.                                                                              
Offline data collection                                                                                                      
capabilities:                    (0x7b) SMART execute Offline immediate.                                                      
                                        Auto Offline data collection on/off support.                                          
                                        Suspend Offline collection upon new command.                                                                    
                                        Offline surface scan supported.                                                      
                                        Self-test supported.                                                                  
                                        Conveyance Self-test supported.                                                      
                                        Selective Self-test supported.                                                        
SMART capabilities:            (0x0003) Saves SMART data before entering power-saving mode.                                                            
                                        Supports SMART auto save timer.                                                      
Error logging capability:        (0x01) Error logging supported.                                                              
                                        General Purpose Logging supported.                                                    
Short self-test routine                                                                                                      
recommended polling time:        (   2) minutes.                                                                              
Extended self-test routine                                                                                                    
recommended polling time:        ( 391) minutes.          
Conveyance self-test routine                                                                                                  
recommended polling time:        (   5) minutes.                                                                              
SCT capabilities:              (0x703d) SCT Status supported.                                                                
                                        SCT Error Recovery Control supported.                                                
                                        SCT Feature Control supported.                                                        
                                        SCT Data Table supported.                                                            
                                                                                                                             
SMART Attributes Data Structure revision number: 16                                                                          
Vendor Specific SMART Attributes with Thresholds:                                                                            
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                              
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0                                      
  3 Spin_Up_Time            0x0027   175   174   021    Pre-fail  Always       -       6241                                  
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       80                                    
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                      
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                      
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       205                                    
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                      
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                      
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       80                                    
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       6                                      
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       77                                    
194 Temperature_Celsius     0x0022   125   118   000    Old_age   Always       -       25                                    
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                      
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0                                      
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                      
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                      
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0                                      
                                                                                                                             
SMART Error Log Version: 1                                                                                                    
No Errors Logged                                                                                                              
                                                                                                                             
SMART Self-test log structure revision number 1                                                                              
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                              
# 1  Extended offline    Completed without error       00%       194         -                                                
                                                                                                                             
SMART Selective self-test log data structure revision number 1                                                                
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                  
    1        0        0  Not_testing                                                                                          
    2        0        0  Not_testing                                                                                          
    3        0        0  Not_testing                                                                                          
    4        0        0  Not_testing                                                                                          
    5        0        0  Not_testing                                                                                          
Selective self-test flags (0x0):                                                                                              
  After scanning selected spans, do NOT read-scan remainder of disk.                                                          
If Selective self-test is pending on power-up, resume after 0 minute delay.

ada1:

Code:

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===                                                                                          
Model Family:     Western Digital Red (AF)                                                                                    
Device Model:     WDC WD30EFRX-68EUZN0                                                                                        
Serial Number:    WD-WMC4N0J49C2N                                                                                            
LU WWN Device Id: 5 0014ee 65aaeddc5                                                                                          
Firmware Version: 82.00A82                                                                                                    
User Capacity:    3,000,592,982,016 bytes [3.00 TB]                                                                          
Sector Sizes:     512 bytes logical, 4096 bytes physical                                                                      
Rotation Rate:    5400 rpm                                                                                                    
Device is:        In smartctl database [for details use: -P show]                                                            
ATA Version is:   ACS-2 (minor revision not indicated)                                                                        
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)                                                                      
Local Time is:    Mon Mar 28 12:55:11 2016 BST                                                                                
SMART support is: Available - device has SMART capability.                                                                    
SMART support is: Enabled                                                                                                    
                                                                                                                             
=== START OF READ SMART DATA SECTION ===                                                                                      
SMART overall-health self-assessment test result: PASSED                                                                      
                                                                                                                             
General SMART Values:                                                                                                        
Offline data collection status:  (0x00) Offline data collection activity was never started.                                    
                                        Auto Offline Data Collection: Disabled.                                              
Self-test execution status:      (   0) The previous self-test routine completed without error or no self-test has ever been run.                                                                            
Total time to complete Offline                                                                                                
data collection:                (40560) seconds.                                                                              
Offline data collection                                                                                                      
capabilities:                    (0x7b) SMART execute Offline immediate.                                                      
                                        Auto Offline data collection on/off support.                                          
                                        Suspend Offline collection upon new command.                                                                    
                                        Offline surface scan supported.                                                      
                                        Self-test supported.                                                                  
                                        Conveyance Self-test supported.                                                      
                                        Selective Self-test supported.                                                        
SMART capabilities:            (0x0003) Saves SMART data before entering                                                      
                                        power-saving mode.                                                                    
                                        Supports SMART auto save timer.                                                      
Error logging capability:        (0x01) Error logging supported.                                                              
                                        General Purpose Logging supported.                                                    
Short self-test routine                                                                                                      
recommended polling time:        (   2) minutes.                                                                              
Extended self-test routine                                                                                                    
recommended polling time:        ( 407) minutes.                  
Conveyance self-test routine                                                                                                  
recommended polling time:        (   5) minutes.                                                                              
SCT capabilities:              (0x703d) SCT Status supported.                                                                
                                        SCT Error Recovery Control supported.                                                
                                        SCT Feature Control supported.                                                        
                                        SCT Data Table supported.                                                            
                                                                                                                             
SMART Attributes Data Structure revision number: 16                                                                          
Vendor Specific SMART Attributes with Thresholds:                                                                            
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                              
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0                                      
  3 Spin_Up_Time            0x0027   176   174   021    Pre-fail  Always       -       6200                                  
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       80                                    
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                      
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                      
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       206                                    
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                      
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                      
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       80                                    
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       6                                      
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       77                                    
194 Temperature_Celsius     0x0022   125   119   000    Old_age   Always       -       25                                    
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                      
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0                                      
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                      
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                      
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0                                    
                                                                                                                             
SMART Error Log Version: 1                                                                                                    
No Errors Logged                                                                                                              
                                                                                                                             
SMART Self-test log structure revision number 1                                                                              
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                              
# 1  Extended offline    Completed without error       00%       195         -                                                
                                                                                                                             
SMART Selective self-test log data structure revision number 1                                                                
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                  
    1        0        0  Not_testing                                                                                          
    2        0        0  Not_testing                                                                                          
    3        0        0  Not_testing                                                                                          
    4        0        0  Not_testing                                                                                          
    5        0        0  Not_testing                                                                                          
Selective self-test flags (0x0):                                                                                              
  After scanning selected spans, do NOT read-scan remainder of disk.                                                          
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Jonas LV]

S.M.A.R.T. results continued:

ada2:

Code:

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                

=== START OF INFORMATION SECTION ===                                                                                             
Model Family:     Western Digital Red (AF)                                                                                       
Device Model:     WDC WD30EFRX-68EUZN0                                                                                           
Serial Number:    WD-WCC4N3HZTLKR                                                                                               
LU WWN Device Id: 5 0014ee 260febc82                                                                                             
Firmware Version: 82.00A82                                                                                                       
User Capacity:    3,000,592,982,016 bytes [3.00 TB]                                                                             
Sector Sizes:     512 bytes logical, 4096 bytes physical                                                                         
Rotation Rate:    5400 rpm                                                                                                       
Device is:        In smartctl database [for details use: -P show]                                                               
ATA Version is:   ACS-2 (minor revision not indicated)                                                                           
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)                                                                         
Local Time is:    Mon Mar 28 12:57:52 2016 BST                                                                                   
SMART support is: Available - device has SMART capability.                                                                       
SMART support is: Enabled                                                                                                       
                                                                                                                                
=== START OF READ SMART DATA SECTION ===                                                                                         
SMART overall-health self-assessment test result: PASSED                                                                         
                                                                                                                                
General SMART Values:                                                                                                           
Offline data collection status:  (0x00) Offline data collection activity was never started.                                                               
                                        Auto Offline Data Collection: Disabled.                                                 
Self-test execution status:      (   0) The previous self-test routine completed without error or no self-test has ever been run.                                                                               
Total time to complete Offline                                                                                                   
data collection:                (41280) seconds.                                                                                 
Offline data collection                                                                                                         
capabilities:                    (0x7b) SMART execute Offline immediate.                                                         
                                        Auto Offline data collection on/off support.                                             
                                        Suspend Offline collection upon new command.                                                                       
                                        Offline surface scan supported.                                                         
                                        Self-test supported.                                                                     
                                        Conveyance Self-test supported.                                                         
                                        Selective Self-test supported.                                                           
SMART capabilities:            (0x0003) Saves SMART data before entering                                                         
                                        power-saving mode.                                                                       
                                        Supports SMART auto save timer.                                                         
Error logging capability:        (0x01) Error logging supported.                                                                 
                                        General Purpose Logging supported.                                                       
Short self-test routine                                                                                                         
recommended polling time:        (   2) minutes.                                                                                 
Extended self-test routine                                                                                                       
recommended polling time:        ( 414) minutes.                                                                                 
:                                                                         
Conveyance self-test routine                                                                                                     
recommended polling time:        (   5) minutes.                                                                                 
SCT capabilities:              (0x703d) SCT Status supported.                                                                   
                                        SCT Error Recovery Control supported.                                                   
                                        SCT Feature Control supported.                                                           
                                        SCT Data Table supported.                                                               
                                                                                                                                
SMART Attributes Data Structure revision number: 16                                                                             
Vendor Specific SMART Attributes with Thresholds:                                                                               
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                 
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0                                         
  3 Spin_Up_Time            0x0027   177   176   021    Pre-fail  Always       -       6133                                     
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       80                                       
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                         
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                         
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       206                                       
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                         
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                         
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       80                                       
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       6                                         
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       77                                       
194 Temperature_Celsius     0x0022   127   119   000    Old_age   Always       -       23                                       
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                         
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0                                         
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                         
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                         
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0                                         

SMART Error Log Version: 1                                                                                                       
No Errors Logged                                                                                                                 
                                                                                                                                
SMART Self-test log structure revision number 1                                                                                 
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                                 
# 1  Extended offline    Completed without error       00%       195         -                                                   
                                                                                                                                
SMART Selective self-test log data structure revision number 1                                                                   
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                     
    1        0        0  Not_testing                                                                                             
    2        0        0  Not_testing                                                                                             
    3        0        0  Not_testing                                                                                             
    4        0        0  Not_testing                                                                                             
    5        0        0  Not_testing                                                                                             
Selective self-test flags (0x0):                                                                                                 
  After scanning selected spans, do NOT read-scan remainder of disk.                                                             
If Selective self-test is pending on power-up, resume after 0 minute delay.

ada3:

Code:

smartctl 6.3 2014-07-26 r3976 [FreeBSD 9.3-RELEASE-p13 amd64] (local build)
Copyright (C) 2002-14, Bruce Allen, Christian Franke, www.smartmontools.org                                                 
=== START OF INFORMATION SECTION ===                                                                                             
Model Family:     Western Digital Red (AF)                                                                                       
Device Model:     WDC WD30EFRX-68EUZN0                                                                                           
Serial Number:    WD-WMC4N0J07CSE                                                                                               
LU WWN Device Id: 5 0014ee 6b0041c1b                                                                                             
Firmware Version: 82.00A82                                                                                                       
User Capacity:    3,000,592,982,016 bytes [3.00 TB]                                                                             
Sector Sizes:     512 bytes logical, 4096 bytes physical                                                                         
Rotation Rate:    5400 rpm                                                                                                       
Device is:        In smartctl database [for details use: -P show]                                                               
ATA Version is:   ACS-2 (minor revision not indicated)                                                                           
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)                                                                         
Local Time is:    Mon Mar 28 12:59:06 2016 BST                                                                                   
SMART support is: Available - device has SMART capability.                                                                       
SMART support is: Enabled                                                                                                       
                                                                                                                                
=== START OF READ SMART DATA SECTION ===                                                                                         
SMART overall-health self-assessment test result: PASSED                                                                         
                                                                                                                                
General SMART Values:                                                                                                           
Offline data collection status:  (0x00) Offline data collection activity                                                         
                                        was never started.                                                                       
                                        Auto Offline Data Collection: Disabled.                                                 
Self-test execution status:      (   0) The previous self-test routine completed without error or no self-test has ever been run.                                                                   
Total time to complete Offline                                                                                                   
data collection:                (40500) seconds.                                                                                 
Offline data collection                                                                                                         
capabilities:                    (0x7b) SMART execute Offline immediate.                                                         
                                        Auto Offline data collection on/off support.                                             
                                        Suspend Offline collection upon new command.                                                                       
                                        Offline surface scan supported.                                                         
                                        Self-test supported.                                                                     
                                        Conveyance Self-test supported.                                                         
                                        Selective Self-test supported.                                                           
SMART capabilities:            (0x0003) Saves SMART data before entering                                                         
                                        power-saving mode.                                                                       
                                        Supports SMART auto save timer.                                                         
Error logging capability:        (0x01) Error logging supported.                                                                 
                                        General Purpose Logging supported.                                                       
Short self-test routine                                                                                                         
recommended polling time:        (   2) minutes.                                                                                 
Extended self-test routine                                                                                                       
recommended polling time:        ( 406) minutes.                                                                                 
:                                                       
Conveyance self-test routine                                                                                                     
recommended polling time:        (   5) minutes.                                                                                 
SCT capabilities:              (0x703d) SCT Status supported.                                                                   
                                        SCT Error Recovery Control supported.                                                   
                                        SCT Feature Control supported.                                                           
                                        SCT Data Table supported.                                                               
                                                                                                                                
SMART Attributes Data Structure revision number: 16                                                                             
Vendor Specific SMART Attributes with Thresholds:                                                                               
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE                                 
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0                                         
  3 Spin_Up_Time            0x0027   177   176   021    Pre-fail  Always       -       6141                                     
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       80                                       
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0                                         
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0                                         
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       206                                       
10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0                                         
11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0                                         
12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       80                                       
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       6                                         
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       77                                       
194 Temperature_Celsius     0x0022   127   118   000    Old_age   Always       -       23                                       
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0                                         
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0                                         
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0                                         
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0                                         
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0                                         
                                                                                                                                
SMART Error Log Version: 1                                                                                                       
No Errors Logged                                                                                                                 
                                                                                                                                
SMART Self-test log structure revision number 1                                                                                 
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error                                 
# 1  Extended offline    Completed without error       00%       195         -                                                   
                                                                                                                                
SMART Selective self-test log data structure revision number 1                                                                   
SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS                                                                                     
    1        0        0  Not_testing                                                                                             
    2        0        0  Not_testing                                                                                             
    3        0        0  Not_testing                                                                                             
    4        0        0  Not_testing                                                                                             
    5        0        0  Not_testing                                                                                             
Selective self-test flags (0x0):                                                                                                 
  After scanning selected spans, do NOT read-scan remainder of disk.                                                             
If Selective self-test is pending on power-up, resume after 0 minute delay.

 

[Jonas LV]

Thank you all for commenting and hand holding through figuring out on what is going on.

So to my understanding the HDD's are fine? It is just a soft issue? - Please comment.
Do I perform RAM test - if the RAM pair is any bad? (what command should I run?)

Please comment, as this will help me to raise questions and make decision should I throw away my current configured hardware and or software.

From you comments I am now convinced that the data that is corrupted is buried for good?