Other HDD (other pool) is causing error other disk?

[mcpaha]

Hi!

I am very noob about Freenas and mainly testing it at the moment. Now I have an interesting situation where I really don't know a solution...

1 have 2 SATA HDD:s striped (3TB & 2TB) with one pool and 5 SATA (1TB) Raidz1 HDD:s at other pool.

I started to get these ATA4 error which caused freenas stopped to start -> so it just checking disk... Freenas operating system is at SSD-disk and it is working.

So... First this occured I checked cables and got everything working well but now it occured again... I started to find which one is causing this (rather than randomly check everything) and it seems (need to check more) when I am unplug that 3TB disk then my system starting... However I am 99% sure that this 3TB disk is healthy and problem is somewhere else...

When I unplug SATA cable from from that 3TB disk - no affect BUT when I unplug power cable then freenas starts (only missing that stripe)...

How can I check is this 3TB disk really faulty and how the heck power cable can cause this?? Or what you suggest...

Error:

Error after unplug 3TB HDD:

So... it is not a huge problem to lose movie/tv-series but how to heck I can solve this problem and what is causing that?

In the attachment is dmesg... Sorry can't tell more at this knowledge...

 

[mcpaha]

edit: 3TB & 2TB hdd:s are attached pci-sata card and both hdd:s are recognized when attached...

 

[mcpaha]

edit2: Actually it seems that even if 3TB HDD is recognized - it causes this error (I removed all 1 TB disks and got this error). Is there any way to try to check & save data from there?

 

[sretalla]

How can I check is this 3TB disk really faulty and how the heck power cable can cause this??

You can run SMART tests (Tasks | SMART in the GUI) on the disk and then look at the report with:

smartctl -a /dev/ada0 (or ada1 or whatever it is currently coming up as based on where and how it is attached)

You may be seeing the symptoms of a dying power supply which can't quite make it to supply enough power for all your disks at the same time... if you have a spare somewhere, now would be the time to test it.

It might also be agreat time for you to share your hardware setup so we can help you better. Look at the RED link at the top of the forum page to see the list the forum is expecting to see.

 

[mcpaha]

Thx for the answer... I'll give hw setup later... Anyway it seems that it is not power failure because I plugged those 2 HDD:s other computer power and freenas is still missing that 3TB drive and won't start...

If I get that HDD working one way or other (at other pc first) - is there a way to get data saved from 2 stripied hdd:s if I am using f.e. ubuntu with zfs support?

 

[mcpaha]

Ok, I attached these 2 disks to the other pc and running kubuntu live... (cannot run test with my freenas pc because it wont start with this 3TB device attached)...

Code:

kubuntu@kubuntu:~$ sudo smartctl -ia /dev/sde
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Green
Device Model:     WDC WD30EZRX-00D8PB0
Serial Number:    WD-WCC4N1379655
LU WWN Device Id: 5 0014ee 2b4fa33a5
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Thu Jul 23 12:04:04 2020 EEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (38580) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 387) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x7035) SCT Status supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   179   175   021    Pre-fail  Always       -       6041
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2204
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   046   046   000    Old_age   Always       -       39596
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   098   098   000    Old_age   Always       -       2136
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       99
193 Load_Cycle_Count        0x0032   167   167   000    Old_age   Always       -       99752
194 Temperature_Celsius     0x0022   122   091   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     39513         -
# 2  Short offline       Completed without error       00%     39319         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

kubuntu@kubuntu:~$ sudo smartctl -t short /dev/sde
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Thu Jul 23 12:07:12 2020 EEST
Use smartctl -X to abort test.
kubuntu@kubuntu:~$ sudo smartctl -l selftest /dev/sde
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%     39596         -
# 2  Extended offline    Completed without error       00%     39513         -
# 3  Short offline       Completed without error       00%     39319         -

and just in case 2 TB HDD test also...>

Code:

kubuntu@kubuntu:~$ sudo smartctl -ia /dev/sdc
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Caviar Green
Device Model:     WDC WD20EADS-42R6B0
Serial Number:    WD-WCAVY4373838
LU WWN Device Id: 5 0014ee 2af40f95e
Firmware Version: 02.00A02
User Capacity:    2,000,398,934,016 bytes [2.00 TB]
Sector Size:      512 bytes logical/physical
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ATA8-ACS (minor revision not indicated)
SATA Version is:  SATA 2.6, 1.5 Gb/s
Local Time is:    Thu Jul 23 12:12:26 2020 EEST
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (40800) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   2) minutes.
Extended self-test routine
recommended polling time:        ( 464) minutes.
Conveyance self-test routine
recommended polling time:        (   5) minutes.
SCT capabilities:              (0x303f) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       486
  3 Spin_Up_Time            0x0027   159   141   021    Pre-fail  Always       -       9041
  4 Start_Stop_Count        0x0032   094   094   000    Old_age   Always       -       6799
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   035   035   000    Old_age   Always       -       47829
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1693
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       80
193 Load_Cycle_Count        0x0032   085   085   000    Old_age   Always       -       347482
194 Temperature_Celsius     0x0022   120   088   000    Old_age   Always       -       32
195 Hardware_ECC_Recovered  0x0036   200   200   000    Old_age   Always       -       0
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       5
198 Offline_Uncorrectable   0x0030   200   200   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   200   200   000    Old_age   Offline      -       3

SMART Error Log Version: 1
Warning: ATA error count 38 inconsistent with error log pointer 5

ATA Error Count: 38 (device log contains only the most recent five errors)
        CR = Command Register [HEX]
        FR = Features Register [HEX]
        SC = Sector Count Register [HEX]
        SN = Sector Number Register [HEX]
        CL = Cylinder Low Register [HEX]
        CH = Cylinder High Register [HEX]
        DH = Device/Head Register [HEX]
        DC = Device Command Register [HEX]
        ER = Error register [HEX]
        ST = Status register [HEX]
Powered_Up_Time is measured from power on, and printed as
DDd+hh:mm:SS.sss where DD=days, hh=hours, mm=minutes,
SS=sec, and sss=millisec. It "wraps" after 49.710 days.

Error 38 occurred at disk power-on lifetime: 47828 hours (1992 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 48 25 40 e8  Error: UNC 8 sectors at LBA = 0x08402548 = 138421576

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 48 25 40 e8 02      00:04:25.158  READ DMA

Error 37 occurred at disk power-on lifetime: 47828 hours (1992 days + 20 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 40 25 40 e8  Error: UNC 8 sectors at LBA = 0x08402540 = 138421568

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 40 25 40 e8 02      00:05:16.917  READ DMA
  c8 00 08 d0 2d 5b e0 02      00:05:16.917  READ DMA
  c8 00 20 b0 21 40 ec 02      00:05:16.890  READ DMA

Error 36 occurred at disk power-on lifetime: 47827 hours (1992 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 48 25 40 e8  Error: UNC 8 sectors at LBA = 0x08402548 = 138421576

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 48 25 40 e8 02      00:01:40.825  READ DMA

Error 35 occurred at disk power-on lifetime: 47827 hours (1992 days + 19 hours)
  When the command that caused the error occurred, the device was active or idle.

  After command completion occurred, registers were:
  ER ST SC SN CL CH DH
  -- -- -- -- -- -- --
  40 51 08 48 25 40 e8  Error: UNC 8 sectors at LBA = 0x08402548 = 138421576

  Commands leading to the command that caused the error were:
  CR FR SC SN CL CH DH DC   Powered_Up_Time  Command/Feature_Name
  -- -- -- -- -- -- -- --  ----------------  --------------------
  c8 00 08 48 25 40 e8 02      00:01:39.737  READ DMA

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Extended offline    Completed without error       00%     47737         -
# 2  Short offline       Completed without error       00%     47580         -
# 3  Short offline       Aborted by host               80%     20926         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

kubuntu@kubuntu:~$ sudo smartctl -t short /dev/sdc
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF OFFLINE IMMEDIATE AND SELF-TEST SECTION ===
Sending command: "Execute SMART Short self-test routine immediately in off-line mode".
Drive command "Execute SMART Short self-test routine immediately in off-line mode" successful.
Testing has begun.
Please wait 2 minutes for test to complete.
Test will complete after Thu Jul 23 12:15:14 2020 EEST
Use smartctl -X to abort test.
kubuntu@kubuntu:~$ sudo smartctl -l selftest /dev/sdc
smartctl 7.1 2019-12-30 r5022 [x86_64-linux-5.4.0-26-generic] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF READ SMART DATA SECTION ===
SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed: read failure       90%     47829         138421568
# 2  Extended offline    Completed without error       00%     47737         -
# 3  Short offline       Completed without error       00%     47580         -
# 4  Short offline       Aborted by host               80%     20926         -

So if (at the end) I cannot get these working with freenas pc... Is there a way to get data with kubuntu live and zfs support...

 

[mcpaha]

I managed to import zpool to kubuntu live and looks like this:

Code:

kubuntu@kubuntu:/mnt$ zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
warehouse  4.53T  2.09T  2.44T        -         -     0%    46%  1.00x    ONLINE  -
kubuntu@kubuntu:/mnt$ zpool status
  pool: warehouse
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        warehouse   ONLINE       0     0     0
          sde2      ONLINE       0     0     0
          sdc2      ONLINE       2     0     0

errors: No known data errors



kubuntu@kubuntu:/mnt$ zpool list
NAME        SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP    HEALTH  ALTROOT
warehouse  4.53T  2.09T  2.44T        -         -     0%    46%  1.00x    ONLINE  -
kubuntu@kubuntu:/mnt$ zpool status
  pool: warehouse
 state: ONLINE
status: One or more devices has experienced an unrecoverable error.  An
        attempt was made to correct the error.  Applications are unaffected.
action: Determine if the device needs to be replaced, and clear the errors
        using 'zpool clear' or replace the device with 'zpool replace'.
   see: http://zfsonlinux.org/msg/ZFS-8000-9P
  scan: none requested
config:

        NAME        STATE     READ WRITE CKSUM
        warehouse   ONLINE       0     0     0
          sde2      ONLINE       0     0     0
          sdc2      ONLINE       2     0     0

errors: No known data errors

How do I mount that pool in Kubuntu? I tried with: sudo zfs set mountpoint=/mnt/zpool warehouse

and it did not gives me any error but it did not mount that pool...

 

[mcpaha]

Hmm... cannot edit sent message this forum??
Edit: sudo zfs mount warehouse

Code:

warehouse on /mnt/zpool type zfs (rw,xattr,noacl)
kubuntu@kubuntu:/mnt$ cd zpool/
kubuntu@kubuntu:/mnt/zpool$ ls -la
total 1
drwxr-xr-x 3 root root  3 Jun 25 12:24 .
drwxr-xr-x 1 root root 60 Jul 23 13:47 ..
drwxr-xr-x 2 root root  2 Jun 25 12:24 Media
kubuntu@kubuntu:/mnt/zpool$ cd Media/
kubuntu@kubuntu:/mnt/zpool/Media$ ls -la
total 1
drwxr-xr-x 2 root root 2 Jun 25 12:24 .
drwxr-xr-x 3 root root 3 Jun 25 12:24 ..
kubuntu@kubuntu:/mnt/zpool/Media$

Disk broken? or why there is no data?

 

[sretalla]

Disk broken? or why there is no data?

Are you sure you're looking where it's mounted... ?

How about looking at zfs list


sdc is definitely dying... do something about that.

Anyway it seems that it is not power failure because I plugged those 2 HDD:s other computer power and freenas is still missing that 3TB drive and won't start...

I'm not sure if you're saying that you plugged the power from a different computer into the disks which were still connected to the original computer... I don't think that usually works well
What's the origin of the 3TB drive? is it shucked from an external HDD enclosure? (maybe it's the 3rd or 5th pin trick).

 

[Yorick]

are attached pci-sata card

This very likely isn't causing your issues, and, while you're at it, seriously consider replacing that with a HBA and SAS-to-SATA breakout cable. You'll have a better supported, more stable setup than an add-in SATA controller.

 

[mcpaha]

Ok I managed to mount my pool:

Code:

root@kubuntu:/mnt# zfs list
NAME              USED  AVAIL     REFER  MOUNTPOINT
warehouse        2.09T  2.30T       88K  /mnt/zpool
warehouse/Media  2.09T  2.30T     2.09T  /mnt/zpool/Media
root@kubuntu:/mnt#

I see my folders & files but if trying to copy then it says that file does not exist...(using Krusader with root privileges).

Should I chmod all folders & files? Because with terminal and more command it shows that there is something inside of file...

 

[mcpaha]

Ok with terminal I can copy (tested only couple of files) files to the save place but I'm confused why this disk is working with other pc but not with other? Tried different sata-cables, different sata ports etc...

Also I'm worried situation that one disk failure is blocking freenas to start and also if I am starting it one disk removed then freenas do not show me all my pools and what there should be and what is missing... or I might noob about this...

 

[sretalla]

Also I'm worried situation that one disk failure is blocking freenas to start and also if I am starting it one disk removed then freenas do not show me all my pools and what there should be and what is missing... or I might noob about this...

You have a failing disk which is in a striped pool (when that disk finally dies, your pool is gone).

That failing disk results in the SATA controller getting instructions to do something with the failing disk, but has to wait for the disk to try (and fail) on every request, causing the controller to have difficulties itself.

Some controllers handle that better than others and it sounds like the host where you could finally mount the pool has one that's dealing with it better than your FreeNAS host's controller did.

As recommended by @Yorick you would be better to have a (forum recommended) HBA in place of the SATA controller you're currently using, which would be able to better handle a failing disk.

It seems like you have a lot of work in front of you, but you should also consider how important that data is to you as a striped pool is not for storing data you're keen on keeping.

 

[mcpaha]

Thx for the help to everybody. Saving this striped data is more or less teaching myself about freenas & zfs. I knew striped data problem.

Still... didn't get any answer for my question what might be faulty part (or reason) because last night I moved 3 TB data from that striped disks where other pc (with freenas) cannot read that 3 TB hdd. At the weekend I'm gonna test that disk all over again for every sata port what I can find ;).