ZFS Raid FAULTED corrupted data

[Vishimtar]

Hello,
My Freenas BOX: FreeNAS Build FreeNAS-9.10.2-U3 (e1497f269) versions Hardware is OEM Computer box Intel (R) Core (TM) i3 CPU 540 @ 3.07GHz Gigabyte Technology Co., Ltd. Motherboard Model: H55M-S2H 8GB DDR3 Memory 4 X Western Digital Red 3TB HDD ZFS Raid Group and FreeNAS 1x 250GB hdd install

In my first installation, the prompt was working entirely with components on the onboard. There is a raid group consisting of ZFS Raid group 4x 3TB WD Red NAS hhds.
ZFS had to repair a raid disk, I changed the disk and repaired it worked fine after a long time started to fail again This time I started to get the ATA Control error, so I changed the SATA cables and power supply
I replaced the defective disk with a new one again and ZFS sync process started and I saw my data in the ZFS raid group I started to get ATA Control error this time under the load while the sync process was going on and the process stopped because the ZFS raid group saw it and therefore the same name system was not included After closing the FreeNAS box, removing the ZFS Raid disks and opening them again
I deleted the zfsraid group defined via the web gui and chose not to delete the data in the Raid Group. I closed the system and reintroduced the ZFS Raid group disks into the system, this time disks SATA controller: Marvell Technology Group Ltd. 88SE9230 PCIe SATA 6Gb / s Controller (rev 11) PCI SATA controller connected and plugged in the disks
I turned on the system and all the disks look right hardware but I can't import the ZFS Raid group.

I used the zpool import -FX zfsraid command in the import transaction. After 2 days of operation, the server restarts and the operation fails. For now, gpartitions appear on disks in glabel, but I can't import zfsraid because I can't do anything to the newly included disk.

ZFS Raid structure and some system outputs

Can you help, how can I recover the data here?

ZFS Raid structure and some system outputs


GpartShow

Code:

=>        34  5860533101  ada1  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada2  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>        34  5860533101  ada3  GPT  (2.7T)
          34          94        - free -  (47K)
         128     4194304     1  freebsd-swap  (2.0G)
     4194432  5856338696     2  freebsd-zfs  (2.7T)
  5860533128           7        - free -  (3.5K)

=>       34  488397101  ada4  GPT  (233G)
         34       1024     1  bios-boot  (512K)
       1058          6        - free -  (3.0K)
       1064  488396064     2  freebsd-zfs  (233G)
  488397128          7        - free -  (3.5K)

GpartList

Code:

Geom name: ada1
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada1p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: b0a83cc6-d8f2-11e8-9d6b-1c6f6521f14f
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada1p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: b0b64d6e-d8f2-11e8-9d6b-1c6f6521f14f
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: ada1
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Geom name: ada2
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada2p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: af28cad4-d8f2-11e8-9d6b-1c6f6521f14f
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada2p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: af37ce9a-d8f2-11e8-9d6b-1c6f6521f14f
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: ada2
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0

Geom name: ada3
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 5860533134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada3p1
   Mediasize: 2147483648 (2.0G)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: ae7be949-d8f2-11e8-9d6b-1c6f6521f14f
   rawtype: 516e7cb5-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2147483648
   offset: 65536
   type: freebsd-swap
   index: 1
   end: 4194431
   start: 128
2. Name: ada3p2
   Mediasize: 2998445412352 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0
   rawuuid: ae8bab76-d8f2-11e8-9d6b-1c6f6521f14f
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 2998445412352
   offset: 2147549184
   type: freebsd-zfs
   index: 2
   end: 5860533127
   start: 4194432
Consumers:
1. Name: ada3
   Mediasize: 3000592982016 (2.7T)
   Sectorsize: 512
   Stripesize: 4096
   Stripeoffset: 0
   Mode: r0w0e0




------------------------------------------------------------------------------
Geom name: ada4
modified: false
state: OK
fwheads: 16
fwsectors: 63
last: 488397134
first: 34
entries: 128
scheme: GPT
Providers:
1. Name: ada4p1
   Mediasize: 524288 (512K)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 17408
   Mode: r0w0e0
   rawuuid: 95745a9f-acf6-11e8-98c9-1c6f6521f14f
   rawtype: 21686148-6449-6e6f-744e-656564454649
   label: (null)
   length: 524288
   offset: 17408
   type: bios-boot
   index: 1
   end: 1057
   start: 34
2. Name: ada4p2
   Mediasize: 250058784768 (233G)
   Sectorsize: 512
   Stripesize: 0
   Stripeoffset: 544768
   Mode: r1w1e1
   rawuuid: 957c5fa1-acf6-11e8-98c9-1c6f6521f14f
   rawtype: 516e7cba-6ecf-11d6-8ff8-00022d09712b
   label: (null)
   length: 250058784768
   offset: 544768
   type: freebsd-zfs
   index: 2
   end: 488397127
   start: 1064
Consumers:
1. Name: ada4
   Mediasize: 250059350016 (233G)
   Sectorsize: 512
   Mode: r1w1e2

Glabel Status

Code:

                                      Name  Status  Components
gptid/b0a83cc6-d8f2-11e8-9d6b-1c6f6521f14f     N/A  ada1p1
gptid/b0b64d6e-d8f2-11e8-9d6b-1c6f6521f14f     N/A  ada1p2
gptid/af37ce9a-d8f2-11e8-9d6b-1c6f6521f14f     N/A  ada2p2
gptid/ae7be949-d8f2-11e8-9d6b-1c6f6521f14f     N/A  ada3p1
gptid/ae8bab76-d8f2-11e8-9d6b-1c6f6521f14f     N/A  ada3p2
gptid/95745a9f-acf6-11e8-98c9-1c6f6521f14f     N/A  ada4p1
gptid/af28cad4-d8f2-11e8-9d6b-1c6f6521f14f     N/A  ada2p1

Camcontrol List

Code:

<WDC WD30EFRX-68EUZN0 82.00A82>    at scbus0 target 0 lun 0 (pass0,ada0)
<WDC WD30EFRX-68EUZN0 80.00A80>    at scbus1 target 0 lun 0 (pass1,ada1)
<WDC WD30EFRX-68N32N0 82.00A82>    at scbus2 target 0 lun 0 (pass2,ada2)
<WDC WD30EFRX-68N32N0 82.00A82>    at scbus3 target 0 lun 0 (pass3,ada3)
<Marvell Console 1.01>             at scbus7 target 0 lun 0 (pass4)
<WDC WD2502ABYS-50B7A1 02.03B06>   at scbus9 target 0 lun 0 (pass5,ada4)

smartctl -a /dev/ada0

Code:

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WCC4N3ZAKJN9
LU WWN Device Id: 5 0014ee 2bc0386bd
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Mar 16 13:04:41 2020 +03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (39120) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 393) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x703d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   100   253   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   100   253   021    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       3
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       69
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       3
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       0
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       74
194 Temperature_Celsius     0x0022   122   118   000    Old_age   Always       -       28
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

smartctl -a /dev/ada1

Code:

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68EUZN0
Serial Number:    WD-WMC4N1279182
LU WWN Device Id: 5 0014ee 2b47b2e4f
Firmware Version: 80.00A80
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-2 (minor revision not indicated)
SATA Version is:  SATA 3.0, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Mar 16 13:04:52 2020 +03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (41640) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 418) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x703d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   181   176   021    Pre-fail  Always       -       5950
  4 Start_Stop_Count        0x0032   098   098   000    Old_age   Always       -       2096
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   200   200   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   083   083   000    Old_age   Always       -       13011
 10 Spin_Retry_Count        0x0032   100   100   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   100   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   099   099   000    Old_age   Always       -       1234
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       128
193 Load_Cycle_Count        0x0032   199   199   000    Old_age   Always       -       3683
194 Temperature_Celsius     0x0022   124   098   000    Old_age   Always       -       26
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   188   000    Old_age   Always       -       302
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

smartctl -a /dev/ada2

Code:

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68N32N0
Serial Number:    WD-WCC7K0SNCAYC
LU WWN Device Id: 5 0014ee 2ba2c5d36
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Mar 16 13:05:20 2020 +03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (30780) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 328) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x303d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   168   166   021    Pre-fail  Always       -       6600
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       64
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       11124
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       63
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       27
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       61
194 Temperature_Celsius     0x0022   123   111   000    Old_age   Always       -       27
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 

smartctl -a /dev/ada3

Code:

smartctl 6.5 2016-05-07 r4318 [FreeBSD 10.3-STABLE amd64] (local build)
Copyright (C) 2002-16, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Western Digital Red
Device Model:     WDC WD30EFRX-68N32N0
Serial Number:    WD-WCC7K2PX8H0H
LU WWN Device Id: 5 0014ee 264d6cb2c
Firmware Version: 82.00A82
User Capacity:    3,000,592,982,016 bytes [3.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5400 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Mon Mar 16 13:05:47 2020 +03
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x00)    Offline data collection activity
                    was never started.
                    Auto Offline Data Collection: Disabled.
Self-test execution status:      (   0)    The previous self-test routine completed
                    without error or no self-test has ever 
                    been run.
Total time to complete Offline 
data collection:         (32400) seconds.
Offline data collection
capabilities:              (0x7b) SMART execute Offline immediate.
                    Auto Offline data collection on/off support.
                    Suspend Offline collection upon new
                    command.
                    Offline surface scan supported.
                    Self-test supported.
                    Conveyance Self-test supported.
                    Selective Self-test supported.
SMART capabilities:            (0x0003)    Saves SMART data before entering
                    power-saving mode.
                    Supports SMART auto save timer.
Error logging capability:        (0x01)    Error logging supported.
                    General Purpose Logging supported.
Short self-test routine 
recommended polling time:      (   2) minutes.
Extended self-test routine
recommended polling time:      ( 345) minutes.
Conveyance self-test routine
recommended polling time:      (   5) minutes.
SCT capabilities:            (0x303d)    SCT Status supported.
                    SCT Error Recovery Control supported.
                    SCT Feature Control supported.
                    SCT Data Table supported.

SMART Attributes Data Structure revision number: 16
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x002f   200   200   051    Pre-fail  Always       -       0
  3 Spin_Up_Time            0x0027   168   168   021    Pre-fail  Always       -       6558
  4 Start_Stop_Count        0x0032   100   100   000    Old_age   Always       -       55
  5 Reallocated_Sector_Ct   0x0033   200   200   140    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x002e   100   253   000    Old_age   Always       -       0
  9 Power_On_Hours          0x0032   085   085   000    Old_age   Always       -       11101
 10 Spin_Retry_Count        0x0032   100   253   000    Old_age   Always       -       0
 11 Calibration_Retry_Count 0x0032   100   253   000    Old_age   Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   000    Old_age   Always       -       55
192 Power-Off_Retract_Count 0x0032   200   200   000    Old_age   Always       -       31
193 Load_Cycle_Count        0x0032   200   200   000    Old_age   Always       -       48
194 Temperature_Celsius     0x0022   124   109   000    Old_age   Always       -       26
196 Reallocated_Event_Count 0x0032   200   200   000    Old_age   Always       -       0
197 Current_Pending_Sector  0x0032   200   200   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0030   100   253   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x0032   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0008   100   253   000    Old_age   Offline      -       0

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
No self-tests have been logged.  [To run self-tests, use: smartctl -t]

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.
 

Thank you.

 

[Basil Hendroff]

I'm no h/w expert, but I get the feeling that some of your h/w is not supported e.g. the Marvell SATA controller. Refer to the User Guide Hardware Recommendations and in particular the section on Storage Disks and Controllers.

 

[sretalla]

You have a bunch of CRC errors on ada1.

That could be a cabling issue or a failure of the Marvell SATA controller port (don't see the same on the others, so the controller hadn't entirely packed up.

The first option would be to re-seat and/or replace the SATA cable to it.

What you're calling a ZFS RAID group is a pool.

I don't see the output of zpool status -v in the above.

 

[Vishimtar]

I'm no h/w expert, but I get the feeling that some of your h/w is not supported e.g. the Marvell SATA controller. Refer to the User Guide Hardware Recommendations and in particular the section on Storage Disks and Controllers.

Hello,
I use the same SATA controller in another system that works without any problems. In this system, I changed a disk because of the disk and while ZFS raid sync operation was happening, the process was halfway due to the onboard sata controller error and I wanted to continue the process by plugging an external sata controller to the system, but this time it did not import because there is a ZFS Raid group with the same name. I closed the FreeNAS box and removed the ZFS Raid group, turned the system back on, and said delete the zfsraid group under Storage via web gui. I shut down the system again I reinserted the ZFS Raid group disks and turned on the system, this time I tried to import the zfsraid group using the zpool import -FX command, but I didn't import the zfs group because it says there was a disk error zpool import -FX operation restarts the system after working for 3 days. I did not see anything as an error.

Thank you.

 

[sretalla]

Please share the output from zpool import

Have you checked the cabling for the disk ada1? you have 302 UDMA CRC errors shown in the SMART data for it:
199 UDMA_CRC_Error_Count 0x0032 200 188 000 Old_age Always - 302

 

[Vishimtar]

You have a bunch of CRC errors on ada1.

That could be a cabling issue or a failure of the Marvell SATA controller port (don't see the same on the others, so the controller hadn't entirely packed up.

The first option would be to re-seat and/or replace the SATA cable to it.

What you're calling a ZFS RAID group is a pool.

I don't see the output of zpool status -v in the above.

thanks for your answer, which is CRC error in logs?
I just replaced SATA cables, but I change it again. "Ada1" does not seem to be a problem. Is it possible to rebuild new disk data missing from the glabel side, then I can import it into zfsraid.

zpool status -v

Code:

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Fri Mar 13 03:45:21 2020
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      ada4p2    ONLINE       0     0     0

errors: No known data errors

zpool import zfsraid

Code:

[root@mnas] ~# zpool import zfsraid
cannot import 'zfsraid': I/O error
        Destroy and re-create the pool from
        a backup source.

 

[sretalla]

Is it possible to rebuild new disk data missing from the glabel side, then I can import it into zfsraid.

You're entirely misunderstanding how ZFS and FreeNAS works.
The pool disks work together and are managed by ZFS, which creates partitions and handles rebuilding (called resilvering) disks when necessary. Manually doing anything with glabel, gpart or gdisk without a full understanding of ZFS is folly and should not be attempted if you want a healthy pool.
If you share the output of zpool import, perhaps we can see what can be done to help.

thanks for your answer, which is CRC error in logs?

Look at my second post.

 

[Vishimtar]

You're entirely misunderstanding how ZFS and FreeNAS works.
The pool disks work together and are managed by ZFS, which creates partitions and handles rebuilding (called resilvering) disks when necessary. Manually doing anything with glabel, gpart or gdisk without a full understanding of ZFS is folly and should not be attempted if you want a healthy pool.
If you share the output of zpool import, perhaps we can see what can be done to help.


Look at my second post.

Thanks for your answer, yes, I saw the CRC error now. I forgot to add these outputs, sorry I share it again

zpool status -v

Code:

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0h0m with 0 errors on Fri Mar 13 03:45:21 2020
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      ada4p2    ONLINE       0     0     0

errors: No known data errors

zpool import zfsraid

Code:

[root@mnas] ~# zpool import zfsraid
cannot import 'zfsraid': I/O error
        Destroy and re-create the pool from
        a backup source.

 

[sretalla]

Please share the output of zpool import (not zpool import zfsraid)

 

[Vishimtar]

Please share the output of zpool import (not zpool import zfsraid)

Sorry,

When I say import, ZFS Raid group shows this information in the structure

Code:

[root@mnas] ~# zpool import
   pool: zfsraid
     id: 6796189392018939324
  state: FAULTED
 status: One or more devices were being resilvered.
 action: The pool cannot be imported due to damaged devices or data.
        The pool may be active on another system, but can be imported using
        the '-f' flag.
 config:

        zfsraid                                         FAULTED  corrupted data
          raidz1-0                                      FAULTED  corrupted data
            gptid/ae8bab76-d8f2-11e8-9d6b-1c6f6521f14f  ONLINE
            gptid/af37ce9a-d8f2-11e8-9d6b-1c6f6521f14f  ONLINE
            8211356002075759672                         UNAVAIL  cannot open
            gptid/b0b64d6e-d8f2-11e8-9d6b-1c6f6521f14f  ONLINE

 

[sretalla]

so, since you're missing only 1 disk, you might try zpool import -f zfsraid

 

[Vishimtar]

so, since you're missing only 1 disk, you might try zpool import -f zfsraid

yes, I had to replace the disk because I couldn't solve the missing disk defective disk problem. I inserted a new disc and started importing.

I tried this process three times. FreeNAS continued this process for 2 days, then the system restarted and the operation failed. During this process, I followed the disk section under repoting via web gui, there was no "I / O" traffic in "ADA0", and "I / O" traffic in "ADA1, ADA2, ADA3". My problem is that the disk I just added does not write data for a reason, I think my problem is that I replaced the disk I just inserted with another disk 2 times.

 

[sretalla]

I replaced the disk I just inserted with another disk 2 times.

Did you do anything in the GUI to tell FreeNAS to replace the disk? (this is not automatic).

 

[Vishimtar]

Did you do anything in the GUI to tell FreeNAS to replace the disk? (this is not automatic).

Yes. When zfsraid was degreded, the system could not see the faulty disk and replaced it with a new one. Then automatic sync process started, but before I could finish this process, the onboard SATA controller failed and I had to install and use the external SATA controller.

 

[sretalla]

If you care about getting to your data, you should get hold of an alternate system where the SATA controller works and put the disks into that. Continuing down the road you are on with an external (presumably USB to SATA) controller in the mix on top of an already partially failed SATA controller on the mainboard, the chances that things get back to normal are very slim indeed.

 

[Vishimtar]

the external SATA controller I use is this https://www.digitus.info/en/product...ies/computer-accessories/io-cards/ds-30104-1/

 

[sretalla]

So that's a Marvell based board, which means trouble as far as this forum is concerned.

Look for an LSI HBA card if you want reliable performance. See here: https://www.ixsystems.com/community/resources/hardware-recommendations-guide.12/download

Page 12 for the Disk Controller part. You can typically find a good (used) controller on ebay for under US$50

 

[Vishimtar]

I use the same SATA control card in another FreeNAS box and it works fine. As far as I guess my problem in another FreeNAS system is the problem caused by 1 disk failure from 4 disks in zfsraid Raidz
I inserted a new disk, while doing this operation, several ports on the onboard SATA controller have failed and I guess some disk could not read data from the disk during the sync process. After that, everything began to deteriorate. When I got zfsraid defined error among the errors I received, as a result of not importing to the system, everything turned out fine after I closed the FreeNas box and removed the disks and deleted the zfsraid volume from the web gui.
The import process does not work because gptid / xxxxxxxxx-xxxx-xxxx-xxxxx-xxxxxxxxxx cannot find the information in "ADA0". Can I recreate ADA0 again by reading data from other "ADA1", "ADA2", "ADA3"?

 

[Vishimtar]

Hi, @sretalla

Code:

zpool replace

If I use this command, but how can I create partitions with gpart in ADA0 or do I use the

Code:

zpool replace 
zfsraid 8211356002075759672 ada0

command in this way?


When I saw the processes here, I thought I could recover. 19.3.6. Dealing with Failed Devices

 

[sretalla]

If you're using FreeNAS, these operations should start in the GUI. https://www.ixsystems.com/documentation/freenas/11.3-U1/storage.html#replacing-a-failed-disk

CLI is only for when things go off the rails.

If zpool import -f doesn't work for you, then you can't perform any of the CLI operations on the pool anyway.