Resilver time

[reks]

hello,

Can you check that this time is a normal time what is needed to resilver?

Code:

root@RSONas[/nonexistent]# zpool status -v
  pool: DANE
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Dec 31 16:38:59 2019
    513G scanned at 377M/s, 19.5G issued at 14.3M/s, 14.6T total
    1.24G resilvered, 0.13% done, 12 days 08:21:02 to go
config:

    NAME                                                  STATE     READ WRITE CKSUM
    DANE                                                  DEGRADED     0     0     0
      raidz1-0                                            DEGRADED     0     0     0
        gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        replacing-5                                       FAULTED      0     0     0
          gptid/48d2a697-2a6e-11ea-9473-0cc47a2049d8.eli  FAULTED      6  103K     0  too many errors
          gptid/8cea069e-2be2-11ea-9473-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:28 with 0 errors on Mon Dec 30 03:45:28 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p2  ONLINE       0     0     0
        ada1p2  ONLINE       0     0     0

errors: No known data errors

We have FREENAS-CERTIFIED-2U-A2Z with 9x4TB sata drive.

 

[Mathis1]

It looks like you just started the resilver on the 4TB disk, so the estimation is likely to change a bit.

12 days does seem a bit long for a 4TB drive, but the pool layout will cause this to run a little longer than usual. Raidz1 will require all other disks to recompute for the resilvering disk, and the vdev is fairly wide at 9 drives wide.

I would keep an eye on it, my guess is that it should take about 5-6 days or so to resilver, but it could take longer due to drive speed and other activity on the box. You should get a better idea how long it will take if you check back in on it later today.

 

[reks]

ok i will check for about 2 days and give a answer :)

 

[Johnnie Black]

What model is the new disk? SMR disks take much longer to resilver.

 

[reks]

Seagate Barracuda 4TB 7200 SATA

 

[reks]

Now it cant estamine xD

Code:

root@RSONas[/nonexistent]# zpool status -v
  pool: DANE
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Dec 31 16:38:59 2019
    576G scanned at 62.5M/s, 80.7G issued at 8.76M/s, 14.6T total
    7.36G resilvered, 0.54% done, no estimated completion time
config:

    NAME                                                  STATE     READ WRITE CKSUM
    DANE                                                  DEGRADED     0     0     0
      raidz1-0                                            DEGRADED     0     0     0
        gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        replacing-5                                       FAULTED      0     0     0
          gptid/48d2a697-2a6e-11ea-9473-0cc47a2049d8.eli  FAULTED      6  103K     0  too many errors
          gptid/8cea069e-2be2-11ea-9473-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0
        gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli    ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:28 with 0 errors on Mon Dec 30 03:45:28 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p2  ONLINE       0     0     0
        ada1p2  ONLINE       0     0     0

errors: No known data errors

 

[Johnnie Black]

Seagate Barracuda 4TB 7200 SATA

We need the complete model, e.g. ST4000DM004 is SMR, you can get with smartctl -i /dev/daX

 

[reks]

Code:

=== START OF INFORMATION SECTION ===
Model Family:     Seagate Barracuda 3.5
Device Model:     ST4000DM004-2CV104
Serial Number:    WFN0MDJM
LU WWN Device Id: 5 000c50 0bdbd366b
Firmware Version: 0001
User Capacity:    4,000,787,030,016 bytes [4.00 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    5425 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-3 T13/2161-D revision 5
SATA Version is:  SATA 3.1, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Tue Dec 31 19:22:13 2019 CET
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

Ops it has 5425rpm

 

[Johnnie Black]

Ops it has 5425rpm

Yep, and much worse, it's SMR, so long resilver times are normal.

 

[Apollo]

All my Seagate Archive drives are SMR and copying resilvering to any one of them isn't a problem as long as you don't delete snapshot often.
There is a Resilvering option that can be set under GUI to increase resilvering priority.
I had a failed SMR disk at one time that caused the drive to remain busy all the time with a 20MB/s throughput or so and the entire pool became slow as a result.
Using Netdata can tel you a lot about the performance of the individual disk.

 

[reks]

@Apollo thanks for advince
Now i have

Code:

root@RSONas[/nonexistent]# zpool status -v
  pool: DANE
 state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Dec 31 19:35:30 2019
        537G scanned at 1.93G/s, 42.8G issued at 157M/s, 14.5T total
        3.57G resilvered, 0.29% done, 1 days 02:55:39 to go
config:

        NAME                                                STATE     READ WRITE CKSUM
        DANE                                                ONLINE       0     0     0
          raidz1-0                                          ONLINE       0     0     0
            gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
            gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
            gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
            gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
            gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
            gptid/f924c103-2bfb-11ea-9473-0cc47a2049d8.eli  ONLINE       0     0     8
            gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
            gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
            gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:28 with 0 errors on Mon Dec 30 03:45:28 2019
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0

errors: No known data errors
root@RSONas[/nonexistent]# uptime
11:42PM  up 4 hrs, 1 user, load averages: 3.39, 2.50, 1.23
root@RSONas[/nonexistent]#

 

[reks]

guys,
after some days i have

Code:

root@RSONas[/nonexistent]# zpool status -v
  pool: DANE
 state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Dec 31 19:35:30 2019
    7.11T scanned at 22.0M/s, 6.88T issued at 21.2M/s, 14.5T total
    589G resilvered, 47.32% done, 4 days 09:03:12 to go
config:

    NAME                                                STATE     READ WRITE CKSUM
    DANE                                                DEGRADED     0     0     0
      raidz1-0                                          DEGRADED     0     0     0
        gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/f924c103-2bfb-11ea-9473-0cc47a2049d8.eli  FAULTED      7 76.7K     8  too many errors
        gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:28 with 0 errors on Mon Dec 30 03:45:28 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p2  ONLINE       0     0     0
        ada1p2  ONLINE       0     0     0

errors: No known data errors

about the faulted this da8 (the new attached disk to resilver)
wtf?

 

[Apollo]

guys,
after some days i have

Code:

root@RSONas[/nonexistent]# zpool status -v
  pool: DANE
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Tue Dec 31 19:35:30 2019
    7.11T scanned at 22.0M/s, 6.88T issued at 21.2M/s, 14.5T total
    589G resilvered, 47.32% done, 4 days 09:03:12 to go
config:

    NAME                                                STATE     READ WRITE CKSUM
    DANE                                                DEGRADED     0     0     0
      raidz1-0                                          DEGRADED     0     0     0
        gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/f924c103-2bfb-11ea-9473-0cc47a2049d8.eli  FAULTED      7 76.7K     8  too many errors
        gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:28 with 0 errors on Mon Dec 30 03:45:28 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p2  ONLINE       0     0     0
        ada1p2  ONLINE       0     0     0

errors: No known data errors

about the faulted this da8 (the new attached disk to resilver)
wtf?

Have you tried running Netdata?
If Netdata is available in your version of Freenas, then look at the BUSY graphs on each of the disks. The faulty one should have BUSY near 100% with marginal throughput. That would be the faulty one in my opinion.

 

[reks]

now i have

Code:

root@RSONas[/var/log]# zpool status -v 
  pool: DANE
state: DEGRADED
status: One or more devices are faulted in response to persistent errors.
    Sufficient replicas exist for the pool to continue functioning in a
    degraded state.
action: Replace the faulted device, or use 'zpool clear' to mark the device
    repaired.
  scan: resilvered 589G in 4 days 10:51:31 with 0 errors on Sun Jan  5 06:27:01 2020
config:

    NAME                                                STATE     READ WRITE CKSUM
    DANE                                                DEGRADED     0     0     0
      raidz1-0                                          DEGRADED     0     0     0
        gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/f924c103-2bfb-11ea-9473-0cc47a2049d8.eli  FAULTED      7 76.7K     8  too many errors
        gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:28 with 0 errors on Mon Dec 30 03:45:28 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p2  ONLINE       0     0     0
        ada1p2  ONLINE       0     0     0

errors: No known data errors

in netdata all disk have busy at 40-60%, but da8 graph are empty (because freenas mark him as failed).
i do

Code:

zpool clear

And lost connection to freenas xD

Ok it started resilver again from 0%

Code:

root@RSONas[/nonexistent]# zpool status
  pool: DANE
state: ONLINE
status: One or more devices is currently being resilvered.  The pool will
    continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jan  5 17:09:31 2020
    746G scanned at 3.26G/s, 188G issued at 843M/s, 14.5T total
    4.73G resilvered, 1.27% done, 0 days 04:57:40 to go
config:

    NAME                                                STATE     READ WRITE CKSUM
    DANE                                                ONLINE       0     0     0
      raidz1-0                                          ONLINE       0     0     0
        gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/f924c103-2bfb-11ea-9473-0cc47a2049d8.eli  ONLINE       0     0    25
        gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0
        gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:28 with 0 errors on Mon Dec 30 03:45:28 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p2  ONLINE       0     0     0
        ada1p2  ONLINE       0     0     0

errors: No known data errors

About netdata da8 has busy at 100% but i think because it is resilvered

For me this is a big problem because i changed faulty drive to other one and change drive slot for sure, and still have the same fault :/

 

[Apollo]

Being busy at 100% is fine when you have high throughput. The problem becomes from having a consistent low throughput of about 20MB/s during the entire time.

 

[reks]

Guys, please look at it..

Code:

root@RSONas[/nonexistent]# zpool status -v
  pool: DANE
state: DEGRADED
status: One or more devices is currently being resilvered.  The pool will
        continue to function, possibly in a degraded state.
action: Wait for the resilver to complete.
  scan: resilver in progress since Sun Jan  5 17:09:31 2020
        11.4T scanned at 73.6M/s, 9.80T issued at 63.5M/s, 14.5T total
        979G resilvered, 67.40% done, 0 days 21:44:36 to go
config:

        NAME                                                STATE     READ WRITE CKSUM
        DANE                                                DEGRADED     0     0   979
          raidz1-0                                          DEGRADED     0     0 1.92K
            gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/f924c103-2bfb-11ea-9473-0cc47a2049d8.eli  ONLINE       0     0    25
            gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0 1.95K  too many errors

errors: Permanent errors have been detected in the following files:

        DANE/rso.backup:<0x1>
        DANE/RSOnline@auto-20191224.0322-4d:/Biuro/arasz ogolne(/asus arasz/a/Desktop/pobrane/Adobe Photoshop CC 2015 (20150529.r.88) (32+64Bit) + Crack/Photoshop                                                   16 LS20 (64-Bit)/Adobe CC 2015/payloads/AdobePhotoshop16-Core_x64/Assets1_1.zip
        DANE/RSOnline@auto-20191224.0322-4d:/Kopie-klientow/Z_23_2014_07-KHAN/KHAN_kopia_clonedisk_compr.img.gz
        DANE/RSOnline@auto-20191224.0322-4d:/Aplikacje/p2p/sa/Yosemite for Mac Pro 1.1 and 2.1.dmg
        DANE/RSOnline@auto-20191224.0322-4d:/Kopie-klientow/piotr/Volume{674c1b49-8d79-11e9-99cd-7071bce4e562}.dsk
        DANE/RSOnline@auto-20191224.0322-4d:/Biuro/arasz ogolne(/asus arasz/a/Desktop/pobrane/WINDOWS 7 ALL IN ONE(PRE-ACTIVATED).ISO

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:32 with 0 errors on Tue Jan  7 03:45:32 2020
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0

errors: No known data errors

EDIT:
I attached log file with errors. If i good understand that i have all disk failed? :rotfl:

EDIT2:
On all HDD i have smart similar to this

Code:

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAGS    VALUE WORST THRESH FAIL RAW_VALUE
  1 Raw_Read_Error_Rate     POSR--   084   064   006    -    224176448
  3 Spin_Up_Time            PO----   097   096   000    -    0
  4 Start_Stop_Count        -O--CK   100   100   020    -    38
  5 Reallocated_Sector_Ct   PO--CK   100   100   010    -    0
  7 Seek_Error_Rate         POSR--   081   060   045    -    140676991
  9 Power_On_Hours          -O--CK   090   090   000    -    9631 (245 215 0)
10 Spin_Retry_Count        PO--C-   100   100   097    -    0
12 Power_Cycle_Count       -O--CK   100   100   020    -    38
183 Runtime_Bad_Block       -O--CK   100   100   000    -    0
184 End-to-End_Error        -O--CK   100   100   099    -    0
187 Reported_Uncorrect      -O--CK   100   100   000    -    0
188 Command_Timeout         -O--CK   100   099   000    -    0 0 4
189 High_Fly_Writes         -O-RCK   100   100   000    -    0
190 Airflow_Temperature_Cel -O---K   069   059   040    -    31 (Min/Max 30/33)
191 G-Sense_Error_Rate      -O--CK   100   100   000    -    0
192 Power-Off_Retract_Count -O--CK   100   100   000    -    407
193 Load_Cycle_Count        -O--CK   100   100   000    -    428
194 Temperature_Celsius     -O---K   031   041   000    -    31 (0 24 0 0 0)
195 Hardware_ECC_Recovered  -O-RC-   084   064   000    -    224176448
197 Current_Pending_Sector  -O--C-   100   100   000    -    0
198 Offline_Uncorrectable   ----C-   100   100   000    -    0
199 UDMA_CRC_Error_Count    -OSRCK   200   200   000    -    0
240 Head_Flying_Hours       ------   100   253   000    -    9618h+45m+09.180s
241 Total_LBAs_Written      ------   100   253   000    -    25516948680
242 Total_LBAs_Read         ------   100   253   000    -    31304020756

EDIT1234:

I think that problem isn't in any HDD, i need to check power supply and backplane

After resilver was finished i got

Code:

root@RSONas[/nonexistent]# zpool status
  pool: DANE
 state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: http://illumos.org/msg/ZFS-8000-8A
  scan: resilvered 1.48T in 2 days 08:01:00 with 1134 errors on Wed Jan  8 01:10:31 2020
config:

        NAME                                                STATE     READ WRITE CKSUM
        DANE                                                DEGRADED     0     0 1.11K
          raidz1-0                                          DEGRADED     0     0 2.23K
            gptid/93d41d5b-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/a47b9a1a-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/b40c512b-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/c5214407-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/d4f9893f-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/f924c103-2bfb-11ea-9473-0cc47a2049d8.eli  ONLINE       0     0    25
            gptid/f1bb3aac-87b3-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/004bd97a-87b4-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0     0  too many errors
            gptid/0df41821-87b4-11e9-bb8f-0cc47a2049d8.eli  DEGRADED     0     0 1.95K  too many errors

errors: 1134 data errors, use '-v' for a list

  pool: freenas-boot
 state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:32 with 0 errors on Tue Jan  7 03:45:32 2020
config:

        NAME        STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          mirror-0  ONLINE       0     0     0
            ada0p2  ONLINE       0     0     0
            ada1p2  ONLINE       0     0     0

errors: No known data errors

After all i was started scrubing the pool beacause all errors are only in old snapshot of this pool.

 

[reks]

SLOVED

After spending much time on it i was sloved this problem.
Some fact:

Code:

1 Raw_Read_Error_Rate     POSR--   084   064   006    -    224176448

This mean nothing. Because "224176448" mean 0, yes! it mean 0! google can explain ;)
I tested backplane, power supply, ram and all was OK
After formating all drive, and make raidz on fromated drive, all works fine. I was restored data from backup (16TB of data) and all works fine.

Conclusion:
Main problem was data corruption on the pool, not any hardware problem

Best regards,
Konrad

 

[no_connection]

Are you saying FreeNAS corrupted data on it's own? That sounds pretty serious if hardware is ok and data still get corrupted. I thought FreeNAS is suppose to prevent corruption, that is the reason of running ZFS in the first place.

 

[reks]

Yep that sound like that, but not by itself in 100%. This could be due to UPS failures and power loss by NAS, and it showed after 30 days when it do scrub. What is worse is that corruption has progressed and corrupt next files... any resilver cannot be succeed in my opinion (i tried 4 times with 4 hdd).
Now is fine, about 2 weeks no errors.

 

[no_connection]

Power loss should at most loose what had not been written to disk, never cause corruption of data already written, that is one of the features of cow and intent log. And corruption should never spread to non corrupt data. If true this is a serious issue for FreeNAS which under no circumstance should corrupt data, ever. In it's defense it did error out. Maybe it's just me being a bit edgy about things that is suppose to be safe.