Degraded Pool

lotustechie · Oct 30, 2020

I was starting to run out of space, so I decided to upgrade my HDDs. Before I had 8 4TB drives. I installed new 12TB drives one at a time until they were resilvered and now I am getting degraded errors on 3 of the 4 new drives. I am currently running a scrub to see if that will resolve the errors. I ran a zpool status and got the following information:

Code:

root@airiam[/]# zpool status -v
  pool: Data
state: DEGRADED
status: One or more devices has experienced an error resulting in data
        corruption.  Applications may be affected.
action: Restore the file in question if possible.  Otherwise restore the
        entire pool from backup.
   see: https://openzfs.github.io/openzfs-docs/msg/ZFS-8000-8A
  scan: scrub in progress since Fri Oct 30 13:58:07 2020
        3.00T scanned at 5.82G/s, 264G issued at 511M/s, 26.3T total
        0B repaired, 0.98% done, 14:49:53 to go
config:

        NAME                                            STATE     READ WRITE CKSUM
        Data                                            DEGRADED     0     0     0
          raidz1-0                                      ONLINE       0     0     0
            gptid/7be29992-a674-11ea-a1ae-a8a159238732  ONLINE       0     0     0
            gptid/81431cc2-a674-11ea-a1ae-a8a159238732  ONLINE       0     0     0
            gptid/845353ab-a674-11ea-a1ae-a8a159238732  ONLINE       0     0     0
            gptid/85f57ef6-a674-11ea-a1ae-a8a159238732  ONLINE       0     0     0
          raidz1-1                                      DEGRADED     0     0     0
            gptid/465eafc7-1a4f-11eb-a086-a8a159238732  DEGRADED     0     0   333  too many errors
            gptid/9772beeb-1a9c-11eb-be62-a8a159238732  ONLINE       0     0   724
            gptid/b48bd088-1995-11eb-b3e4-a8a159238732  DEGRADED     0     0   312  too many errors
            gptid/b00d0686-1935-11eb-92a3-a8a159238732  DEGRADED     0     0   302  too many errors

errors: Permanent errors have been detected in the following files:

        Data/iocage/jails/LT-PMS01/root@ioc_update_11.3-RELEASE-p11_2020-09-24_12-09-16:/Plex Media Server/Metadata/Movies/0/848f6d7b66011d0e9f96767834689bfa8e7b05d.bundle/Contents/_stored/posters/com.plexapp.agents.imdb_456e3b4ece52f98fdff17bcd3be45fd3aa7b6e9e
        /mnt/Data/Media/Movies/The Assistant (2020)/The Assistant (2020).mp4
        /mnt/Data/Media/Movies/House Party 2 (1991)/House Party 2 (1991).mp4
        /mnt/Data/Media/Movies/Tape (2020)/Tape (2020).mp4
        /var/db/system/samba4/winbindd_cache.tdb
        /var/db/system/samba4/registry.tdb
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/auth.log
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/console.log
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/daemon.log
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/middlewared.log
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/utx.log
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/samba4/log.winbindd
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/messages
        /var/db/system/syslog-445a1355f3314ba4946bd845d9821e43/log/debug.log
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-dbuf_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_latency-da2.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da7/pending_operations.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_latency-da3.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops-da0.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops-da1.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops-da3.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-download-11.3-RELEASE/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops_rwd-da1.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops_rwd-da2.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops_rwd-da3.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases-11.3-RELEASE/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_queue-da0.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_queue-da1.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_queue-da2.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_queue-da3.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases-11.3-RELEASE-root/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases-11.3-RELEASE-root/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/interface-bridge0/if_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-jails-LT-PMS01/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/interface-re0/if_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-jails-LT-PMS01-root/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/interface-vnet0.1/if_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/interface-vnet0.1/if_packets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/load/load.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/memory/memory-active.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/memory/memory-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/memory/memory-laundry.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/memory/memory-wired.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da5/disk_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da5/disk_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da5/disk_io_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da6/disk_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da6/disk_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_queue-da4.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops-da4.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops_rwd-da4.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_latency-da4.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_queue-da5.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops-da5.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops_rwd-da5.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_latency-da5.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_busy_percent-da5.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da7/disk_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/aggregation-cpu-average/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/processes/ps_state-running.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da7/disk_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/processes/ps_state-sleeping.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da7/disk_io_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/aggregation-cpu-average/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_queue-da6.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops-da6.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/aggregation-cpu-sum/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/processes/ps_state-zombies.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/aggregation-cpu-sum/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_bw-da6.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/aggregation-cpu-sum/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_latency-da6.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/aggregation-cpu-sum/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/aggregation-cpu-sum/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/rrdcached/gauge-tree_nodes.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-0/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops-da7.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/rrdcached/operations-receive-update.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_ops_rwd-da7.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-0/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/rrdcached/operations-write-data_sets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-0/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/rrdcached/operations-write-updates.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_busy_percent-da7.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/swap/swap-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-1/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-1/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/uptime/uptime.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_eviction-eligible.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-10/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_eviction-ineligible.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-10/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_operation-deleted.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disktemp-da7/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_ratio-arc.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-11/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-demand_data-miss.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-11/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-demand_metadata-hit.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-jails-radarr/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-11/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-mfu-hit.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-jails-radarr-root/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-2/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-mfu_ghost-hit.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-mru-hit.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-jails-OpenVPN/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-2/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-prefetch_data-miss.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-3/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-prefetch_metadata-hit.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-3/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_result-prefetch_metadata-miss.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-3/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-jails-OpenVPN-root/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-3/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-anon_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-3/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-arc.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-4/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-c.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-4/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-c_max.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-4/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-c_min.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-4/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-4/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-5/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-mfu_ghost_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-5/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-5/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-mru_ghost_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-5/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-mru_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-5/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-6/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-p.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-6/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/hash_collisions.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da6/pending_operations.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-6/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-6/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-6/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/mutex_operations-miss.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-7/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/arcstat_ratio_arc-hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-7/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-7/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-download-11.4-RELEASE/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-7/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/arcstat_ratio_arc-misses.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-7/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/arcstat_ratio_data-demand_data_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases-11.4-RELEASE/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-8/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-8/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-8/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases-11.4-RELEASE/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-8/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/arcstat_ratio_metadata-demand_metadata_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases-11.4-RELEASE-root/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-8/cpu-user.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-9/cpu-idle.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/arcstat_ratio_metadata-prefetch_metadata_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-9/cpu-interrupt.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-9/cpu-nice.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/arcstat_ratio_mu-mfu_ghost_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cpu-9/cpu-system.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/arcstat_ratio_mu-mru_ghost_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-0/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-10/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-2/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-3/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-4/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_arcmeta-arc_meta_used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-6/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-7/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_counts-deleted.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-8/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/cputemp-9/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_cp-c_max.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_cp-c_min.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_demand-demand_data_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_demand-demand_data_misses.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_evict-evict_skip.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_hash-hash_collisions.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_hits_misses-hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_hits_misses-misses.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-dnode_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc/cache_size-bonus_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da0/pending_operations.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da1/pending_operations.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da4/pending_operations.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da5/pending_operations.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-root/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_mru-mfu_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_mru-mru_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da0/disk_io_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_prefetch-prefetch_data_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da0/disk_ops.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_prefetch-prefetch_metadata_hits.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da0/disk_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_prefetch-prefetch_metadata_misses.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da1/disk_io_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_size-data_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_size-hdr_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da1/disk_ops.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_size-other_size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-download-12.1-RELEASE/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/zfs_arc_v2/gauge_arcstats_raw_size-size.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da2/disk_io_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da2/disk_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da2/disk_ops.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da3/disk_io_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da3/disk_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da3/disk_ops.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-releases-12.1-RELEASE-root/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-da3/disk_time.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disk-nvd0/disk_octets.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/disktemp-da2/temperature.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-download/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_busy_percent-da0.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_busy_percent-da1.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-images/df_complex-used.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_busy_percent-nvd0.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/df-mnt-Data-iocage-jails/df_complex-free.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_bw-da0.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_bw-da1.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_bw-da2.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_bw-da3.rrd
        /var/db/system/rrd-445a1355f3314ba4946bd845d9821e43/localhost/geom_stat/geom_latency-da0.rrd
        Data/iocage:<0x0>
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/3c/7759a20d368a7d3f8c2e2568844b41aa87a28c
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/fc/a061815626506dd42bfe8f33ec37d15681d5c0
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/ce/69cfc3c965e1609159898958ca11fb1b3270be
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/8f/4bf8c44731af61b89012ba044b63d73effcc8c
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/4e/160df40d0044b64af2558529d313856703a1d7
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/objects/9b/3c4e5e75045333a07de6d5c3595d058b68b447
        /mnt/Data/iocage/.plugins/github_com_freenas_iocage-ix-plugins_git/.git/index
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/INDEX
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/icons/rslsync.png
        /mnt/Data/iocage/.plugins/github_com_ix-plugin-hub_iocage-plugin-index_git/.git/logs/refs/remotes/origin/master
        /mnt/Data/iocage/jails/LT-PMS01/config.json
        Data/iocage/jails/LT-PMS01/root:<0x0>
        /mnt/Data/iocage/jails/LT-PMS01/root/var/log/messages
        /mnt/Data/iocage/jails/LT-PMS01/root/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-wal
        /mnt/Data/iocage/jails/LT-PMS01/root/Plex Media Server/Plug-in Support/Databases/com.plexapp.plugins.library.db-shm

  pool: freenas-boot
state: ONLINE
status: Some supported features are not enabled on the pool. The pool can
        still be used, but some features are unavailable.
action: Enable all features using 'zpool upgrade'. Once this is done,
        the pool may no longer be accessible by software that does not support
        the features. See zpool-features(5) for details.
  scan: scrub repaired 0B in 00:00:10 with 0 errors on Thu Oct 29 03:45:10 2020
config:

        NAME          STATE     READ WRITE CKSUM
        freenas-boot  ONLINE       0     0     0
          nvd0p2      ONLINE       0     0     0

errors: No known data errors

lotustechie · Oct 30, 2020

I ran a short SMART test on one of the drives and got the following information:

Code:

root@airiam[/]# smartctl /dev/da7 -a
smartctl 7.1 2019-12-30 r5022 [FreeBSD 12.2-RC3 amd64] (local build)
Copyright (C) 2002-19, Bruce Allen, Christian Franke, www.smartmontools.org

=== START OF INFORMATION SECTION ===
Model Family:     Seagate IronWolf
Device Model:     ST12000VN0008-2JH101
Serial Number:    ZHZ387N7
LU WWN Device Id: 5 000c50 0c61fcd4f
Firmware Version: SC60
User Capacity:    12,000,138,625,024 bytes [12.0 TB]
Sector Sizes:     512 bytes logical, 4096 bytes physical
Rotation Rate:    7200 rpm
Form Factor:      3.5 inches
Device is:        In smartctl database [for details use: -P show]
ATA Version is:   ACS-4 (minor revision not indicated)
SATA Version is:  SATA 3.3, 6.0 Gb/s (current: 6.0 Gb/s)
Local Time is:    Fri Oct 30 14:16:51 2020 EDT
SMART support is: Available - device has SMART capability.
SMART support is: Enabled

=== START OF READ SMART DATA SECTION ===
SMART overall-health self-assessment test result: PASSED

General SMART Values:
Offline data collection status:  (0x82) Offline data collection activity
                                        was completed without error.
                                        Auto Offline Data Collection: Enabled.
Self-test execution status:      (   0) The previous self-test routine completed
                                        without error or no self-test has ever
                                        been run.
Total time to complete Offline
data collection:                (  567) seconds.
Offline data collection
capabilities:                    (0x7b) SMART execute Offline immediate.
                                        Auto Offline data collection on/off support.
                                        Suspend Offline collection upon new
                                        command.
                                        Offline surface scan supported.
                                        Self-test supported.
                                        Conveyance Self-test supported.
                                        Selective Self-test supported.
SMART capabilities:            (0x0003) Saves SMART data before entering
                                        power-saving mode.
                                        Supports SMART auto save timer.
Error logging capability:        (0x01) Error logging supported.
                                        General Purpose Logging supported.
Short self-test routine
recommended polling time:        (   1) minutes.
Extended self-test routine
recommended polling time:        (1178) minutes.
Conveyance self-test routine
recommended polling time:        (   2) minutes.
SCT capabilities:              (0x50bd) SCT Status supported.
                                        SCT Error Recovery Control supported.
                                        SCT Feature Control supported.
                                        SCT Data Table supported.

SMART Attributes Data Structure revision number: 10
Vendor Specific SMART Attributes with Thresholds:
ID# ATTRIBUTE_NAME          FLAG     VALUE WORST THRESH TYPE      UPDATED  WHEN_FAILED RAW_VALUE
  1 Raw_Read_Error_Rate     0x000f   082   069   044    Pre-fail  Always       -       163376136
  3 Spin_Up_Time            0x0003   087   087   000    Pre-fail  Always       -       0
  4 Start_Stop_Count        0x0032   100   100   020    Old_age   Always       -       17
  5 Reallocated_Sector_Ct   0x0033   100   100   010    Pre-fail  Always       -       0
  7 Seek_Error_Rate         0x000f   068   060   045    Pre-fail  Always       -       6510396
  9 Power_On_Hours          0x0032   100   100   000    Old_age   Always       -       19
 10 Spin_Retry_Count        0x0013   100   100   097    Pre-fail  Always       -       0
 12 Power_Cycle_Count       0x0032   100   100   020    Old_age   Always       -       17
 18 Unknown_Attribute       0x000b   100   100   050    Pre-fail  Always       -       0
187 Reported_Uncorrect      0x0032   100   100   000    Old_age   Always       -       0
188 Command_Timeout         0x0032   100   100   000    Old_age   Always       -       0
190 Airflow_Temperature_Cel 0x0022   062   060   040    Old_age   Always       -       38 (Min/Max 36/39)
192 Power-Off_Retract_Count 0x0032   100   100   000    Old_age   Always       -       5
193 Load_Cycle_Count        0x0032   100   100   000    Old_age   Always       -       29
194 Temperature_Celsius     0x0022   038   040   000    Old_age   Always       -       38 (0 20 0 0 0)
195 Hardware_ECC_Recovered  0x001a   082   069   000    Old_age   Always       -       163376136
197 Current_Pending_Sector  0x0012   100   100   000    Old_age   Always       -       0
198 Offline_Uncorrectable   0x0010   100   100   000    Old_age   Offline      -       0
199 UDMA_CRC_Error_Count    0x003e   200   200   000    Old_age   Always       -       0
200 Multi_Zone_Error_Rate   0x0023   100   100   001    Pre-fail  Always       -       0
240 Head_Flying_Hours       0x0000   100   253   000    Old_age   Offline      -       17 (154 162 0)
241 Total_LBAs_Written      0x0000   100   253   000    Old_age   Offline      -       6420980722
242 Total_LBAs_Read         0x0000   100   253   000    Old_age   Offline      -       6441475402

SMART Error Log Version: 1
No Errors Logged

SMART Self-test log structure revision number 1
Num  Test_Description    Status                  Remaining  LifeTime(hours)  LBA_of_first_error
# 1  Short offline       Completed without error       00%        19         -

SMART Selective self-test log data structure revision number 1
 SPAN  MIN_LBA  MAX_LBA  CURRENT_TEST_STATUS
    1        0        0  Not_testing
    2        0        0  Not_testing
    3        0        0  Not_testing
    4        0        0  Not_testing
    5        0        0  Not_testing
Selective self-test flags (0x0):
  After scanning selected spans, do NOT read-scan remainder of disk.
If Selective self-test is pending on power-up, resume after 0 minute delay.

I would appreciate any recommendations on next steps. I still have the old drives. Should I put them back in to fix the data?

lotustechie · Oct 30, 2020

The one thing that I found odd was that it looks like most of the errors are on the SSD, which I would think would be pretty bad.

Important Announcement for The TrueNAS Community.

Degraded Pool

lotustechie

Dabbler

lotustechie

Dabbler

lotustechie

Dabbler

Similar threads

Important Announcement for The TrueNAS Community.