Permanent errors have been detected after hardware repair.

[raigan21]

Hello, community,

here are my specs.

Storinator 45 bays

OS: TrueNAS 13
RAM: 32 GiB
Total Disks: 45
CPU: 2 X intel Xeon E5-2620
HBA: Rocket 750
Motherboard: SuperMicro X9DRL-3F

so the problem appear after we had a motherboard failure and I had to replace it with the same model, after that I ran a scrub and multiple permanent errors appeared in my main pool all of them pointing to the .system folder.

errors: Permanent errors have been detected in the following files:

VTS01Z3/.system/syslog-f36704f2fe794cb6a75657843255655d:/log/daemon.log
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disktemp-da33/temperature.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/nfsstat-server/nfsstat-access.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da11/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da12/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-ada2/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc/cache_size-hdr_size.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da14/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da26/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da2/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da27/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da17/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da29/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da29/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da18/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da6/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc/cache_result-prefetch_metadata-hit.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_busy_percent-ada0.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-ada0/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da11/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da6.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da14/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-ada3.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da15/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-ada4/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da24/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da0/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-ada3.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da1/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da28/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da33/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-mnt-VTS01Z3-Administration-VOLTAGE/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da5/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da36/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da37/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da30.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da10/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-da32.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da8/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da11/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da12/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da13/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da14/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da15/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da16/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da17/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da18/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da19/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da1.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da15/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da17/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da19/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da24/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-da7.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da25/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da26/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da20.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da25.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_latency-da4.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da32/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-mnt-VTS01Z3-Production-Nomads/df_complex-free.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc_v2/gauge_arcstats_raw_evict-evict_l2_cached.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da14.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_latency-da17.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-da30.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc_v2/gauge_arcstats_raw_hits_misses-misses.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops-da31.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc_v2/gauge_arcstats_raw_demand-demand_data_hits.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da25.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-var-db-system-cores/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da26.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da36.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-da38.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/memory/memory-inactive.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disktemp-da12/temperature.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da2/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disktemp-da30/temperature.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disktemp-da7/temperature.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da0/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc/cache_size-mfu_size.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da3/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-mnt-VTS01Z3/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da20/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da32/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da21/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-root/df_complex-free.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops-da1.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da24/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_busy_percent-da3.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da26/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-mnt-VTS01Z3-Products/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da8.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-mnt-VTS01Z3-prod/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da9.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_latency-ada4.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da12.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc_v2/gauge_arcstats_raw_size-data_size.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-ada2/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-ada5/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_busy_percent-da19.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da27.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/zfs_arc_v2/gauge_arcstats_raw_l2_compress-l2_compress_successes.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da9/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da11.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da35.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da11/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da39.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_latency-da25.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-da28.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_latency-da19.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_queue-da0.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-da34.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_busy_percent-da28.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disk-da40/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-var-db-system-webui/df_complex-reserved.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_bw-da38.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/rrdcached/gauge-tree_depth.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/geom_stat/geom_ops_rwd-da14.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/cpu-5/cpu-user.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-mnt-VTS01Z3-prod-cinque_terre/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/disktemp-ada3/temperature.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d:/localhost/df-mnt-VTS01Z3-prod-night_in_the_village/df_complex-free.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da11/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da12/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-ada2/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da14/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da26/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da2/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da27/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da17/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da29/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da29/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da18/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-ada0/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da11/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da14/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da15/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-ada4/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da24/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da0/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da1/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da28/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-mnt-VTS01Z3-Administration-VOLTAGE/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da10/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da11/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da12/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da13/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da14/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da15/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da16/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da17/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da18/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da19/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da15/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da17/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da19/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da24/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da25/disk_ops.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da26/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/geom_stat/geom_bw-da20.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/geom_stat/geom_latency-da4.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-mnt-VTS01Z3-Production-Nomads/df_complex-free.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-var-db-system-cores/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da2/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da0/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da3/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-mnt-VTS01Z3/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da20/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da21/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-root/df_complex-free.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da24/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da26/disk_io_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-mnt-VTS01Z3-Products/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-mnt-VTS01Z3-prod/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-ada2/disk_octets.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-ada5/disk_time.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/disk-da11/pending_operations.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-var-db-system-webui/df_complex-reserved.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/cpu-5/cpu-user.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-mnt-VTS01Z3-prod-cinque_terre/df_complex-used.rrd
VTS01Z3/.system/rrd-f36704f2fe794cb6a75657843255655d@auto-2023-05-24_12-00:/localhost/df-mnt-VTS01Z3-prod-night_in_the_village/df_complex-free.rrd

any tips, I'm really lost on what to do here

 

[jgreco]

HBA: Rocket 750

This is known not to be compatible with TrueNAS. Please see the following article:

What's all the noise about HBAs, and why can't I use a RAID controller?

1) An HBA is a Host Bus Adapter. This is a controller that allows SAS and SATA devices to be attached to, and communicate directly with, a server. RAID controllers typically aggregate several disks into a Virtual Disk abstraction of some sort...

www.truenas.com

The damage to your system dataset is probably not too terrible, but you're lucky it is limited to that.

 

[raigan21]

I may be wrong but is because FreeBSD may not have good support over this HBA, now if I change the system to Trunas scale that is not based on Free BSD you think it could work normally?

 

[jgreco]

No. It's a RAID controller, not an HBA, despite Highpoint's confusion about the term. The problem is that it uses a driver that does some stuff that isn't compatible with ZFS. My warning article was written predating TrueNAS SCALE, but it would be a mistake for you to think that somehow Linux is different. You really need to go read points 2), 4), 4a), 5), and 10) in the article I referred you to. Just because it says "FreeBSD" doesn't mean that it doesn't apply equally to Linux.

I lean not-batshit-crazy libertarian and respect your right to do whatever the hell you'd like to do, but I also encourage you to make such decisions from a position of being properly informed.

At one time, we had Cyberjock, an iXsystems staffer, who had bought one of these HighPoints, did a bunch of rounds of battle with it, trying to work with HighPoint on driver issues, and his final analysis is at the end of the following thread.

import fails after adding RocketRaid controller

Hi, I've been running FreeNAS for a while and I haven't had any issues. It's generally sweet. :-) I want to add more disks and my EVGA 790i mobo only has 6 sata ports. So I added a RocketRaid 1820A. It's just a controller - no hw RAID. If I export and move the drives over to the...

www.ixsystems.com

These things will work fine until one day for whatever reason it doesn't, and it stabs you in the back. It's an excellent example of where 99.999% is not quite good enough for ZFS.

 

[raigan21]

well if that is the case, nothing else I can do, anyways the server is more than 10 years old o think is time for a hardware upgrade.

 

[jgreco]

Be aware that you can inexpensively replace the HighPoint with an LSI 9207-8i or 9211-8i, or any of their OEM (Dell/IBM/Fujitsu) equivalents but then you have to crossflash them. These used to be findable for as little as $25 in quantity. Oh wow there's one for $19. They had been up around $40 for awhile. But it comes at the cost of your having to do the crossflashing yourself. If you want to avoid annoyance, buy one that is already flashed to IT mode. It will cost a bit more but is probably worth it unless you're a masochist.

The -8i card only gets you 8 SAS lanes and you've currently got more like 40(?) on that Rocket card. You can get more out of a SAS HBA using an SAS expander. These take an SFF-8087 as input, such as this Intel RES2SV240, which has 24 total ports, so you get 20 ports free for hooking up disks. You could hook two of those up to an -8i card (one on each SFF-8087) for a total of 40 ports. You do not need actual PCIe slots for the SAS expanders, they are just often added that way because PCIe provides a convenient power source. But there are also expanders with more or less SAS ports.