SOLVED TrueNAS SCALE 21.06-BETA replacing drive issue

[KristijanL]

Hi,

i have been running personal storage from FreeNAS 9 up to TrueNAS Core latest version, then imported the same zfs pool into a fresh install of TrueNAS SCALE 21.06 BETA, i was enjoining the beta, no issues in the general usage ( iscsi shares, SMB shares, few users, few docker apps ), but when one of my drive failed i bought a new one and try to replace it, but i got weird errors.

The drive i am trying to replace is shown as SDD and there are a lot of errors of this type

Code:

blk_update_request: I/O error, dev sdd, sector 7814036920 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0

i have try to replace the new drive with another new drive, same issues, any suggestions?

the only thing that i could find is diffrent from other drives is block size

Code:

truenas# blockdev --getbsz /dev/sdd
4096
truenas# blockdev --getbsz /dev/sdd1
512
truenas# blockdev --getbsz /dev/sdd2
512

truenas# blockdev --getbsz /dev/sdg
4096
truenas# blockdev --getbsz /dev/sdg1
4096
truenas# blockdev --getbsz /dev/sdg2
4096

current status of my pool

Code:

    NAME                                      STATE     READ WRITE CKSUM
    vault                                     DEGRADED     0     0     0
      raidz2-0                                DEGRADED     0     0     0
        893ae457-7120-11eb-9ce5-000c2903bd67  ONLINE       0     0     0
        d3c182f5-7876-11eb-bb89-000c2903bd67  ONLINE       0     0     0
        eeb35ab5-e897-11e5-971d-0026558310c6  ONLINE       0     0     0
        504962f6-188b-11e9-a15d-000c2973b683  ONLINE       0     0     0
        0ca0a654-6b22-11eb-a445-000c2903bd67  ONLINE       0     0     0
        4581ec09-1554-4a61-8309-5226187393e6  FAULTED      0   155     0  too many errors  (resilvering)
    logs
      mirror-1                                ONLINE       0     0     0
        9db4f4d0-189c-11e9-a15d-000c2973b683  ONLINE       0     0     0
        a1e2a28f-189c-11e9-a15d-000c2973b683  ONLINE       0     0     0
    cache
      a45001f9-189c-11e9-a15d-000c2973b683    ONLINE       0     0     0
      a6b4d552-189c-11e9-a15d-000c2973b683    ONLINE       0     0     0

dmesg logs

Code:

[1445357.177300]  end_device-3:8: add: handle(0x000a), sas_addr(0x4433221101000000)
[1445357.181921] sd 3:0:8:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[1445357.183837] sd 3:0:8:0: [sdd] 4096-byte physical blocks
[1445357.190670] sd 3:0:8:0: [sdd] Write Protect is off
[1445357.192464] sd 3:0:8:0: [sdd] Mode Sense: 7f 00 10 08
[1445357.194117] sd 3:0:8:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[1445357.274387]  sdd: sdd1
[1445357.284463] sd 3:0:8:0: [sdd] Attached SCSI disk
[1445404.929893]  sdd: sdd1
[1445415.885731]  sdd: sdd1
[1445421.140129]  sdd: sdd1 sdd2
[1455131.570902] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1455131.570968] sd 3:0:8:0: [sdd] tag#2688 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s
[1455131.572823] sd 3:0:8:0: [sdd] tag#2751 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=6s
[1455131.576075] sd 3:0:8:0: [sdd] tag#2688 CDB: Write(16) 8a 00 00 00 00 00 cc bb 97 c0 00 00 00 80 00 00
[1455131.579440] sd 3:0:8:0: [sdd] tag#2751 Sense Key : Illegal Request [current]
[1455131.579449] sd 3:0:8:0: [sdd] tag#2751 Add. Sense: Logical block address out of range
[1455131.582882] blk_update_request: I/O error, dev sdd, sector 3434846144 op 0x1:(WRITE) flags 0x700 phys_seg 16 prio class 0
[1455131.584753] sd 3:0:8:0: [sdd] tag#2751 CDB: Write(16) 8a 00 00 00 00 00 cc bb 97 80 00 00 00 38 00 00
[1455131.586607] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=5 type=2 offset=1756493676544 size=65536 flags=40080caa
[1455131.590137] blk_update_request: critical target error, dev sdd, sector 3434846080 op 0x1:(WRITE) flags 0x700 phys_seg 7 prio class 0
[1455131.601170] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=121 type=2 offset=1756493643776 size=28672 flags=40080caa
[1455139.221154] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1455139.223252] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1455139.225214] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1455139.227243] sd 3:0:8:0: [sdd] tag#2724 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[1455139.230915] sd 3:0:8:0: [sdd] tag#2724 Sense Key : Illegal Request [current]
[1455139.232789] sd 3:0:8:0: [sdd] tag#2724 Add. Sense: Logical block address out of range
[1455139.234717] sd 3:0:8:0: [sdd] tag#2724 CDB: Write(16) 8a 00 00 00 00 00 cc bb 98 48 00 00 00 38 00 00
[1455139.238288] blk_update_request: critical target error, dev sdd, sector 3434846280 op 0x1:(WRITE) flags 0x700 phys_seg 7 prio class 0
[1455139.242096] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=121 type=2 offset=1756493746176 size=28672 flags=40080caa
[1455149.832535] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1455149.832563] sd 3:0:8:0: [sdd] tag#2737 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
[1455149.834679] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1455149.838717] sd 3:0:8:0: [sdd] tag#2737 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 bb b8 00 00 00 08 00 00
[1455149.845009] blk_update_request: I/O error, dev sdd, sector 7814036408 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1455149.849114] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=5 type=2 offset=3998639091712 size=4096 flags=180ac0
[1455149.853399] sd 3:0:8:0: [sdd] tag#2738 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
[1455149.857470] sd 3:0:8:0: [sdd] tag#2738 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 bd b8 00 00 00 08 00 00
[1455149.861619] blk_update_request: I/O error, dev sdd, sector 7814036920 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1455149.865828] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=5 type=2 offset=3998639353856 size=4096 flags=180ac0
[1455149.870252] sd 3:0:8:0: [sdd] tag#2736 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[1455149.874416] sd 3:0:8:0: [sdd] tag#2736 Sense Key : Illegal Request [current]
[1455149.876627] sd 3:0:8:0: [sdd] tag#2736 Add. Sense: Logical block address out of range
[1455149.878843] sd 3:0:8:0: [sdd] tag#2736 CDB: Write(16) 8a 00 00 00 00 00 00 40 03 b8 00 00 00 08 00 00
[1455149.882866] blk_update_request: critical target error, dev sdd, sector 4195256 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1455149.887150] zio pool=vault vdev=/dev/disk/by-partuuid/5557e7ba-cc2a-4506-ab21-5a78c09a1e71 error=121 type=2 offset=421888 size=4096 flags=180ac0
[1456001.139496] INFO: task txg_sync:4797 blocked for more than 120 seconds.
[1456001.141781]       Tainted: P           OE     5.10.42+truenas #1
[1456001.144031] "echo 0 > /proc/sys/kernel/hung_task_timeout_secs" disables this message.
[1456001.146292] task:txg_sync        state:D stack:    0 pid: 4797 ppid:     2 flags:0x00004000
[1456001.148651] Call Trace:
[1456001.150675]  __schedule+0x282/0x870
[1456001.152746]  schedule+0x46/0xb0
[1456001.154708]  cv_wait_common+0x14e/0x290 [spl]
[1456001.156730]  ? add_wait_queue_exclusive+0x70/0x70
[1456001.158868]  arc_read+0x1d0/0x19d0 [zfs]
[1456001.160937]  ? arc_can_share+0x90/0x90 [zfs]
[1456001.162823]  ? __kmalloc_node+0x141/0x2b0
[1456001.164727]  ? spl_kmem_alloc_impl+0x75/0xc0 [spl]
[1456001.166708]  dsl_scan_visitbp+0x4ff/0xd60 [zfs]
[1456001.168578]  ? kfree+0xc3/0x480
[1456001.170438]  ? dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.172371]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.174221]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.176069]  dsl_scan_visitbp+0x7a9/0xd60 [zfs]
[1456001.177810]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.179563]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.181257]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.182885]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.184502]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.186002]  dsl_scan_visitbp+0x591/0xd60 [zfs]
[1456001.187509]  dsl_scan_visitbp+0x3a6/0xd60 [zfs]
[1456001.188905]  dsl_scan_visit_rootbp+0xf5/0x1e0 [zfs]
[1456001.190279]  dsl_scan_visitds+0x140/0x450 [zfs]
[1456001.191513]  ? spl_kmem_alloc_impl+0x75/0xc0 [spl]
[1456001.192700]  ? sched_clock+0x5/0x10
[1456001.193795]  ? sched_clock_cpu+0xc/0xa0
[1456001.194904]  ? __smp_call_single_queue+0x23/0x40
[1456001.196051]  ? x2apic_send_IPI+0x49/0x50
[1456001.197083]  ? __kmalloc_node+0x141/0x2b0
[1456001.198080]  ? spl_kmem_alloc_impl+0x75/0xc0 [spl]
[1456001.199102]  ? __raw_spin_unlock+0x5/0x10 [spl]
[1456001.200128]  ? tsd_hash_search+0x75/0xa0 [spl]
[1456001.201067]  ? __list_add+0x12/0x30 [spl]
[1456001.201965]  ? __raw_spin_unlock+0x5/0x10 [spl]
[1456001.202866]  ? tsd_hash_add+0x14a/0x200 [spl]
[1456001.203931]  dsl_scan_visit+0x1b4/0x340 [zfs]
[1456001.204912]  dsl_scan_sync+0x5ed/0xb20 [zfs]
[1456001.205891]  ? spa_sync_iterate_to_convergence+0x12e/0x310 [zfs]
[1456001.206982]  spa_sync_iterate_to_convergence+0x139/0x310 [zfs]
[1456001.208117]  spa_sync+0x2e6/0x860 [zfs]
[1456001.209121]  txg_sync_thread+0x2ef/0x440 [zfs]
[1456001.210155]  ? txg_dispatch_callbacks+0xf0/0xf0 [zfs]
[1456001.211063]  thread_generic_wrapper+0x78/0xb0 [spl]
[1456001.212025]  ? IS_ERR+0x10/0x10 [spl]
[1456001.212872]  kthread+0x11b/0x140
[1456001.213684]  ? __kthread_bind_mask+0x60/0x60
[1456001.214551]  ret_from_fork+0x22/0x30
[1528186.139816] sd 3:0:8:0: device_block, handle(0x000a)
[1528187.639883] sd 3:0:8:0: device_unblock and setting to running, handle(0x000a)
[1528187.644797] sd 3:0:8:0: [sdd] Synchronizing SCSI cache
[1528187.645856] sd 3:0:8:0: [sdd] Synchronize Cache(10) failed: Result: hostbyte=DID_NO_CONNECT driverbyte=DRIVER_OK
[1528187.712981] mpt2sas_cm0: mpt3sas_transport_port_remove: removed: sas_addr(0x4433221101000000)
[1528187.714327] mpt2sas_cm0: removing handle(0x000a), sas_addr(0x4433221101000000)
[1528187.715575] mpt2sas_cm0: enclosure logical id(0x5003048000000001), slot(2)
[1528335.641931] scsi 3:0:9:0: Direct-Access     ATA      WDC WD40EFAX-68J 0A83 PQ: 0 ANSI: 6
[1528335.643444] scsi 3:0:9:0: SATA: handle(0x000a), sas_addr(0x4433221101000000), phy(1), device_name(0x0000000000000000)
[1528335.645991] scsi 3:0:9:0: enclosure logical id (0x5003048000000001), slot(2)
[1528335.647741] scsi 3:0:9:0: atapi(n), ncq(y), asyn_notify(n), smart(y), fua(y), sw_preserve(y)
[1528335.649249] scsi 3:0:9:0: qdepth(32), tagged(1), scsi_level(7), cmd_que(1)
[1528335.653608] sd 3:0:9:0: Attached scsi generic sg3 type 0
[1528335.653677] sd 3:0:9:0: Power-on or device reset occurred
[1528335.655191]  end_device-3:9: add: handle(0x000a), sas_addr(0x4433221101000000)
[1528335.659191] sd 3:0:9:0: [sdd] 7814037168 512-byte logical blocks: (4.00 TB/3.64 TiB)
[1528335.660747] sd 3:0:9:0: [sdd] 4096-byte physical blocks
[1528335.665504] sd 3:0:9:0: [sdd] Write Protect is off
[1528335.666954] sd 3:0:9:0: [sdd] Mode Sense: 7f 00 10 08
[1528335.667971] sd 3:0:9:0: [sdd] Write cache: enabled, read cache: enabled, supports DPO and FUA
[1528335.749868]  sdd: sdd1
[1528335.759338] sd 3:0:9:0: [sdd] Attached SCSI disk
[1528380.440457]  sdd: sdd1
[1528391.413781]  sdd: sdd1
[1528391.509664]  sdd: sdd1
[1528391.522038]  sdd: sdd1 sdd2
[1528396.516379]  sdd: sdd1 sdd2
[1535906.783268] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535906.783295] sd 3:0:9:0: [sdd] tag#2124 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
[1535906.784959] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535906.787910] sd 3:0:9:0: [sdd] tag#2124 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 bc 30 00 00 00 08 00 00
[1535906.792653] blk_update_request: I/O error, dev sdd, sector 7814036528 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1535906.795948] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=3998639153152 size=4096 flags=180ac0
[1535906.799715] sd 3:0:9:0: [sdd] tag#2125 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
[1535906.803195] sd 3:0:9:0: [sdd] tag#2125 CDB: Write(16) 8a 00 00 00 00 01 d1 c0 be 30 00 00 00 08 00 00
[1535906.806624] blk_update_request: I/O error, dev sdd, sector 7814037040 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1535906.810148] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=3998639415296 size=4096 flags=180ac0
[1535906.813890] sd 3:0:9:0: [sdd] tag#2123 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[1535906.817463] sd 3:0:9:0: [sdd] tag#2123 Sense Key : Illegal Request [current]
[1535906.819404] sd 3:0:9:0: [sdd] tag#2123 Add. Sense: Logical block address out of range
[1535906.821318] sd 3:0:9:0: [sdd] tag#2123 CDB: Write(16) 8a 00 00 00 00 00 00 40 04 30 00 00 00 08 00 00
[1535906.824910] blk_update_request: critical target error, dev sdd, sector 4195376 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1535906.828671] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=483328 size=4096 flags=180ac0
[1535922.556636] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535922.558883] sd 3:0:9:0: [sdd] tag#2117 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=8s
[1535922.558910] sd 3:0:9:0: [sdd] tag#2135 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s
[1535922.562831] sd 3:0:9:0: [sdd] tag#2117 Sense Key : Illegal Request [current]
[1535922.566913] sd 3:0:9:0: [sdd] tag#2135 CDB: Write(16) 8a 00 00 00 00 00 24 56 47 30 00 00 01 38 00 00
[1535922.569137] sd 3:0:9:0: [sdd] tag#2117 Add. Sense: Logical block address out of range
[1535922.573356] blk_update_request: I/O error, dev sdd, sector 609634096 op 0x1:(WRITE) flags 0x700 phys_seg 39 prio class 0
[1535922.575661] sd 3:0:9:0: [sdd] tag#2117 CDB: Write(16) 8a 00 00 00 00 00 24 56 44 08 00 00 02 88 00 00
[1535922.580070] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=309985107968 size=159744 flags=40080caa
[1535922.584294] blk_update_request: critical target error, dev sdd, sector 609633288 op 0x1:(WRITE) flags 0x700 phys_seg 81 prio class 0
[1535922.593171] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=309984694272 size=331776 flags=40080caa
[1535930.133065] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535930.133144] sd 3:0:9:0: [sdd] tag#2127 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=7s
[1535930.135469] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535930.139787] sd 3:0:9:0: [sdd] tag#2127 CDB: Write(16) 8a 00 00 00 00 00 24 56 49 98 00 00 00 78 00 00
[1535930.142093] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535930.142097] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535930.150909] blk_update_request: I/O error, dev sdd, sector 609634712 op 0x1:(WRITE) flags 0x700 phys_seg 15 prio class 0
[1535930.155234] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=309985423360 size=61440 flags=40080caa
[1535930.159836] sd 3:0:9:0: [sdd] tag#2125 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[1535930.164216] sd 3:0:9:0: [sdd] tag#2125 Sense Key : Illegal Request [current]
[1535930.166509] sd 3:0:9:0: [sdd] tag#2125 Add. Sense: Logical block address out of range
[1535930.168748] sd 3:0:9:0: [sdd] tag#2125 CDB: Write(16) 8a 00 00 00 00 00 24 53 27 10 00 00 00 08 00 00
[1535930.172938] blk_update_request: critical target error, dev sdd, sector 609429264 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1535930.177261] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=309880233984 size=4096 flags=1808aa
[1535952.457084] mpt2sas_cm0: log_info(0x31080000): originator(PL), code(0x08), sub_code(0x0000)
[1535952.457163] sd 3:0:9:0: [sdd] tag#2158 FAILED Result: hostbyte=DID_SOFT_ERROR driverbyte=DRIVER_OK cmd_age=6s
[1535952.459489] sd 3:0:9:0: [sdd] tag#2142 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=9s
[1535952.463692] sd 3:0:9:0: [sdd] tag#2158 CDB: Write(16) 8a 00 00 00 00 00 24 5b c2 48 00 00 00 08 00 00
[1535952.467952] sd 3:0:9:0: [sdd] tag#2142 Sense Key : Illegal Request [current]
[1535952.472123] blk_update_request: I/O error, dev sdd, sector 609993288 op 0x1:(WRITE) flags 0x700 phys_seg 1 prio class 0
[1535952.474365] sd 3:0:9:0: [sdd] tag#2142 Add. Sense: Logical block address out of range
[1535952.478602] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=5 type=2 offset=310169014272 size=4096 flags=1808aa
[1535952.480857] sd 3:0:9:0: [sdd] tag#2142 CDB: Write(16) 8a 00 00 00 00 00 24 5b c1 88 00 00 00 38 00 00
[1535952.489342] blk_update_request: critical target error, dev sdd, sector 609993096 op 0x1:(WRITE) flags 0x700 phys_seg 7 prio class 0
[1535952.493633] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=310168915968 size=28672 flags=40080caa
[1535960.088166] sd 3:0:9:0: [sdd] tag#2162 FAILED Result: hostbyte=DID_OK driverbyte=DRIVER_SENSE cmd_age=7s
[1535960.092453] sd 3:0:9:0: [sdd] tag#2162 Sense Key : Illegal Request [current]
[1535960.094758] sd 3:0:9:0: [sdd] tag#2162 Add. Sense: Logical block address out of range
[1535960.097023] sd 3:0:9:0: [sdd] tag#2162 CDB: Write(16) 8a 00 00 00 00 00 24 5b c2 e0 00 00 00 18 00 00
[1535960.101229] blk_update_request: critical target error, dev sdd, sector 609993440 op 0x1:(WRITE) flags 0x700 phys_seg 3 prio class 0
[1535960.105555] zio pool=vault vdev=/dev/disk/by-partuuid/4581ec09-1554-4a61-8309-5226187393e6 error=121 type=2 offset=310169092096 size=12288 flags=40080caa

 

[Kris Moore]

Those write errors seem legit as an indicator something is funky with the HW. Don't think block-sizes would matter in this case.

Did you replace the cable as well? Can you plug it into another port?

 

[KristijanL]

I am using Dell r720, i doubt that the backplate have issues, as i swap two new disks with same errors.

I will try to boot old TrueNAS Core and see what it does.

 

[impovich]

I replaced all SMR disks with CMR on scale with zero issues. just sharing my experience

 

[KristijanL]

OMG i have just found that they sold me SMR drives and not CMR as i have ordered, no wonder i have problems!
Just to clarified, there are not issues with TrueNAS SCALE with replacing drives, just my personal bad luck.