Disk busy but no heavy read or write

[oracle_sod]

I've been trying to work out whats going on here with my FreeNAS box. I have one drive (in this instance DA0) that ends up being 100% busy even when there is little to no utilization. A reboot and its back to being good for a while. I replaced the drive with a new drive and now the same thing is still happening with the new drive.

This setup is running on VMware with the RAID controller passed through (I have two boxes like this)

FreeNAS-11.3-U2
Motherboard: SuperMicro SYS-5028D-TN4T
CPU: 8 Core Intel(R) Xeon(R) CPU D-1541 @ 2.10GHz
RAM: 8GB DDR4 ECC
Raid-Z
Seagate 8TB DA0 ST8000DM004
Seagate 8TB DA1 ST8000VN004
Seagate 4TB DA2 ST4000DM004
Seagate 8TB DA3 ST8000VN004
VMware 16GB ADA0 VMDK on Toshiba-OCZ RD400 Series NVMe
Dell PERC H200 (IT Mode)

I'm trying to work out how to troubleshoot this, the drive is healthy, there is nothing in the DMESG log or the VMware logs or any of the other logs that i could see in /var/logs on FreeNAS

Could there was an issue with the H200 ? Where else could I look for reasoning behind this ?

 

[HoneyBadger]

I see two SMR drives (da0/da2) which are likely your culprits. What is the pool configuration (RAIDZ? Striped?) and is there any scrubbing or resilvering occurring right now?

I'm not sure if the Seagate SMR drives expose their delete behavior upstream but try gstat -dp and see if you notice a high volume of deletes per second or d/s

 

[oracle_sod]

Yes they are SMR devices, im in the process of upgrading however, the other box has all SMR devices and doesn't see this issue at all.. This is a 4 drive Raid-Z with a single parity.

this is the gstat -dp
dT: 1.047s w: 1.000s


L(q) ops/s r/s kBps ms/r w/s kBps ms/w d/s kBps ms/d %busy Name
0 0 0 0 0.0 0 0 0.0 0 0 0.0 0.0 | ada0
0 2 0 0 0.0 1 23 0.4 0 0 0.0 1.6. | da2
0 2 0 0 0.0 1 23 0.4 0 0 0.0 0.1 | da3
0 2 0 0 0.0 1 19 0.4 0 0 0.0 0.0 | da1
3 5 0 0 0.0 4 126 2014 0 0 0.0 172.4 | da0

Also DA1 is currently rebuilding (i just replaced that disk as well), I should be seeing way more considering its resilvering

 

[oracle_sod]

Ok i just went and checked the disks on the other box, they are all non SMR.... ok so I guess i know my answer ugh....I guess ill have to let it very slowly rebuild DA1 and then replace the two SMR disks....

I dont really understand SMR or why it makes the disks this slow... but from what im reading, the more data on the drive, the worse it is...