Strange 'disk busy' activity on some disks

[lukeren]

I recently upgraded from my old raidz of 6 x 3TB WD Reds to a raidz2 of 3 x WD Red 8TB and 2 x 8TB Seagate Archive disks.
I'm experiencing some weird business on the two Archive disks, which I'm guessing is because of the SMR design, but though I'd ask here anyway.

da1 and da2 are the Archive disks, the rest are Red disks.

I'm thinking I need to replace the Archive disks with Reds, or could there be anything else that causes just those 2 disks to be at 100% busy most of the time, without any real IO?

 

[m0nkey_]

Do you have anything running that could be causing it, such as jails, VMs, etc?

 

[lukeren]

There are some jails running, but shouldn’t that generate busy on all disks, and/or also show on the bytes/s graph?

 

[genBTC]

The busyness is for sure due to the SMR. But it seems pretty high. It could be Swap. (default is set to create some on each disk) Or could it be some kind of excessive scrub or snapshot routine you setup ? Or just in use from shared access ?

In all likelyhood its just probably due to how the disks lay out responds to being RaidZ-2'ed in that way.

 

[lukeren]

I haven't defined any extra scrubs or snapshots. In fact, I'm a worried that the scrub will take a week with how the performance is right now. The activity is there all the time, even if nothing is accessed via shares.

I do think I have found the cause though, and it IS because of the jails. Well, maybe just one of them, it will need some digging in to.
After I stopped all jails, the 'disk busy' graph flatlined. I guess even the modest file access that the jails do impact the SMR disks. They really don't call them Archive for nothing.