Single disk has high utilisation in pool

[Newfoundland.Republic]

Folks - I have (what seems to be a weird) issue with one of my ZFS pools. I have 5 WD Red 3TB drives in a RAIDZ2 that is attached to a LSI 9210-8i controller. The drives are in a Supermicro 825TQ-R740LPB chassis (hot-swap drive bays). (Full spec below and in signature). I should note that I have had this happen with U5 and now U6.

During heavy activity such as a scrub or large and long file copy (SMB, NFS) one drive in the pool reports high utilisation. It is only reported for one drive in the pool that, ironically, is the newest (less than 2 months old) of the five. SMART does not report any errors on that drive (or any other, for that matter).

Anyone have any ideas while only drive would report this?

I have a LSI SAS9220-8i (IBM M1015) available if the issue is with the 9210. If I swap them out, is it as simple as swapping them or would I need to export/import the pool, etc.?

Thanks all!

Spec:

• Operating System: FreeNAS-11.2-U6
• Chassis: Supermicro SuperChassis 825TQ-R740LPB 2U 8 x 3.5" Drive Bays
• Power Supply: 2 x 740 Watt PWS-741P-1R Power Supply Platinum
• Backplane: Supermicro BPN-SAS-825TQ 8-port 2U TQ (W/ AMI 9072)
• Motherboard: Supermicro X9DR3-LN4F+

• CPU: 2 x Intel Xeon E5-2630 V1 Hex (6) Core 2.3GHz
• RAM: 32GB DDR3 ECC (8 x 4GB - DDR3 - REG)

• Storage Controller: LSI 9210-8i 6 GB/S
• Boot Pool: 2 x Kingston AS400 120 GB SSD Mirrored (using motherboard SATA 6 GB/s)
• Pool_1: 5 x WD Red 3 TB RAIDZ2

• 2 x MiniDLNA servers (seperate subnets) with shared content
• NFS server for VMware ESXi 6.0
• CIFS for Windows clients

 

[dlavigne]

Did you determine the reason for this?

 

[Newfoundland.Republic]

No... There does not seem to be any consistency. I will manually schedule a scrub tonight and report back.

 

[Newfoundland.Republic]

Started running the scrub. Netdata report da1, da2, da3 and da4 running between 60-85ish. da5 is running 90+ with a high of 113....

 

[Newfoundland.Republic]

First warning:
disk_util.da5
CHART
10min disk utilization = 90.3%
the percentage of time the disk was busy, during the last 10 minutes

ALARM
da5 FAMILY

WARNING
SEVERITY

 

[Newfoundland.Republic]

Okay... 5 HOURS after the scrub completed I get the error:
disk_util.da5
CHART
10min disk utilization (was warning for 5 hours and 36 minutes)
the percentage of time the disk was busy, during the last 10 minutes
ALARM

da5
FAMILY

Recovered from WARNING
SEVERITY

Anyone have any idea what is happening here?

 

[Newfoundland.Republic]

Scrub just finished on Pool_1. It is a 5-disk RAIDZ2 pool. This time all 5 disks warned on high utilization. (Pool_0 is a single disk that I just use for shytes and giggles if I need to)

Code:

root@freenas[~]# zpool list
NAME           SIZE  ALLOC   FREE  CKPOINT  EXPANDSZ   FRAG    CAP  DEDUP  HEALTH  ALTROOT
Pool_0         296G  6.87M   296G        -         -     0%     0%  1.00x  ONLINE  /mnt
Pool_1        13.6T  1.51T  12.1T        -         -     2%    11%  1.00x  ONLINE  /mnt
freenas-boot   111G  1.49G   110G        -         -      -     1%  1.00x  ONLINE  -
root@freenas[~]# zpool status
  pool: Pool_0
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:06 with 0 errors on Sun Sep 29 00:00:06 2019
config:

    NAME                                          STATE     READ WRITE CKSUM
    Pool_0                                        ONLINE       0     0     0
      gptid/60361f33-c1b2-11e9-831b-0025902f87c4  ONLINE       0     0     0

errors: No known data errors

  pool: Pool_1
state: ONLINE
  scan: scrub repaired 0 in 0 days 01:08:49 with 0 errors on Sat Oct  5 19:23:50 2019
config:

    NAME                                            STATE     READ WRITE CKSUM
    Pool_1                                          ONLINE       0     0     0
      raidz2-0                                      ONLINE       0     0     0
        gptid/5448fd06-c388-11e9-831b-0025902f87c4  ONLINE       0     0     0
        gptid/56c276c6-c388-11e9-831b-0025902f87c4  ONLINE       0     0     0
        gptid/5925dbce-c388-11e9-831b-0025902f87c4  ONLINE       0     0     0
        gptid/5ce7c14d-c388-11e9-831b-0025902f87c4  ONLINE       0     0     0
        gptid/60a9b0d5-c388-11e9-831b-0025902f87c4  ONLINE       0     0     0

errors: No known data errors

  pool: freenas-boot
state: ONLINE
  scan: scrub repaired 0 in 0 days 00:00:06 with 0 errors on Wed Oct  2 03:45:06 2019
config:

    NAME        STATE     READ WRITE CKSUM
    freenas-boot  ONLINE       0     0     0
      mirror-0  ONLINE       0     0     0
        ada0p2  ONLINE       0     0     0
        ada1p2  ONLINE       0     0     0

errors: No known data errors

 

[Newfoundland.Republic]

Confirmed: High Utilization on all disks occurs during scrubs. Is this normal?