Device /dev/disk/by-partuuid/blah-blah is causing slow I/O on pool

[ArchatParks]

I recently moved some four drives from a RAID array hosted by a Dell perc controller to the same raw drives hosted by an HBA controlelr (LSI 9207-8i HBA P20 IT mode).

I now get lots of these messages (and its different drives....message below is just one of the messages):

New alerts:

• Device /dev/disk/by-partuuid/991b2d8e-38b6-4dde-b9d5-7c130086dc12 is causing slow I/O on pool vol74.

Any ideas?

 

[HoneyBadger]

We're going to need significantly more hardware detail here to help sort it out, but my initial suspicion/question is "do you have any SMR drives in your system?"

Please post your system configuration and hardware specs - the more detailed, the better - pay special attention to the motherboard, HBA, and make/model of HDD.

 

[ArchatParks]

Yes, you are right about the drives. I ran the script you mentioned and it said they were the bad drives :/


Known SMR drive(s) detected.

Device | Model | Serial Number |
--------------------------------------------------
sdd | ST4000LM024 | WCK50KJZ |
sde | ST4000LM024 | WCK59WM2 |
sdf | ST4000LM024 | WCK5968H |
sdg | ST4000LM024 | WCK536HD |

Is there a quick way to gather and post the hardware info? I ran lspci and here is what it said:

00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:15.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:16.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:17.7 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.0 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.1 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.2 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.3 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.4 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.5 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.6 PCI bridge: VMware PCI Express Root Port (rev 01)
00:18.7 PCI bridge: VMware PCI Express Root Port (rev 01)
02:00.0 USB controller: VMware USB1.1 UHCI Controller
02:01.0 USB controller: VMware USB2 EHCI Controller
02:03.0 SATA controller: VMware SATA AHCI controller
03:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
13:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 540a (rev 01)
1b:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

 

[HoneyBadger]

Yes, you are right about the drives. I ran the script you mentioned and it said they were the bad drives :/

Unfortunately the ST4000LM024's are indeed SMR. The largest 2.5" HDD that was confirmed as non-SMR was the 2TB Spinpoint M9T (3x667G platters) - there was some suggestion or insinuation that the 4TB Spinpoint M10P was made of 5x800G CMR platters, but they may have been "refreshed" by Seagate to use SMR when they were acquired and rebranded as the ST4000LM016.

2.5" HDDs in general have been either relegated to SMR or supplanted by SSD's. There's an edge market for SAS HDDs disks but it's rapidly shrinking.

Is there a quick way to gather and post the hardware info? I ran lspci and here is what it said:

Code:

00:00.0 Host bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX Host bridge (rev 01)
00:01.0 PCI bridge: Intel Corporation 440BX/ZX/DX - 82443BX/ZX/DX AGP bridge (rev 01)
00:07.0 ISA bridge: Intel Corporation 82371AB/EB/MB PIIX4 ISA (rev 08)
00:07.1 IDE interface: Intel Corporation 82371AB/EB/MB PIIX4 IDE (rev 01)
00:07.3 Bridge: Intel Corporation 82371AB/EB/MB PIIX4 ACPI (rev 08)
00:07.7 System peripheral: VMware Virtual Machine Communication Interface (rev 10)
00:0f.0 VGA compatible controller: VMware SVGA II Adapter
00:11.0 PCI bridge: VMware PCI bridge (rev 02)
00:15.0 PCI bridge: VMware PCI Express Root Port (rev 01)
...
02:00.0 USB controller: VMware USB1.1 UHCI Controller
02:01.0 USB controller: VMware USB2 EHCI Controller
02:03.0 SATA controller: VMware SATA AHCI controller
03:00.0 Serial Attached SCSI controller: VMware PVSCSI SCSI Controller (rev 02)
0b:00.0 Ethernet controller: VMware VMXNET3 Ethernet Controller (rev 01)
13:00.0 Non-Volatile memory controller: Micron/Crucial Technology Device 540a (rev 01)
1b:00.0 Serial Attached SCSI controller: Broadcom / LSI SAS2308 PCI-Express Fusion-MPT SAS-2 (rev 05)

This appears to be a VMware virtual machine for your TrueNAS SCALE installation. You should be able to take a screenshot of the host information as well as the virtual hardware assigned to your TrueNAS VM. On the upside, it does appear that you've passed the HBA (LSI SAS2308) to the VM via PCI passthrough, which is a critical component of a stable VM installation.

 

[ArchatParks]

Well...crud (I cleaned up that word :) ). I guess on to plan B.

In reading I thought the main issue with SMR drives was when you lost a drive, replaced it, then when the dataset started to resilver, the SMR drivers would barf.

 

[Arwen]

To be fair, one main problem with SMR is that it appears Western Digital has a firmware bug in their WD-Red SMR drives which ZFS can trigger.

The 8TB Archive SMR drive that I have from Seagate seems to work fine with ZFS, (on Linux). Well, fine is relative, it's slow for writes, and my use case is backups. But, hey, I am not backup window constrained.

 

[ArchatParks]

Thanks!