Drive Read Light on Constantly

[Jeeves]

All, my tertiary 8TB HGST drive/pool is exhibiting a weird behavior. Not sure whether to post this hear in hardware or over on Storage.

- The system boots normally
- The Pools are imported appearing to be normal
- As soon as the system is up, the 8TB goes into a solid/constant read state
- If I detach the pool, the light goes off. If I re-import the exiting 8TB pool, the read light goes on solid again.
- The other WD 3TB drives show normal/very low read rates in the "Reporting/Disk" graphics
- When the HGST 8TB pool is imported and active, the "Reporting/Disk" graph shows a maxed out read rate
- Read light even stays on constantly when the network is unplugged, indicating that it is a "local" problem?

I cannot discover any errant processes, I use no jails. I only have one VM defined, the read error happens whether the VM is active or not. I had SMART on, now turned it off, no change.

Any thoughts/ideas as to what might be causing this?

Many thanks,
Jeeves

 

[dlavigne]

What's happening in top or ps -acux when the activity occurs? Also, anything being logged to /var/log/messages at that time?

 

[Jeeves]

Thanks for your reply, Ms. Lavigne... The first screen snap below is with the pool detached, the second is after it was imported and the read light was on solid. Looks like "zfskern" and "uwsgi-3" are the add'l processes? And here are the msgs from the interval. I did get one error when trying to re-detach the pool, presumably timing out from the constants reads on the drive?

Feb 7 19:27:25 freenas ZFS: vdev state changed, pool_guid=4069310295345955240 vdev_guid=142284055689647569
Feb 7 19:27:25 freenas ZFS: vdev state changed, pool_guid=4069310295345955240 vdev_guid=142284055689647569
Feb 7 19:30:01 freenas syslog-ng[6931]: syslog-ng shutting down; version='3.14.1'
Feb 7 19:31:08 freenas syslog-ng[26210]: syslog-ng starting up; version='3.14.1'
Feb 7 19:31:08 freenas uwsgi: [middleware.exceptions:36] [MiddlewareError: Failed to detach Bkup8TB with "zpool export Bkup8TB" (exited with 1): cannot unmount '/mnt/Bkup8TB/iocage/jails': Invalid argument
]
Feb 7 19:31:28 freenas syslog-ng[26210]: syslog-ng shutting down; version='3.14.1'
Feb 7 19:31:28 freenas syslog-ng[26518]: syslog-ng starting up; version='3.14.1'

 

[Jeeves]

FWIW, I was not able to find a solution to this. Since it is my tertiary drive, I opted to detach the pool, wipe the 8TB disk, and create a new 8TB pool. Everything seems to be fine with the 8TB drive now, the read light is now off as expected, with both pools attached.

Now to Rsync again to fill it back up...

J

 

[Jeeves]

Wonderful, just completed a FreeNAS software upgrade, now my FreeNAS instance is consuming 20% of systems resources at "idle" and the drive light is on solid on the PRIMARY drive array (not just the tertiary backup drive). This is getting ridiculous. Same processes eating it up as prior screen snapshots, BHYVE and Python... Anyone else experiencing this? Suggestions on resolution? Only "abnormal" thing I observe is the system telling me that my 32GB Samsung USB drive is at 84% and the recommendation is to be under 80% utilization. Should I rebuild my USB boot media and re-import the pool?

 

[Jeeves]

Learning as I go, just sharing in case others have this... I re-installed to a fresh USB 32GB drive, and imported my volumes. Everything looked ok for a bit, but then the drive read light went on solid again, same system and process states and usage, and whether the main Vol and Backup Vol were online, or just the main Vol.

Managed to find an article that talked about ZVols hanging around after VM's were deleted. I had long ago deleted some VM's I was trying out, and discovered that I had 4 leftover ZVols in such a state. Switched to legacy UI per article instructions and deleted the 4 ZVols, and (knock on wood) so far so good, meaning no drive read light on solid, and system load looks normal.