How to troubleshoot one drive (noises) w/o causing Zpool corruption?

[ohboi]

Two weeks ago I started noticing noises from my NAS, I think it started, when...in just little frustration I might have just tapped with my fist (top of my Fractal R6) and I think that if something happened it probably happened then with that little bit of vibration.

Okay, anyway, I believe one drive is making these noises and I am afraid it is soon to die and I'd rather replace it now, however Zpool has no errors, SMART showed no errors either, so I would like to try to unplug drives and listen when the noises stop.

Here is a recording, it is actually continuous, every 5 seconds there is this noise, sometimes it is louder, sometimes quieter
https://mega.nz/file/ciZGFZrK#2Vp8z5KXTr6DpI4ouXuvq-2-5SW_CNOVmvqUEwoYgc8 (at 1st, 6th and 11th second mark) .
A question here is, is it safe to assume this drive sooner than later will go haywire?

Do you have any recommendations how shall I approach this testing? I was thinking rebooting server to BIOS and unplug one drive at a time and listen when the noise is gone, but I am not sure if this may not cause a damage to any of the drives as I have just a Fractal R6, so no hot-swap bays.

Edit: Bad news, this is load-related, I rebooted to BIOS, quiet. When FreeNAS is up and pool is active, it does it. Is there any work-around? Maybe putting a whole pool offline and running some read test (which wouldn't obviously touch data) to simulate the load?

 

[ohboi]

Anyone?

 

[AlexGG]

I use a tool called "mechanics stethoscope", which may or may not fit your use case. I mostly use it for fans, though.

 

[ohboi]

That's actually a really clever way to do this, I was thinking more about something like taking the pool offline (does "Export/Disconnect" option in Storage>Pools do this?) in UI and running some read benchmark command on a specific drive (one by one) to see which one's the culprit, but I admit this is the point where I don't exactly know if the formatting by ZFS allows such read test without any issues or possible data alteration and what command would be the optimal to run for this read test.

 

[AlexGG]

Another option is to run a short (or long) SMART test on each drive in turn, listen for sounds. This can be done without stopping the pool (SMART tests are read only and will give way to any actual read or write requests), just make sure it is not actively doing something, least the activity interferes with the sound pattern. Someone else will have to provide the actual command though.

 

[sretalla]

"every five seconds" sounds like the system dataset being written to.

Depending on what boot media you use, you might want to try moving the system dataset to the boot pool and see if that makes it stop. (System | System Dataset)

 

[ohboi]

"every five seconds" sounds like the system dataset being written to.

Depending on what boot media you use, you might want to try moving the system dataset to the boot pool and see if that makes it stop. (System | System Dataset)

Are there any implications of doing so? Currently I am running OS on two USBs, but I am switching to two SSDs in a couple of days (USB solution is not very reliable).

 

[sretalla]

Are there any implications of doing so? Currently I am running OS on two USBs, but I am switching to two SSDs in a couple of days (USB solution is not very reliable).

Do it after you switch. It's not necessarily going to kill the USB sticks, but it probably isn't great for them.

 

[ohboi]

So all is done now, USBs replaced with SSDs and dataset moved, however that noise is still there, it is less consistent with that 5 second delay and still, it is load-related, I have some slow rendering on the pool directly and it growls quietly along.

 

[Jailer]

Get a plastic handled screwdriver and hold the handle end to your ear. Touch each drive lightly with the driver end and listen for the noise. You'll know when you've found the offending drive.

Got that trick from a local blind mechanic that uses it to diagnose engine noises.