Disk drive suddenly got slow

[sef]

I had a WD Green 1TByte drive and a WD Green 3TByte drive in a mirror; I added a WD Red 3TByte drive to the mirror, and let it resilver, which went nice and quickly.

I was going to remove the 1TByte drive from the pool tomorrow, so tonight I started a scrub. It started out fine, but then about 80% of the way through, it got horribly slow. Some checking showed that the 1TByte drive was going very slowly -- iostat reported a multiple 5 second periods with no I/O at all. I verified this using dd from the device; it was getting less than 25MBytes/sec consistently.

I rebooted a couple of times, with no difference. I ended up removing it from the pool, physically removing it from the system, and rebooting, and it seems to be working properly now (without the 1TByte drive).

There were no I/O errors or timeouts reported that I could see, and as far as I could tell, smartctl for it reported no errors as well.

So my questions are: any suggestions for what I should have done differently? Any ideas for what might be going on? (My best guesses are drive failure -- of course -- or my power supply.)

 

[jgreco]

Does SMART indicate any problems? Disk read errors increasing, etc? Does a SMART long test pass or fail?

 

[sef]

As far as I could tell, no SMART failures. I'm not adept reading the output of smartctl (as I said), but I compared it and one of the other drives using "smartctl -a" for each, and neither seemed to report errors. The "smartctl -a" on the slow drive was quite a bit slower -- a couple of seconds, as I recall, compared to almost instantly for the other drive. (Redirected to a file in /tmp in both cases.) Exit status was 0 in both cases.

 

[jgreco]

Once you've removed it from the pool, it seems like it'd be a good time to do a long offline test.

 

[sef]

Thanks to scroll buffers, I had both outputs. The first one was "smartctl -a" from the bad drive; the second was the same from one of the good drives.

At this point, I still have the drive unattached to the system; if it's the power supply, I'm somewhat afraid of hooking it back up...

 

[cyberjock]

Those both look good to me. Run a SMART long test on the drive and see what the result is. The test will take approximately 174 minutes so be patient. You can view the results by looking at the SMART Self-test log list(where your Short offline test results currently are) after the test completes. Just don't do a scrub or load the system heavily while the test is running or it will take alot longer to complete.

If it passes the long test(and its already passing a short test every 12 hours.. those are useless by the way) then your issue isn't with the drive and something else is broken. If the drive is good then you have something else wrong. That's when big troubleshooting will have to start and it may be a PITA to identify. You never specified how much RAM you have, but hopefully you have at least 6GB and preferably 8GB+. Insufficient RAM can cause all sorts of weird results. On my server I thought I had a failing disk but adding more RAM fixed the issue.

Unrelated comment: WD Green drives should have the idle setting changed. You can only do this from DOS with a tool called wdidle.exe(sometimes wdidle3.exe). If you don't know what I'm talking about you should Google it and change it. My recommendation is setting it to 300 seconds. I've discussed this in the forum before so you may be able to find my long drawn out explanation around here somewhere. There is no risk to your data making this change. In fact I fixed this on a friend's FreeNAS server simply by putting the drives in a different computer that had a SATA controller that worked in DOS and making the change in just a few seconds. In less than 10 minutes I had all 10 of his drives fixed.

 

[sef]

Thanks. I'll try that tonight, or this weekend.

The system currently has 8GBytes of RAM; I have another 8GBytes that should be delivered today, and I'll install it, er, tonight or this weekend :).

I had set the idle settings using the FreeNAS GUI; that doesn't actually work?

ETA: Where are the smart test logs? I looked in /var/log, and didn't see them.

 

[cyberjock]

I had set the idle settings using the FreeNAS GUI; that doesn't actually work?

ETA: Where are the smart test logs? I looked in /var/log, and didn't see them.

The idle settings may work. It depends on your hardware. If you are using a RAID controller it may not work. For standard SATA it should. There's a post around here somewhere that shows the commands you can run to verify if it works. Also some hard drive brands and models won't spin down if you send the command.

The SMART logs are part of your SMART printout. Look at your SMART text files and you'll see the SMART test log.

 

[sef]

Oh, okay, you meant the output of smartctl, not that it had logged the output somewhere else :). Whew. Okay.

Hardware has no RAID other than via ZFS. I think I'd tested that the idle settings worked, when I first set it up, but I haven't checked lately.

Thank you very much, again. I can't do anything until I get home, but I will see what I can figure out then.