Why would a fresh install of 9.10 cause Load_Cycle_Count to skyrocket?

[tolakyrks]

See the attached graph. The inflection point you see in the graph was shortly after I did a clean install of 9.10 and imported my configuration (2016-04-10, to be specific).

To avoid confusion, drives 2-4 are the same age. Drive 5 is newer. Drive 1 was installed during the summer.

I didn't realize that things were out of control until after the numbers started getting ridiculous and I plotted them, otherwise I would have posted months ago. I was always more interested in the various error counters.

Since I made that graph, I have tried changing the HDD settings in FreeNAS with no positive results. HDD Standby has always been set to "Always On." I even tried changing to a different setting and then back, thinking maybe it wasn't displaying correctly. I've tried several Advanced Power Management settings. I've also tried changing the Acoustic Level. Nothing changes. The trend continues upward.

The first 11,000 hours of life for drives 2-4 were spent in servitude to various versions of FreeNAS, so it's obviously not a drive firmware issue. I have not changed anything else about my system.

What would cause this?

 

[m0nkey_]

Please provide hardware details. Not a lot to go on here without at least the make/model of the drive you are using.

 

[tolakyrks]

Apologies. Newbie oversight on my part.

Xeon 1231v3
Supermicro X10SL7F
5x 3TB WD Red

But I honestly don't see why the particular drive model matters since the problem didn't exist prior to the change in software.

 

[BigDave]

Ok, this just a theory, i have no proof or done any testing.
please take what I'm saying with a grain of salt, but if you
look at these things closely, you might find the answer.

Question 1 - For the FreeNAS version you just installed,
where is the default location of the System Dataset? The
reason for asking is that my dataset is kept on the boot
drive and I notice there is a constant level of activity on
that boot drive, while the pool drives show no activity at
all.

Question 2 - What are all your WD hard drives wdidle set
for? The reason for asking this is that I own drives that
were purchased within the last 6 months and the Load
Cycle happens once every 70min. so I never have to worry
about the number of cycles exceeding the hours.

Now think about this, If you have a drive that has low
cycle count numbers and the System Dataset was on the
boot pool for the machine it lived in ( for a longish period),
and then you upgraded/updated the software and BANG,
all of a sudden the System Dataset is now contained on the
main pool and the activity is causing your drives to unpark,
park, unpark, park.

I don't have the time to prove this, it's in your hands now.
I keep a close eye on that Reporting tab now, I just did not
like the fact that there was a constant activity on the disks
in my pool even though it was idle!!! I got a used SLC SSD
and let that get pounded 24/7/365!

 

[SweetAndLow]

You probably have some drives with messed up firmware. Look into using wdidle on them.

Sent from my Nexus 5X using Tapatalk

 

[tolakyrks]

Thanks for your responses. I will have to check the current settings with wdidle.

Three of my five drives have been in service since 2014 and were just fine prior to that fresh install of FreeNAS 9.10. Are you suggesting the firmware just recently became messed up?

To clarify, the graph in my first post is power on hours on the x-axis, not date. Drive 5 is at 3k hours and 2-4 are at 11.5k on the same day, shortly after the change to 9.10. Drive 1 was replaced after this issue manifested. The problem is not that I have a high load cycle count. The problem is that I didn't used to and now it's rapidly accelerated.

The system data set is on my ZFS pool. I spun up a VM running 9.3 and restored an old 9.3 config and the settings for that page were identical. I've had snapshots every 15 minutes and jails running 24/7 since 2014, so there's been something touching the drives pretty much constantly. I haven't changed my usage.

The underlying version of FreeBSD changed between FreeNAS 9.3 and 9.10, correct? Maybe there is some default setting related to hard drive power management that got changed along the way?

I've tried setting syslog and the reporting database to write to the pool with the vain hope that it'll increase the reads/writes to the point where the heads won't park. I'll pull the numbers in a few days to see if the trend is the same.

 

[Ericloewe]

The "how" isn't particularly relevant. Slightly different disk access strategies could be enough to cause the drives to park their heads when they previously didn't.

Three of my five drives have been in service since 2014

November/December 2014? You may have drives from the batch that shipped with an incorrect head parking timer setting.

Use wdidle to confirm the setting. I'd recommend 300 seconds (typical for WD Reds and enough for the drives to effectively never park their heads when used on a pool that contains a .system dataset.

If they already are at 300 seconds, we'll work from there.

 

[SweetAndLow]

The "how" isn't particularly relevant. Slightly different disk access strategies could be enough to cause the drives to park their heads when they previously didn't.


November/December 2014? You may have drives from the batch that shipped with an incorrect head parking timer setting.

Use wdidle to confirm the setting. I'd recommend 300 seconds (typical for WD Reds and enough for the drives to effectively never park their heads when used on a pool that contains a .system dataset.

If they already are at 300 seconds, we'll work from there.

This is exactly what I was thinking.

Sent from my Nexus 5X using Tapatalk