Hard Drive Burn-In Testing - Discussion Thread

[alwu]

i'm working on my first freenas build and will start burn-in soon. i'm using a kingston a400 ssd for the boot drive and 6 2tb wd red drives for storage. i won't run burn-in on the ssd since it's been discouraged in this thread. instead of running freenas from a flash drive for the burn-in, can i install and run freenas from the boot ssd, run the burn-in, and then complete the freenas setup to create the zpool, jails, etc.?

 

[Ericloewe]

Yes, of course.

 

[pjc]

In my first look at non-HGST drives, I discovered that other vendors specify a workload rate limit like SSDs do!

On a 10TB drive, 180TB/year is only reading the entire media 18 times. With a transfer speed of, say, 200MB/s, a read of the entire media takes about 14 hours, so you could blow through this limit in 250 hours! (Higher-end drives are only advertising 300TB/year or 550TB/year, which only gets you up to 770 hours.)

So how does this (or should this) affect the burn-in process?

Historically I've run badblocks (4 reads, 4 writes) once, and then the solnet burn-in script (1 read parallel + 5 reads stress test per pass) until the drives have been running for at least 1000 hours. On previous 8TB drives, that was about 12 passes. On a 10TB drive (let's assume 10 passes), that would come to about 680TB in the first 1000 hours, well over even the highest annual specifications, even before considering that's less than 1/8th of a year.

Is the workload rate limit bogus? Or should I have a gentler burn-in process for the first 1000 hours so that I don't exceed at least the annual limit? Or something else?

 

[Chris Moore]

Or should I have a gentler burn-in process

I would say that your burn-in process is more extreme than it needs to be. The badblocks testing is more than enough on it's own. The solnet test is about finding out if any of the drives is running slow, which would drag down the performance of the pool, it is not supposed to be a burn-in. Your test is putting two to three years worth of work on the drive in the first, as you say 1/8th of a year. That is just begging for early death from the drives.
I normally only run three passes on the write / verify test on the disks to ensure they are initially healthy. Then load data into the pool and hope for the best. I am still on 4 TB drives though and it will probably be a few years before I even consider upgrading to 8 TB drives. I was actually thinking of going to a larger number of 4 TB drives instead of going to larger drives.

 

[pjc]

Well, it's been several years since I last did a real burn-in, but back then jgreco recommended a burn-in of 1000 hours, so I assumed he meant with his script.

@jgreco, how do you recommend doing the 1000-hour burn-in given the workload ratings?

 

[joeschmuck]

Oh yes, @pjc you are running into overkill on your burn-in tests.

This qoute is about having the entire server powered on and running for 1000 hours and is meant to identify faulty hardware such as power supply, motherboard, etc... This is not meant to burn-in test your system for 1000 hours but more of use the system for 1000 hours, look for infant mortality. Just my point of view.

@jgreco advises that you should test your server for 1000 hours (!) before declaring it production-ready. If the drives fail, having them fail before committing any data to them is better.

 

[jgreco]

Oh yes, @pjc you are running into overkill on your burn-in tests.

This qoute is about having the entire server powered on and running for 1000 hours and is meant to identify faulty hardware such as power supply, motherboard, etc... This is not meant to burn-in test your system for 1000 hours but more of use the system for 1000 hours, look for infant mortality. Just my point of view.

Well, the expectation is that you're actually exercising the system, which provides you an opportunity to identify faults that include things such as memory errors, bus faults, driver issues, heat problems, and many other types of things. Part of this is DEFINITELY exercising the disk I/O subsystem, preferably heavily, as if you do not do this, ZFS will happily do it for you during scrubs and resilvers. There is less of an incidence of driver issues if you follow the recommended hardware, as Intel SATA/ethernet and LSI HBA are well-supported, and there is also less of an incidence of bus issues now that PCI is basically dead - we used to have all sorts of crap like this in the days of PCI. However, things like memory errors are much less likely to reveal themselves with the massive memory on modern systems, and heat problems still exist.

I tend to like to run at least 100 hours of memtest, ideally once at the beginning and once at the end of the burn-in testing (that's 200 hours), but much of the rest of the time should be spent trying to break the machine, preferably while running UNIX, so it is a fine time to be running tools including the array test, burnBX or other CPU-killing tools, network stress testing, and doing those things all at the same time. Doing this for 800 hours? Without an error? Your system is pretty solid, IMHO.

The badblocks testing is more than enough on it's own. The solnet test is about finding out if any of the drives is running slow, which would drag down the performance of the pool, it is not supposed to be a burn-in.

Incorrect.

Back in the day, there were no meaningful standard tools to do burn-in testing of disks, such as the full rack array of ST173404LW's (8 shelves of 9 drives each) I had to build for a customer almost 20 years ago. SMART wasn't a thing. Each drive type reported stats in slightly different ways. Testing meant actually doing it through disk I/O. Trying to identify issues with controllers and cabling and all that was really annoying. So, yes, solnet-array-test was designed to actually do burn-in testing. The fact that drives now include some basic capabilities to do this with SMART testing, or that other people have discovered the advantages to testing drives, does not change the genesis of the tool. It is definitely supposed to be a burn-in. It's just supposed to be kinda-smart in that it looks for meta-issues while doing so.

To get back to the question about the practice of rating drives for endurance, I will note that there is actually an argument to be made with some of the new recording technologies that you do have the potential to be stressing high-capacity disks a lot more these days when writing to them. I have not seen any credible argument that merely reading disks places these stresses on hard disks, so one has to assume that the endurance of modern hard drives for reads is similar to the endurance of older hard drives for reads. From a marketing perspective, hard drive manufacturers have spent more time trying to differentiate their drives in an effort to wrangle more dollars out of them.

I'm sure WD would be plenty P.O.'d to find out that I've had good luck with using WD Red drives in RAID1 behind LSI RAID controllers for many years, now, which deprives them of the additional revenue of a "data center" or "RAID edition" class drive of the same capacity, and makes my capex budget stretch a lot more. Have to run WDIDLE on them and that's that. I've always been terrible that way.

I think it's fair to say that the less expensive drives do have a tendency to fail at a somewhat higher frequency, but when the cost differential is so great...

Anyways, anyone else looking forward and hoping for flash prices to crash in 2019? Even right now you can get an 860 Evo 1TB for ~$200 which is only 3x-4x the cost of a decent 1TB HDD.

 

[Ericloewe]

Have to run WDIDLE on them and that's that.

Seen any with the timer not set to 300 seconds lately?

Anyways, anyone else looking forward and hoping for flash prices to crash in 2019? Even right now you can get an 860 Evo 1TB for ~$200 which is only 3x-4x the cost of a decent 1TB HDD.

That and DRAM.

 

[jgreco]

Seen any with the timer not set to 300 seconds lately?

Doesn't matter. If you are using these with an LSI RAID and 300 seconds, and you have something like a datastore where the disk is infrequently accessed, the act of idling after 300 seconds will still anger the RAID controller and can cause the disk to drop out of the array. So you still have to run WDIDLE on them, set it to 300, then set it to disable, then power it down, and then check to make sure that the setting stuck. Three of those steps are belt-and-suspenders paranoia, so you need not point that out. Being careful and methodical has paid off over the years here.

 

[joeschmuck]

Anyways, anyone else looking forward and hoping for flash prices to crash in 2019? Even right now you can get an 860 Evo 1TB for ~$200 which is only 3x-4x the cost of a decent 1TB HDD.

That is nice to know. So in about or 4 years I may end up replacing my FreeNAS hard drives with pure SSDs. I'm not sure how long a 3D NAND SSD will last in FreeNAS, I know there have been discussions about it in other threads, and I'm not trying to start another thread about SSD longevity in a NAS. But I do look forward to lower prices, well if they do get lower, never know with a 50% tarrif.

 

[jgreco]

50% tarrif.

At the current rate, the ship is sinking and there's a really good chance a bunch of chickens are going to come home to roost this fall, at which point there might be a full turkey roast in the not-too-distant future.

The fact of the matter is that Congress largely abdicated its tariff authority long ago, and the President is relying on a Kafkaesque interpretation of a national security tariff scheme that was meant to protect the US from enemies and adversaries, not neighbors and significant trade partners. It is very possible that this could backfire spectacularly and result in Congress reasserting itself in this role, at which point the likely thing is for things to return to the previous status quo, because Congress historically had a hell of a time agreeing to any tariffs even for good reasons.

We might be all good a year from now.

 

[Chris Moore]

the likely thing is for things to return to the previous status quo

The status quo was a broken system.
It was a bad decision to move manufacturing out of the US to begin with. Only looking at cost or profit, it might make sense, but if you look at the economy as a whole, the people of the US need jobs so they can earn the money to buy the products. If you move the jobs to another country, it breaks the economy. It might help that country, but it doesn't help this one. Cheap might be nice for the short term, but over the last fifty to sixty years, the country has been on a steady downward spiral. We have to bring manufacturing back to the US or the country will be bankrupt. It already is, but the central bankS are printing money fast enough to make everyone think the economy isn't broken but the national debt can't keep going up forever.

 

[jgreco]

The status quo was a broken system.
It was a bad decision to move manufacturing out of the US to begin with. Only looking at cost or profit, it might make sense, but if you look at the economy as a whole, the people of the US need jobs so they can earn the money to buy the products. If you move the jobs to another country, it breaks the economy. It might help that country, but it doesn't help this one. Cheap might be nice for the short term, but over the last fifty to sixty years, the country has been on a steady downward spiral. We have to bring manufacturing back to the US or the country will be bankrupt. It already is, but the central bankS are printing money fast enough to make everyone think the economy isn't broken but the national debt can't keep going up forever.

There's certainly lots of truth to all of that. It's complicated, though, and involves intractable factors such as economic disparity, regulation, environmental issues, etc.

As an example, during the '80's and '90's, many call centers were offshored to locations such as India, where labor was cheap, and the early adopters found a reasonably large supply of fluent English-speakers willing to work at wages that were great for them at the time. This was not a particularly sustainable model over time, as the demand spike caused the solicitation of less-fluent English-speakers, and sparked the growth of a new, more affluent "middle-class" who simultaneously demanded higher wages while not offering the same level of fluency and understand-ability. Starting after the economic collapse, some rural localities in the US saw opportunity to attack that and offer competitive rates, drawing some call center work back. That's what happens over the long run as economic disparity eventually becomes less severe. But it took ~30 years for that cycle to happen.

Manufacturing is more complicated. While some in the US rail against regulations, unions, and other things that make it "hard" on businesses to conduct manufacturing operations here, it is not clear that there is a willingness to do the right things. Unions exist in large part due to abuses by employers, and to help keep wages at a livable rate. Regulations often exist for very good reasons as well. I don't really want the manufacturers around here spewing pollutants or doing dangerous things. OSHA exists to help keep workers safe and off the disability rolls. It is cheaper when you have some unseen company in Asia doing something without the oversight etc. This doesn't mean that that's a good solution, but it is a tradeoff made by many companies who have found American consumers expect to get all their stuff cheap-cheap-cheap. I might suggest that that last bit is a significant issue that we're unwilling to address.

Further, we have reached a point where a certain percentage of manufacturing has gone "lights out," which refers to factories where only a few workers are employed and much of the actual manufacturing has been automated. This has stark implications for the economy if it continues, which it is likely to. Even other businesses are looking to minimize employment. ATM's have eliminated many bank tellers. Self-checkout lanes have eliminated clerks at many stores. Company mergers whittle down overall employee counts. McDonald's desperately wants you to use their new ordering kiosks instead of interacting with an employee at the counter. Computerization and the Internet has replaced lots of customer support jobs. Etc. This seems likely to continue.

A lot of the traditional answer to job loss has been "re-train and re-educate." To the extent that new jobs exist, that's fine, but I'm really not sure what's going to happen to the three million truck drivers who are likely to lose their jobs in the coming years, etc. Where are millions of jobs going to come from? There is already significant evidence that the current "low" unemployment numbers are failing to account for underemployed and workers so discouraged by low demand for their skills that they are no longer seeking work. The labor force participation rate may be more telling.

Now, to bring this around to a vaguely relevant-to-the-forum post, I'll point out that it is interesting that Foxconn has come to Wisconsin, a site I expect was selected in large part due to the ready availability of water from the world's largest fresh water supply, which their manufacturing processes require a lot of. I am skeptical of the claims of lots of high paying jobs in the long term for Wisconsin, although I'd be happy to be wrong. The past fifty years, as you note, has not shown a lot of manufacturing job creation in the US, and the motive for this move seems suspicious. Manufacturing their technology in the US isn't really resolving the issues we're discussing if it doesn't also create a crapton of jobs.

But if Samsung comes and opens a factory here, I'll see if I can hijack an autonomous semi via the Internet and then I'll sell stolen SSD's to everyone on the forum here cheap, hahaha!

Seriously, though, the continual downward slide of technology prices is driven by automation. This seems like it's in direct conflict with the issues we're discussing. :-/

 

[pjc]

I have not seen any credible argument that merely reading disks places these stresses on hard disks, so one has to assume that the endurance of modern hard drives for reads is similar to the endurance of older hard drives for reads.

That makes a lot of sense. Assuming the rating is analogous to the workload rating of SSDs, it should only apply to writes. (Otherwise monthly scrubs would consume half of your annual workload!) Of course, WD is terrible about publishing technical specifications, unlike HGST and Seagate, so we're left to guess about what makes up their rating.

Given that, I plan to use badblocks (40TB of writing) and then the solnet array test (no writing) until 1000h.

Have to run WDIDLE on them and that's that.

Ah yes, WDIDLE, I'd forgotten about that. That would explain why I'm seeing 5W of power draw from these Reds even though the spec sheet says 2.8W for idle. Presumably you only see 2.8W when it parks the head and goes into WD's not-quite-standby idle mode. Again, there's no decent documentation available to describe the various workloads that produce the specified power draws, so that's just a guess.

On the plus side, their temperature stays reasonably cool despite the higher-than-expected power consumption.

 

[pjc]

I'm not sure how long a 3D NAND SSD will last in FreeNAS, I know there have been discussions about it in other threads, and I'm not trying to start another thread about SSD longevity in a NAS.

Personally, I'm hoping that Intel's 3D XPoint technology will come down in price (though presumably it will always command a premium). 30x improvement in latency, and absolutely insane writability: 10 drive writes per day.

But it's a bit power-hungry: 6W/TB idle, 16W burst write, 11W burst read. And expensive: $1200/TB.

 

[Stux]

Burst write/read are probably over faster though ;)

What’s the W/GB burst read/written?

 

[pjc]

Interesting preliminary results from solnet (that I haven't seen before):

Serial read is about 200MB/s. Parallel is 180MB/s (90%), across all 8 drives I'm burning in. That's a big drop!

There shouldn't be a bandwidth bottleneck: the dd processes are the only substantial CPU usage, using only 10% of my CPU cores, and the drives are on a direct-attached backplane to an LSI 9201-16i (PCIe 2.0 x8, which should be 4GB/s). No active scrubs, and no error messages in dmesg either.

This is the first time I've tried to burn in 8 drives simultaneously (generally 4 at a time previously), but the system has been rock solid for several years, so there shouldn't be a systemic infant mortality problem either.

Does the LSI HBA not fully saturate the PCIe bus? At 200MB/s, 8 drives would push less than half the available PCIe bandwidth.

 

[Chris Moore]

That's a big drop!

What is a big drop?

Does the LSI HBA not fully saturate the PCIe bus? At 200MB/s, 8 drives would push less than half the available PCIe bandwidth.

That is likely the maximum mechanical speed of the drive. What are you expecting?

 

[pjc]

Serial read is about 200MB/s. Parallel is 180MB/s (90%), across all 8 drives I'm burning in. That's a big drop!

10% drop in performance by reading all 8 drives simultaneously instead of each individually. Thus, entirely unrelated to the mechanical maximum.

I'd expect 200MB/s in both instances, since there's not an obvious bottleneck to cause the reduction in throughput.

 

[Stux]

10% drop per drive when scaling out from 1 drive to 8 seems okay to me.

And it’d also depend on what part of the drive is being read.

You’re getting 1.45GB/s instead of 1.6.

Maybe your system just can’t process 1.6GB/s?

Try with six drives?

Solnet is just a clever script orchestrating dd sessions. It’s not like it’s super optimized.