Backblaze names names... gives failure data on specific HDD models

[scurrier]

http://blog.backblaze.com/2014/01/21/what-hard-drive-should-i-buy/

They specifically call out the Western Digital Green 3TB drives and Seagate LP (low power) 2TB drives as being bad for their application because they start accumulating errors as soon as they're put into production. They suspect vibration and excessive spin-down are the culprits.

 

[cyberjock]

Yeah.. and you know, if you search around you'll find that Backblazes make HORRIBLE FreeNAS servers. Port Multipliers do NOT make stable servers.

 

[scurrier]

I assume by "backblazes" you mean the their open source hardware pods. I'm sure that is true for FreeNAS. But they roll with their own custom software system and have redundancy at a higher level.

But you highlight a good point. Their drive reliability data may not look the same as someone who only runs a few drives in a box, rather than their vibrating mess of 45 drives or whatever. Still, I for one will take this as the best data available to me as an individual.

 

[cyberjock]

Well, I have 24 WD Green drives in my server and I've had amazing reliability. The issue may be related to firmware or whatnot. If I were building a backblaze server I'd definitely read up on problems people had. But for people around here the value of their opinions has almost no weight regarding backblazes(in my opinion). The reason is backblazes just aren't a good fit for a FreeNAS server.

 

[fracai]

These drives are designed to be energy-efficient, and spin down aggressively when not in use. In the Backblaze environment, they spin down frequently, and then spin right back up. We think that this causes a lot of wear on the drive.

That sounds like the WDIdle issue. It would certainly add to the effective cost if they had to modify the idle time before putting the drives into use.

 

[Ramboxman]

Interesting Stuff, wonder if they disables the timers on the Green Drives. I think if you running jails you better check your WDIdle timer even on the new WD Reds since they are defaulting to 8 seconds now.

 

[scurrier]

Whatever happened to letting the controller decide when to spin down? The disk would have to be pretty presumptive to assume it knows better than the controller. That's my newb take on it.

 

[jgreco]

Because EIDE and then ATA and SATA integrate the controller into the drive electronics, and have made the communications channel to the PC is much higher level.

 

[scurrier]

When I said "controller" I meant "HBA.". I was thinking "host controller." I understand the confusion.

 

[jgreco]

A host bus adapter generally has no intelligence of its own (slightly more complex for SAS but let's not go there). It is an adapter that connects the device to the host's I/O busses. The controller logic was moved to the hard drive, for PC's beginning with IDE. In that model, which was little more than some buffering and address decode logic between the PC/AT ISA bus and the IDE cable, it's easy to see that there is no "controller" except for what's on the drive. More recent innovations made the drive interface more complicated in order to simplify cabling, but a HBA is still basically just a conduit for passing arbitrary commands back and forth to a SATA device.

I remain confused as to why you would think that an HBA should have sufficient knowledge of what's going on to accomplish the task you suggest.

 

[jgreco]

http://www.enterprisestorageforum.c...ng-a-disk-drive-how-not-to-do-research-1.html

 

[scurrier]

I thought the HBA was passing messages from the higher level OS or applications which know when the drive can spin down. Better than the drive does.

I think I see some errors in that article you posted although I think both sides have their valid points.

One thing that irks me is the 1e14, 1e15 numbers you see on drive errors. Sorry, clean numbers like these do not exist in real life and it's a dead giveaway that the drive manufacturers are playing some kind of game. What they really should do if they're interested in helping the customer understand the reliability of the drive is publish their test methodology that they used to come up with that number and expose their data to customer interpretation. They don't have to give away trade secrets to do this, they just would have to be willing to share how good their drives really are. "We observed x UREs among this population of drives in the following distribution, which predicts that UREs are distributed according to the following poisson distribution..."

Instead, (taking a wild guess) I think each drive manufacturer doesn't want to compete with the others on this stat for fear that they will one day be inferior. So instead, they go for something that lends more stability to the business and they say, "uh, yeah...we're as good as everyone else in this category. Buy based on other factors."

 

[jgreco]

A pair of strands of fiber optic pass messages between my network switches but doesn't imply that they have any intelligence.

Let's examine your hypothetical machine. Your applications have no idea when the drive should spin down. One application could assume that it knows when it won't be accessing the drive anytime soon and issue a shutdown command, but then another application that is actively using the drive gets screwed. So your applications don't actually know, unless they all participate in a common system to cooperatively manage the drive. In computer science we call a metalayer that manages shared access to an I/O resource an "operating system."

Your operating system can have knowledge of when the last time a drive was accessed and could use historical patterns to guess at future access patterns, or could actually have a formal API to allow applications to indicate what'll happen next. UNIX, however, has never really done a formal job of this. Kernel designers have deemed it a userland problem, making it difficult for designers of laptop OS distributions to do a good job of power management.. well anyways, a bit off track.

In the end, UNIX kernel developers have largely avoided trying to broker drive spinup/shutdown in the kernel or syscall API.

The drive itself is a fully intelligent device, containing its own processor and operating system, able to monitor its own health (SMART), queuing and handling multiple outstanding requests (NCQ/TCQ/etc), automatic error recovery and sector remapping, etc., etc., etc. It is an intelligent processor dedicated to the management of the physical drive. To say that the HBA - a pretty stupid bit of hardware - knows better when to spin the drive down than the drive's controller is a bit odd.

 

[cyberjock]

I thought the HBA was passing messages from the higher level OS or applications which know when the drive can spin down. Better than the drive does.

I think I see some errors in that article you posted although I think both sides have their valid points.

One thing that irks me is the 1e14, 1e15 numbers you see on drive errors. Sorry, clean numbers like these do not exist in real life and it's a dead giveaway that the drive manufacturers are playing some kind of game. What they really should do if they're interested in helping the customer understand the reliability of the drive is publish their test methodology that they used to come up with that number and expose their data to customer interpretation. They don't have to give away trade secrets to do this, they just would have to be willing to share how good their drives really are. "We observed x UREs among this population of drives in the following distribution, which predicts that UREs are distributed according to the following poisson distribution..."

Instead, (taking a wild guess) I think each drive manufacturer doesn't want to compete with the others on this stat for fear that they will one day be inferior. So instead, they go for something that lends more stability to the business and they say, "uh, yeah...we're as good as everyone else in this category. Buy based on other factors."

No, you do not understand engineering practices that determine that value.

They'll sit down and come up with this equation that is suppose to reflect the reality for component wear and tear, tolerances allowed per the engineering specs, and other factors. They'll get some specific number. Then, they'll add in some fudge factor to ensure that the number they are posting is worse than any drive they could sell that would pass their testing criteria at the factory. At the end those numbers should represent the worst case scenario for their disks. So, in theory, any disk arriving at your doorstep should never perform any worse than the value provided. Of course, you can also factor in infant mortality due to any number of reasons, etc. Remember that they don't have to include in their calculations any failures due to poor shipping practices, the dumb guy at the store that dropped a drive on the floor, or the fact that your PSU might not be providing good clean power to the drive. They're simply providing raw number values and an engineer will use these numbers in their calculations along with engineering numbers from the other components to derive potential error rates and whatnot.

So no, those number are no "some game". There's actual serious engineering behind them along with a certain amount of fake engineering(which I won't explain because I'm not the manufacturer). The problem is the people that read those numbers really don't understand where they come from and how they are derived, so they put a different kind of 'value' to the number. If you aren't an engineer you likely will not understand what those numbers mean and will NOT be capable of properly applying them to the world.

 

[scurrier]

jgreco,
Thanks for the explanation. That makes sense to me.

cyberjock,
I respectfully disagree at this point. Or, I should say that I'm skeptical and not convinced. Not saying that it is your job to convince me.

The reason I'm not convinced is that I have some knowledge of statistics (more than the average engineer I know, but much less than any statistician) and regularly work with simple distributions to model manufacturing processes. It's possible that what you're saying is true and they just like to round. But why do they all round to the same number? Coincidence? They all have the same exact technology applied in the same ways?

Even if there is a fudge factor i.e. "factor of safety" it is uncommon to apply it after making the main calculation. To do so makes it seem like playing games to me.

Here's another way to look at it. Do you think they use their published numbers for their own internal engineering calculations? I doubt it. Then why are they asking their customers to use their bogus published numbers? I guarantee they have characterized the real distribution and use that in their internal calcs.

Any distribution can result in an expected number of bits read before encountering an error. The characteristics of that distribution are important for making real calculations and predictions about drive failures, and instead of supplying that information, they are boiling it all down to a single number. If they are not using that information, they are overdesigning. By not publishing that information or applying unexplained fudge factors in proprietary ways, they are causing their customers to overdesign their storage systems.

About the boiling it all down to a single number: Here is an example. If I have a part that has a dimension of 10 +/- 1mm and a part that is 10 +/- 5mm, they both have an expected dimension of 10 mm. And knowing the expected dimension is 10mm doesn't help me use that part in a bigger system without knowing the distribution behind it.

It just doesn't smell right to me. I don't have much specific knowledge on this, but it still smells wrong based on my general knowledge of engineering design.

 

[cyberjock]

OK. But if you've done MTBF calcs before the materials used in the parts and the parts manufacturers will provide some values. WD and Seagate but the disk heads and whatnot. As far as I know there are two disk head manufacturing plants in the world. And if both provide the same values for reliability and WD just provides that number everyone would match.

The bottom line is unless you have inside info all we can do is discuss opinions related to hard drives.

And the fact you mentioned rounding tells me you aren't experienced with MTBF calcs.

Sent from my DROID BIONIC using Tapatalk