WD Red drives 5 drive limtation

[mintra]

Hi

I have a need for a large storage server and was looking at using 16 3Tb drives which would be configured in a RAIDZ2 configuration. That is it is using all 16 drives in the array.

The WD30EFRX and no doubt its 4Tb cousin say in the blurb that they are only recommended in 5 drive arrays.

I can not see why this would be the case and if it were true then would this limitation not also apply to Seagate 3Tb and 4Tb drives.

A google shows another forum where a poster suggests this is a marketing thing, but I am not sure.

I expect to connect these through jbod and have 24Gb RAM.

Any ideas, or indeed experience of running a load of drives in this configuration.

Thanks

 

[DrKK]

From an engineering standpoint, I have no idea why they state this limitation, but, it might be actually real.

http://www.avsforum.com/t/1454542/issue-with-wd-red-drives

 

[cyberjock]

Personally, I'd never be worried about that 5 drive thing. That sounds like a hardware issue on the motherboard and not a hard drive issue. If I had to build my server again today I'd go with WD Reds 100% and without question.

As for your pool preference OP, I hope you are meaning to do 2 vdevs of 8 in a RAIDZ2 than a single large vdev of 16 disks. If not you are really asking for trouble for a whole laundry list of reasons. If you do plan to do a single vdev please read up in the forums as I've answered this question a lot in the last few weeks.

 

[DrKK]

I can't tell you how much I'd love to believe that CJ, because it's logical.

But there are definitely people out there who do not appear to be stupid that say it is specific to the WD red, and they take an equally-sized sixth drive that's a Seagate, plug it into the same mobo SATA port, and it initializes.

I know it's insane. But I'm just saying.

 

[cyberjock]

I'm pretty sure we've had people build systems with more than 5 reds here on more than one occasion. I'll see if I can find one.

 

[cyberjock]

http://forums.freenas.org/threads/mountroot-prompt-after-8-3-9-1-gui-upgrade.15534/

Done. 8x2TB WD Reds on a RAIDZ2.

Edit: Another... http://forums.freenas.org/threads/running-out-of-space.15959/

 

[warri]

DrKK, in the forum you linked there was one person running 12 reds.

I'm running 12 WD Red 2TB drives in my storage server without issue. All the drives are connected to an Adaptec 5805 controller via a Chenbro CK13601 SAS expander.

I'd also say it's a controller/mobo issue.

 

[klayman]

Hi, apparently the reason for the limit are vibrations originating from the drives. Server grade drives have sensors which slightly alter the drive speed to prevent resonance, the WD and Seagate ones for NAS haven't.

regards,
Klayman

 

[cyberjock]

Hi, apparently the reason for the limit are vibrations originating from the drives. Server grade drives have sensors which slightly alter the drive speed to prevent resonance, the WD and Seagate ones for NAS haven't.

regards,
Klayman

Ok, then why do all these people have more than 5 drives. I'm really, and I mean really don't buy the vibration argument at all. I've had hard drives tested for vibrations before, and while their vibrations are noticable, you can get more than 1 magnitude more vibration from a DVD ROM drive at a high speed than from 5 hard drives.

So is someone going to argue with me that if I put a DVDROM in my server the WD Reds in the same case are going to go offline because of high vibration? I really can't buy that they are that sensitive or would be programmed to that level of sensitivity by design.

 

[jonnn]

So suspend the drives with rubber bands. Boom, problem solved. I am an ME and know a few things about resonance...

 

[cyberjock]

I was on Submarines.. so I know a thing or two about vibrations ;)

 

[klayman]

Well, vibrations originating from a single disk (or optical drive) aren't the problem. But it may become a problem if many disks oscillate at the same frequency and these oscillations build up. If you know the natural resonance frequency of your running drive you could decouple it from the case with the right type of rubber. So much for the theory ;-) So this could be a problem in large setups without any rubber buffers, but I doubt it will ever become relevant for home users...

best regards,
Klayman

 

[cyberjock]

For example, I have 24 identical model drives(non-server) in a 4U case and I have none of these issues.

Hard drives actually have very very little resonant frequency vibrations(which is why I dismissed this whole idea as soon as I saw someone type that). They do have some, but not much. This is because the rotating parts of the drive are balanced(and are necessarily required to be balanced within specs to prevent vibrations from ruining the head or causing the data to be corrupted when written or read due to excessive head movement in relation to the platter). Most of the vibration people feel when they hold a drive is from the mechanical head moving. It moves very quickly and has a respectable amount of kinetic energy. But it definitely won't resonate unless you try doing seeks in a given pattern continuously. And then for 5 drives to have that you have to synchronize all the drives' seeks to occur at the exact same moment(which isn't even all that likely with a zpool as the commands are sent to the drives individually but slightly offset in time. Then different firmware can behave differently and other factors could make it even harder to synchronize.

As for the rubber grommetted drive bays, that's a whole other mess of worms. Believe it or not most of those rubber grommeted drive bays actually hurt your drive more than it helps. Sure, it makes your computer a little quieter. But your drive life suffers as a consequence. That's because when the head moves and then stops that kinetic energy isn't transferred and absorbed into the stationary case via the screws that you are supposed to use to mount your drive to the case. Instead the whole disk moves slightly as that kinetic energy is absorbed. In some cases you can actually see the drive physically move ever so little during heavy disk usage times such as a defrag. That motion is really bad for disks. It's so bad I pretty much can't exaggerate how bad that is. But people that don't know better and gladly buy that "super cool case with those super awesome and very useful rubber grommets". The only thing those grommets really do well is dampen the noise on your disk. But your disk lifespan suffers. That's why I don't buy cases with those rubber grommet things for the disks. That's also why back in August in this thread I said to the OP...

How are the drives mounted? Did you mount them with all 4 screws? All 4 screws are highly recommended to minimize vibration on the x, y, and z axis.

It really is important to install all of the screws and make them tight for the longevity of your disks. A lot of people don't do things that seem extraneous like use all of the screws or prefer the tool-less designs. But the old designs are quite often better for reasons that aren't always obvious unless you study the engineering very closely. ;)

What I do like though, are those cases with the rubber grommets for the fans. Those are useful in both minimizing vibrations from fans which aren't necessarily precision balanced anyway and to make them quieter because the vibrations are absorbed. Who cares if the fan physically moves 1/2 a mm when running, right?

 

[gpsguy]

Reminds me of a song: Good, good, good vibrations ...


Sent from my phone

 

[joeschmuck]

The reason is exactly due to vibrations, not anything electrical in nature. I recall this when the Red's hit the street during my research of the drive.

The Red drives do sense vibration and adjust rotational speed (roughly 5400 RPM but it's called "3D Active Balance" because it's not always 5400 RPM) to minimize the vibrations but supposedly the firmware can only compensate for up to 5 drives vibrating. There are no vibration sensors specifically like the enterprise drives may have but it's some method to sense the vibrations using the drives internal components. Don't quote me on this but I thought it measured the rotational speed changes caused by the vibrations. I'm sure there is some engineering reason it technically cannot reduce all the vibrations when you have more than 5 drives in one hard mounted cage but I have been running 6 WD Red 2TB drives in a hard mounted (no rubber grommets) cage without issue. I had 7 drives at one time running. And again, it's not an electrical issue.

Of course if you have vibration dampeners then you really don't need to worry about this. It only pertains to solidly and closely mounted drives. Hope this helps clear it up some.

 

[DrKK]

I was on Submarines.. so I know a thing or two about vibrations ;)

Your {fill in female family member of choice} knows a thing or two about vibrations.

(Sorry. The setup was too good.)

 

[cyberjock]

Your {fill in female family member of choice} knows a thing or two about vibrations.

(Sorry. The setup was too good.)

lol

 

[cyberjock]

The reason is exactly due to vibrations, not anything electrical in nature. I recall this when the Red's hit the street during my research of the drive.

The Red drives do sense vibration and adjust rotational speed (roughly 5400 RPM but it's called "IntelliPower" because it's not always 5400 RPM) to minimize the vibrations but supposedly the firmware can only compensate for up to 5 drives vibrating. There are no vibration sensors specifically like the enterprise drives may have but it's some method to sense the vibrations using the drives internal components. Don't quote me on this but I thought it measured the rotational speed changes caused by the vibrations. I'm sure there is some engineering reason it technically cannot reduce all the vibrations when you have more than 5 drives in one hard mounted cage but I have been running 6 WD Red 2TB drives in a hard mounted (no rubber grommets) cage without issue. I had 7 drives at one time running. And again, it's not an electrical issue.

Of course if you have vibration dampeners then you really don't need to worry about this. It only pertains to solidly and closely mounted drives. Hope this helps clear it up some.

I can't find the website I read, but the "IntelliPower" monicker was for the "5400 RPM" drives. The rumor was that the monicker was invented for marketing purposes to compete with something Seagate was working on (forgot the name off the top of my head). But the drives really have only 2 speeds; 5400RPM and 0RPM. This was determined by noise monitoring of the drive as the sound from the drive will tell you the frequency of the rotation. I'd have preferred to use a stroboscope or those laser guns that tell you the rotation(forgot their name... can you tell I'm tired?).

In any case, I stand by my own comment so we'll just have to agree to disagree on this until there's more information. My next server(next summer if I'm lucky) will be 24 WD Reds in a Supermicro case. That will surely cause headaches if this vibration thing is accurate. I guess we'll find out. :)

 

[Basil]

I've also been reading about the limit, both on the seagate NAS edition and the WD RED edition, even though I have built a few arrays now with more than 5 drives. I have a 20 drive WD RED 3TB array, in a super micro 24 bay - running in a RAID 50, 3 X 6 drive raid 5s. 2 Hotspares, taking the actual 18 drives used to 20, and 2 OS drives in a RAID 1, plain old 250GB drives.

I rented it out to a film company who hammered it flat out for 4 months - only had one drive go offline. Actually had one go offline right in the beginning as well, but that was during the initial build. This ran on an ATTO. But I had 16 4TB WD REDs in a Dell MD 1000, at the same studio, and there was no issue with those - and that DELL actually feels like it vibrates!! I am obviously exagerating - but the super micro enclosures feel rock solid. I run the fans from the server board rather than the back plane - so they dont spin flat out - which I guess is even less vibration - but more importantly less noisy - and they ramp up accordingly to the internal temp sensors so the drives are kept cool. This wasn't even in a server room. It was just a dumping ground, for the data, and from there straight to 2 sets of LTO 5 - and that was shipped daily to another location - so I figured risk wise its worth building and testing it and it seems to have held up in those conditions.

I build another RED array, in a 45 bay super micro server attached to 36 bay primary storage server - but again it was for caching data to speed up renders - this was at another site and in a server room - and the array held solid for over a year.Added I think 36 TB - if I remember correctly - also in a RAID 50 , but 2 X 12 Drive raid 5s. Hmm, actually I think it was 2TB drives, had to be for that size.

So I think depending on the situation it makes sense to go that route. I personally think its more a marketing thing - they are trying to cover the whole market. Yes - we sell drives for the small office - but only buy 5 of them - if you have a 16 bay or 24 bay you have to buy more expensive drives!! -

I've had constellation drives fail in a 6 drive array :) , go figure.. ,

And yes obviously drives to fail at some stage, but I havent found any pattern. I even use the green drives in purely backup servers - in a 24 bay super micro - and absolutely no issue , they don't any more often than the RED or Enterprise counterpart. Except, I have had issues with low powered drives going offline in a SAN - but I think its related to the spin up time.

 

[GaiusBaltar]

i wonder if the drive mounting method makes a difference regarding their tolerance for vibration. e.g.: if the drives are mounted directly to a tray or caddy or bay vs mounted with rubber grommets (as is the norm with most tower cases) to minimize the drives' noise/vibrations?

my new setup is 6x 2TB WD Reds and so far (knock on wood) have had 0 issues compared to the link DrKK provided. i'm fairly sure whatever issue that was, that user's problem was motherboard/driver specific, not drive hardware specific.

and cyberjock, that 24 drive setup you're planning sounds fun :)