Vdev Configuration

[Richard Henry]

We just purchased one of the Storage Pod's from 45Drives and I plan to install FreeNAS on it. We are going to start out with 15 4 TB drives and add the other 30 later in the year as it becomes needed. I also want to make sure things are setup from the get-go for future expansion. I have been doing a fair amount of research, but have not seen much that talks about the number of drives this particular array will have.

From what I can tell due to the number of drives I will have, the best bet is to go with RaidZ1 in either a 5 or 9 drive vdev setup with 3 or 2 pools respectively. This scares me a bit though as Raid5/RaidZ1 is not really considered to be a good option in this day and age.

I am interested to know if my line of thinking is correct on this or if I should be looking a different route. A majority of this data will large graphic files that will be "parked" for small periods of time until jobs are completed. The initial transfers themselves will be the majority of the transfers it will see.

Thank you in advance for your help!

 

[indy]

As you say raidz1 is not a good idea for a production server.
How about (up to 4x) 11 disk raidz3, maybe even in one pool?

 

[Yatti420]

Indeed raidz1 isn't a good idea? Why would research point you to that setup?? 6x4tb vdev raid z2 would be nice.. I would probably stick with 6x4tb with 42 drives.. Leave 3 in there hooked up for failures.. I don't mess with hotswapping etc (not sure this is supported) so keep 3 drives live/avail for failures with the remaining 3 or put tri mirrored ssds in for jails etc..

 

[Yatti420]

To put all the drives in a single pool .. with raidz1 vdevs is asking for trouble imho.. It might work at first but give those drives some age and I doubt you will be very happy then.. Play it safe because you can afford to!..

 

[cyberjock]

If that storage pod uses the SATA port multipliers you shouldn't even bother installing FreeNAS. Every user that has used port multipliers and openly admitted it has lost their pools.

RAIDZ1 is NOT a good option, and with 9 drives it's suicide. If you want to do that, just delete your data and don't order the disks. It's cheaper that way.

Storage Pods are one of those things that are designed around the business that uses them. In this case, Backblaze. They do redundancy in each pod and each pod has it's own complete redundancy. They also have custom in-house software to manage all of this stuff. You can't meet that kind of reliability with FreeNAS on that hardware. Even Backblaze's latest version that came out just 2-3 weeks ago, they mentioned that the port multipliers have been a total nightmare for their business and their latest revision was built around getting rid of them at any cost because they have cost the company so much.

 

[Richard Henry]

Thanks for all of the quick replies. I am not a fan of that Raid set, but everything I was reading showed that the only multiples of 45 recommended it.
I like the idea of just having 3 drives ready to go if need be, almost like a psudo hot spare. I think I will make a run by MicroCenter for 3 more drives and go with 3 vdev's of the 6x4TB in RaidZ2.

Cyberjock - This is the new 4.0 version that has the bus adapter cards. While I know it is more heavily designed towards Backblaze itself, the team that requested this wanted to be able to reach up to 90TB of storage and not pay a lot for it. We would be looking in the $40k range to be able to reach that otherwise. Being that this is not going to be resource intensive data that requires lighting fast speeds, I feel like FreeNAS on it should do the trick rather nicely (being that it is the newer hardware revision).

 

[cyberjock]

$40k? What!? I've helped businesses build a 48 drive setup for less. Something like $12k or so.

The new 4.0 does look promising. You'll be the first guinea pig for it. I don't see any reason why it wouldn't work, but I would never ever want to be the first guy to test it. Several guys jumped into Storage Pods, some doing contract work, and it ended badly for them. One guy lost money because the contract demanded some reasonable uptime and his server couldn't stay up for 2 weeks straight even with almost no load on it.

It's sometimes better to build out than build up.

 

[Richard Henry]

$40k was probably an exaggeration, but it was like $25-30k. Heck, 48 drives alone will run you almost $10k by themselves.
I would definitely like to know what solution you used for that. I am pretty familiar with hardware and a majority of the quotes we got were in the $10k range for 15 drives. I haven't really had to look for this large an amount of storage however.
This was the only solution I could find that was anywhere in the ball park of what I would get approved.
Probably the worst part of this all is that I just put my two week notice in this week even though we started down this path over a month or so ago. I am trying to do my due diligence to make sure this works for them after I leave. I really appreciate the help that FeeNAS has available everywhere and am making full use of it.

 

[cyberjock]

We got 48 drives at $105 each. Almost $5k total.

Sorry, but I keep my trade secrets to myself.. it was a paying customer, and I do custom builds for people that are willing to pay for it. ;) When I "build" a server for someone I usually provide a parts list and let them do the shopping and assembly. Then I do the actual software installation and setup remotely. Usually it's a smoking deal for them because they can buy the parts from whatever place they want to use and always make sure they get a good deal. I also often recommend ebay for some of the parts which can turn into a savings that is more than my fee. So many businesses actually get an extra good deal. ;)

If you are trying to walk away from your employer without feeling guilty about it you can have them contact me. I'll take all the side-work I can get. They'll probably find my prices very reasonable too. Of course, I can't do actual hardware stuff for them so they'd have to have someone do that part.

 

[Richard Henry]

That makes a lot more sense. No need to divulge trade secrets. :) I thought you were talking about a solution provided by a vendor. You know how some places can be. Doing a non vendor solution can scare the crap out of them as there appears to be no safety net in their eyes. Although $105 each for 4TB drives? I am thinking they are a bit smaller. :)

My boss is already talking to a company for support after I leave. I don't feel guilty at all about the solution, I just want to make sure it is working well and they don't have tons of problems. I will add your info to my notes should they have any problems with it though. I have read through a lot of your stuff and I hope you know that it is probably appreciated by a lot of people. Too many places just throw people to the wolves to figure things out in the open source community.