Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

BUILD Looking for serious input on a Enterprise Grade Build.

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE
Status
Not open for further replies.

ETSCORP

Newbie
Joined
Aug 31, 2014
Messages
3

Let me start out by saying I am new to the community and I appreciate the efforts of everyone to provide constructive feedback and help for those who are learning. I especially want to thank Cyberjock for his “ZFS Storage Design and other FreeNAS information” powerpoint guide.

With introductions out of the way I want to layout what I am thinking and ask for your honest feedback.

Goals:
Learn as much as I can about ZFS and FreeNAS.

Learn and Understand Best Pratices, and NOT cut corners.

Provide high storage solutions for a small datacenter that I own. The core hardware in the datacenter is Dell 10 GbE switches (Powerconnect 8024F) and Dell Power Edge Servers. The software layer is VMware ESXi 5.5.

Storage will be provided to the hosts via 10GbE iSCSI (2 x 10GbE with LACP per VMware host). I am open to suggestions if there are better options than iSCSI.

Design a highend storage solution based upon FreeNAS and commodity hardware and then build 2 identical units. (1 storage unit for live production and 1 storage unit for backup).

Basic Design Concepts

Known Details for Build/Hardware on Hand (I currently own the following):

Software:
FreeNAS 9.2.1.7


I had considered Windows 2012 R2 Storage Server (ReFS) but I feel ZFS had to many advantages.

Head end for JBOD Storage:
Server(s) for FreeNAS: 2 x Dell PowerEdge R420 (Identical Build)
2 x Intel Xeon E5-2430 Processor (2.20GHz, 15M Cache, 6 Core,)
192 GB of (12 x KTD-PE316/16G) DDR3, 1600MHz, ECC, Registered
Dual Hot Plug Redundant Power Supplies 550W
PERC H710P Integrated RAID Controller, 1GB NV Cache
8 x Intel Pro 2500 2.5" 120GB SATA 6Gb/s MLC Enterprise SSD on PERC H710P
Dell iDRAC Enterprise (Remote Access and Management)
1 x Intel X520-DA2 Dual Port 10GbE
1 x Intel i350qp Quad Port 1 GbE

Unknown Details for Build/Hardware to buy (I am looking for input):

JBOD enclosure options:

Option 1: Chenbro 9U 50 Bay Storage Chassis (Preferred Option Based on drive QTY)
http://usa.chenbro.com/corporatesite/products_detail.php?sku=45

I believe for Option 1 I would need to use the following SAS Expander in order to use the above-mentioned chassis. Is this a deal breaker? http://usa.chenbro.com/corporatesite/products_detail.php?sku=76

Option 2: SuperChassis 847E26-RJBOD1 (Second choice but only 45 drives, might be a winner due to the fact as I understand it the backplanes have dual-port expanders that access all the hard drives. These dual-port expanders support cascading, failover, and multipath. I think that is desirable?)
http://www.supermicro.com/products/chassis/4U/847/SC847E26-RJBOD1.cfm?parts=SHOW

Option 3: E9M50 9U 50-Bay Storage Server Rackmount Chassis (Losts of unknows. I am not able to find info on connection options or even a manual for this chassis)
http://rma.istarusa.com/rackmount_chassis/estorm/raid/e9m50.aspx

HBA Controller:

Option 1: LSI Logic SAS 9300-16e SGL
http://store.lsi.com/index.cfm/Host-Bus-Adapters/12Gb-SAS-9300-Adapters/LSI00342/

Option 2: LSI SAS9206-16e
http://store.lsi.com/index.cfm/Host-Bus-Adapters/6Gb-SAS-9206-Adapters-PCIe-3.0/LSI00299/

Option 3: LSI SAS 9201-16e
http://store.lsi.com/index.cfm/Host-Bus-Adapters/6Gb-SAS-9201-Adapters/LSI00276/

Hard Drive Options:

Hard Drive Option 1: Seagate Constellation ES.3 ST4000NM0023 4TB 7200 RPM 128MB Cache SAS 6Gb/s 3.5" Enterprise Internal Hard Drive
http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM1PU0823

Hard Drive Option 2: TOSHIBA MG03SCA400 4TB 7200 RPM 64MB Cache SAS 6Gb/s 3.5" Enterprise Hard Drive
http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM1R03009

Hard Drive Option 3 (If I were a rich man option) HGST He6 HUS726060ALS640(0F18370) 6TB 7200 RPM 64MB Cache SAS 6Gb/s 3.5" Enterprise Hard Drive
http://www.newegg.com/Product/Product.aspx?Item=9SIA5EM1PY9126

So what I am looking for is an affirmation that the hardware above looks solid. I am also open to alternatives. Including purchasing a prebuilt enterprise grade unit if my plans look too foolish. I figure that with the hardware I have on hand I should be able to assemble the storage enclosure with the 4TB option and get it connected for about ~$25,000.00 or less.

I have considered a BackBlaze style unit and looked at 45Drives offering but I ruled it out as I want easy front load access to the drives and I am willing to provide the rack space to due so. I am going SAS because I believe it will provide for better stability and hopefully that will translate into a safer environment. I am also somewhat confused by how I would go about connecting the storage enclosure to the server. I know I will use an 8088 or something like it cable or a series of cables or a fan 8088. If any one has any insight I would appreciate as this is my first JBOD build.

As far as the actually storage design I am considering using a ZPOOL composed of two VDEVs each with 25 drives or 22 if I go with the super micro chassis configured as RAID-Z3. Suggestions or improvements or betters configuration options are welcomed.

I am also very confused by the lack of support for a true hot spare. How can iXsystems, which is based on FreeNAS, offer this but FreeNAS lack it. I know some people think it is over kill but I like a security blanket.

I am sorry if this is not enough information. Please ask me questions and make lots of suggestions. I am open and willing to learn and I want to build something that works for me but also excites and attracts the interest of the community. If others have built what I am describing please point me to the posts and I will read before asking further questions.

Thanks in advance, Jacob
 
Last edited:

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
To be honest, you should just call iX and buy a system with what you need. You're talking high-end and lots of money. You're going to be very pissed if you drop $25k+ on your server only to realize some small (but really freakin' important) thing was left out that would cause you go to back to the drawing board.

And how does IX offer hot spares? It's called 'commercial support'. As in, you pay for their hardware and software and you get some extra features. You're going to find that life can be difficult handling all of this yourself and you are probably better off buying from iX. iX gets regular calls from people wanting support and you'll learn that you should just buy TrueNAS or be ready for some "tough love" from FreeNAS if things go bad.
 

ETSCORP

Newbie
Joined
Aug 31, 2014
Messages
3
Thanks CyberJock for you straight-up honest answer. I have reached out to them for a quote but I am afraid they are going to want way more than $25K per device for ~170TB. I was kind of looking forward to trying this myself but with that said I understand that you get what you pay for. Mixed emotions what can I say. :) :-(

With that being said this FreeNAS for Enterprise an unruly beast? Tough Love does not sound desirable as I am more a gentle kisses type of guy! :) If nothing else I will build something with FreeNAS just to learn it better as maybe you say what you say because I a green and inexperienced with FreeNAS. Does iXsystems offer a software license for build your own with the additional features?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
I don't do licensing for iX.. that's for the salespeople to handle. But from my experience the licensing is basically a "buy it from us and we'll support it... don't buy it from us and we won't support it" situation. Companies don't exactly want to support some franken-machine you built out of cheap hardware, then call them up demanding they make it work.

Time is money though. And to be honest, for the amount of money you are talking I have to say that either:

1. Your data isn't worth as much as you think (or the price from iX would be palatable)
2. Your data isn't worth as much as you think (and you're going to find out when you potentially lose data).

I just talked to a guy last week that lost a 200TB pool. Had the pool for over a year and kept making small mistakes that aren't immediately obvious until you find your pool is trashed and virtually every file on your pool is throwing I/O errors. ;)

Feel free to not take my advice (it was free) but generally the people that think they are a big roller and are going to roll out a big-ol' system on-the-cheap find out that neither did they roll out a good system, but ultimately it wasn't on the cheap (aka .... the cost was their data).

People like this really don't know what they are getting themselves into, and very few people on this forum are going to have actual advice based on experience.

I've always take then stance that if your data is worth a lot of money you'll happily spend the money to protect it. The bank that spends $1million on a safe to protect $20k in cash probably hasn't thought out their situation too thoroughly. Likewise, you don't spend $20k on a safe that will store millions in cash. It sure is sounding to me like you are trying to do the latter in the name of 'saving money'. As a joke.. you know how I can save you lots of money? Just delete the data. :D
 

ETSCORP

Newbie
Joined
Aug 31, 2014
Messages
3
Once again thanks for your candor. I think we are and have always been on the same page. I have reached out to iXsystems, Dell, Nexenta, etc. I am not looking for a headache but was curious about the community and the resilience of FreeNAS. At what point do you personally go commercial? Solely based on risk vs reward? Do you personally ever trust data of great value/enterprise to FreeNAS? I always explore my options and try to support great open source project by contributing in what ever way I can. Based upon your experience and rather pessimistic view of my proposed self build I will more than likely go a different route as you are the one with experience. I will see what I can get commercially for my $50k.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
It's really about how "comfortable" you feel. Some companies might want a contract with 20TB because they want someone they can call and say 'hey, your shiz is broke.. fix it now!' while others are more home-brew. For me, if you are going out beyond what is generally acceptable by the community I'd say you are getting into rough terrain. There's tons of people with 10TB of storage, so experience is vast. Go to 20TB and there's fewer people, go to 50TB and there might be 10 in the whole forum, active and inactive users.

At your scale you are talking about things that 99.9% of users could only dream about. Myself, I don't own any systems of that size but I've been involved with building two and had to admin a few for customers. I would consider going at it yourself rather risky (not necessarily pessimistic) just because of the amount of money and quantity of data involved. If things go bad you can bet your job will be on the line. Is that worth the risk to you? Almost certainly, if you have a problem unique to your situation there will be nobody in the forums with the experience to be able to provide good advice. Is that okay with you?

If this is your first FreeNAS build I'd call you foolhearty for trying. Definitely do NOT, under any circumstances, make a build of this magnitude your first one. Considering your first sentence said you were new to the community I tend to think this is your first build. Definitely not the place to be, especially as a job-related task.

Remember, at the end of the day the data isn't what really matters.. it's your job. If you think your job can handle the kind of mistake that may come with a build of this magnitude, by all means give it a shot. Personally I wouldn't wager my job on it as a first build. Even as I did my first build I couldn't possibly have gotten it all right the first time. Your boss is almost certainly going to be upset when you drop $25k and then realize you need to spend another $3k, and then another 3k. And what if FreeNAS has some bug that makes it impossible to implement on your network. Now you are in *real* trouble and you are going to be all alone to deal with it. (Yes, I've seen this happen to a few people).

These aren't imaginary scenarios. This is stuff I've seen for real. Buying from iX is pretty painfree. You write a check and iX makes it work. iX treats their customers pretty fairly from what I've seen. You can bet dollars to donuts that if you call and say "hey, my server won't go on the domain" they'll throw everything they can at the problem, including the kitchen sink. It's not the kind of thing you normally would expect from a company in this day and age, but it's how iXsystems operates.

FreeNAS is great because they get an extremely wide 'testing' audience for literally free. Then the audience finds the bugs and they get fixed in both FreeNAS as well as their commercial product, TrueNAS.
 

jamiejunk

Member
Joined
Jan 13, 2013
Messages
119
Just a few notes. I agree with Cyberjock about buying from IX. If i had it to do over I would have bought from them. Unfortunately dealing with FreeNas issues takes up a lot of my time. If I had bought from IX we would have had better support. Not to say the support i get here on the board is bad. But sometimes people on the boards can’t help and you’re just stuck. If you had iX behind you, they would have to see the problem through to the end I would think.

Getting ZFS to work correctly especially in an enterprise environment isn't much fun.
But if you decide to go it alone here are some suggestions. Go with Supermicro. Their hardware is awesome. Can be a little more money, but tends to be worth it.
Intel 10gig nics do NOT work with freenas at this point. I think they have a fix planned for 9.3

Don’t do 2 vdevs. You will be limited to the write performance of two drives. Go with mirrors and back that stuff up using ZFS send to another freenas if you can.
Do a bunch more research on this.
Remember if you lose one vdev you lose the array.
 

mjt5282

Member
Joined
Mar 19, 2013
Messages
137
while I am not a professional user of Freenas for a business, I have built a Supermicro chassis + SM motherboard with a 3U and a raid3z 11-wide 22Tb usable 3Tb (green and red) . For primarily storing movies, music, data, backups, it works fine. I will say, for storage, you should learn enough about ZFS to ask about specific configurations and multi-customer data storage. Build a proof of concept (don't try to shoot the moon). Learn about raid2z/3z and zvols and mirrors for VM's (and why you don't want to host them on 2z/3z...) Learn about L2ARC and ZIL's ... Learn about iSCSI and synchronise writes and how to use SSD ZIL's to speed them up .. Myself i wouldn't try to build a single "mega" server but rather a few 3U or 4U SM chassis/motherboards with Intel CPU's and ECC RAM. The basic ideas of check-summing, snapshots, RAID flexibility (at birth!) are all enterprise features that FreeNAS offers but moving up to EMC SAN or Netapp filer may not be realistic or affordable.
 
Status
Not open for further replies.
Top