An Enterprise Deployment 100TB+

[framewrangler]

I’ve recently been tasked with architecting and supporting a large storage deployment for replicated backups. It’s my hope that this thread (my first!) serves as an introduction and “decision tree” for my pending project. I anticipate referring to it in further threads/conversations. I apologize in advance for any rambling.

With that out of the way, some overview on my intentions and decision points:

• The initial workload will be replicated and direct backups from Dell AppAssure (DAA) servers
  I need a lot more info on the details of this workload
• I intend to use block level iSCSI delivery to a small number of physical and virtual (DAA) servers
• I have a significant budget to work with and will not spare it - $12k+ is expected and budgeted
• Performance of this deployment is not the 1st concern, but must be sufficient
• Data integrity and my nerves are the highest priority
• Intended usable capacity – starting @ 20TB+ scaling to 100TB+ as business units grow (will span “instances” as required)
• Current hardware plan calls for enterprise class server and JBOD/DAS chassis (nothing new here)
  128gb+ ram, 10GbE redundant NICs, OOB via iLO, all the whistles, dual domain SAS
  Leaning towards SuperMicro for JBOD and HP for server as I have one to spare
• SoftwareSoftwareSoftware!! – This is where 75% of my open questions exist.
  ILLUMOS (OI), FreeBSD, Linux, FreeNAS, NexentaStor....
• I am only 80% committed to a non-replicated storage architecture. No plans for second deployment at this time thus no alternative for backups….of my backups. Exacerbating this issue is the fact that this solution will be customer facing. Customer expectations are TBD.
  
• I do not intend to use deduplication as it’s easier for me to add capacity than to battle poor performance.
• Compression will be flirted with but the previous statement stands.
• Power/Cooling is not an issue as I own my datacenter (don’t tell the CEO I said that).
• I'm a Unix/Linux novice but been around CLI for over a decade and appreciate it.
• I'm not afraid of ANY learning curve yet acutely aware that community support is essential for me. A large installed user-base also makes for easier Google'ing.

These are just some of my thoughts and all of them are open for conversation and debate. I’ve been on this topic for a few weeks and come up from knowing nothing of ZFS and my commodity storage options. Previous to this all of my storage experience is with EqualLogic and NetApp. I’ll be opening up numerous threads and linking back here as appropriate.

tldr; I'll be back soon and annoying many of you with additional questions.

 

[jgreco]

Look at the guy who asked about VMware View. This was all answered in that thread, IIRC.

Why wouldn't you just toss a Supermicro MB in that JBOD unit? Just more bits of hardware to worry about if you use a separate server.

 

[framewrangler]

I'll look for that thread, thanks.

As far as combining a compute node and JBOD chassis goes I was looking at higher 45 and 60 disk JBODS and also trying to keep a high "expansion slot" count in the server ino order to ensure I'd have room for future HBAs and 10GbE adapter as required down the road.

 

[jgreco]

Heat is the enemy of hard drives. Suggest sticking with 4U/24 bay. Higher density involves air flowing past multiple layers of drives.

Look at the E5 boards from SuperMicro. Six PCIe with built in dual 10GbE and LSI2208, 1.5TB max RAM... how's your HP stack up?

 

[framewrangler]

The HP is not a given and my first choice was a Supermicro board with 10 I/O slots and dual sockets. RAM capacity was north of 1TB as well. I'll try to dig up my quote and SM part number. They make SO MANY DAMN PRODUCTS.

 

[framewrangler]

This was my original spec server: Supermicro SuperServer 6037R-TXRF
http://www.supermicro.com/products/motherboard/Xeon/C600/X9DRX_-F.cfm

 

[cyberjock]

To be honest, as soon as you said "Exacerbating this issue is the fact that this solution will be customer facing. Customer expectations are TBD." I was ready to close the browser tab.

That makes a big f'ing difference in how you build the server. If you plan to provide NFS to ESXi or iSCSI you're going to need to look at far different hardware requirements to ensure you have low latency pool I/O than if throughput is more important. If it were a very large home server you might not care about a few ms of latency when streaming videos or opening a word file.

 

[framewrangler]

Cyberjock, I've seen your work on the forum here and hope you stick around to see that I'm actually trying to do this right. I'll need your guidance along with that of the rest of the community.

That said, this solution will NOT be supporting VM datastores of any type. I have a rack full of NetApp for that purpose. The customer facing comment was in relation to a lack of replication of this solution and my previous comment about performance needing to be sufficient but not top priority still stands.

 

[jgreco]

If you were to have a separate head unit, the 10-slot board might be an interesting choice. But from my point of view, it just eats rack space. Sadly you can't put that board in a more useful chassis.

We got an X9DR7-TF+ here and it is very interesting. Using the LSI2208 strictly for SLOG with a BBU, you still get dual 10GbE and six slots of PCI-e, which should allow the attachment of a rack full of disks AND another dual 10GbE card.

 

[framewrangler]

I think I'm pretty well set on that SuperMicro board with the dense I/O slots. I'm ok with expending 3u on it as I have a dedicated rack for this deployment. Admittedly I'm still weighing my options on the actual JBOD chassis. I understand cooling is more of a concern with a denser chassis but will place some confidence in the engineers who design and QA these things in this particular case - and my $100K bank of chillers in the backyard ;-)

I'm in my final stages of spec'ing hardware and also have had preliminary conversations with Dell about their application's I/O profile. the word of the day is RANDOM. I'm not fully confident in that answer and definitely need to review for myself.

The current questions I'm researching are:

• Dual domain SAS (is it a valid option on my budget and applicable)
• Validating by HBA selection (fun topic here)
• Selecting my 10GbE NICs (base-t wont work for me due to distance to 10GbE switching)

 

[framewrangler]

Just a quick update.

I've purchased and initially configured the following hardware for this project:

1 - JBOD - SC847E26-RJBOD1
http://www.supermicro.com/products/chassis/4U/847/SC847E26-RJBOD1.cfm

1 - Server - 6037R-TXRF
http://www.supermicro.com/products/system/3U/6037/SYS-6037R-TXRF.cfm

1 - X520-SR2 10GbE Dual-Port Server Adapter (82599ES)
http://www.intel.com/content/www/us/en/network-adapters/converged-network-adapters/ethernet-x520.html

13 - HDD - WD4001FYYG - 4TB SAS HDD
http://www.wdc.com/en/products/products.aspx?id=580

128 GB ECC RAM (64GB Per CPU)

2 - Intel XEON E5-2643 v2 @ 3.50GHz (Hex Core)
http://www.intel.com/content/dam/www/public/us/en/documents/product-briefs/xeon-e5-brief.pdf

1 - SanDisk Cruzer Fit 16 GB USB Flash Drive

2 - LSI SAS9207-8e
https://store.lsi.com/index.cfm/Host-Bus-Adapters/6Gb-SAS-9207-Adapters-PCIe-3.0/LSI00300/


After basic testing I can report that all hardware is fully recognized and functional under 9.2.0 release with zero intervention on my part. I've still got a lot of testing and validation ahead of me but wanted to share what I've found so far. One of the more pleasant surprises was that gmultipath was completely automated and showed all active/passive routes out of the box.

 

[jgreco]

Well gmultipath is sufficiently obscure that if you knew to look for it, your clue level is high and I suspect you're going to be just fine and aren't likely to have any problems you can't figure out pretty easily.

 

[framewrangler]

jgreco, Today at 6:36 AM

Well gmultipath is sufficiently obscure....​

Thanks for the vote of confidence. I'll continue to update this thread as my testing continues.

 

[louisk]

I hope you did the math behind this. So frequently people don't analyze the numbers. How much are you needing to push in what time frames? What IOP requirements are there to meet these timeframes? The list goes on.

 

[framewrangler]

louisk, Yesterday at 7:16 PM

I hope you did the math behind this. So frequently people don't analyze the numbers. How much are you needing to push in what time frames? What IOP requirements are there to meet these timeframes? The list goes on.​

​

​

LouisK, you make valid points and certainly highlight a shortcoming in my planning up until this point.​

​

​

In this particular case I have the benefit of being in a greenfield environment and am dealing with a fledgling business unit. The existing storage needs are being met and exceeded by an appliance (freenas) of significantly lesser quality and performance. That anecdotal evidence and the fact that my current build is scalable is where I'm filling in the gaps in my knowledge. I've contacted Dell/AppAssure twice looking for information on their products workload and have received little to nothing in return. With more time and resources I'd be able to do those forensics on my own...​

​

​

Long story short, I'm sure this build will get us off the ground and scale well, to a point. I admit that the future pain-points and bottlenecks are unknown to me. If this project and storage were that critical I'd simply be throwing another $100K at my NetApp vendor.​

 

[louisk]

I wouldn't expect much information from the vendor on that. I would rely on the information you can gather (from things like vmstat/iostat running for a week on each host connected to your storage). You can then plug this information into a spreadsheet to crunch numbers and find out what your actual io requirements are. If you feel like impressing management, you can also generate graphs from your numbers.

 

[framewrangler]

The connected host machines are all Windows VMs using the native iSCSI initiator so my toolset will probably have to change. Is there a way to use my existing FreeNAS instance which is actually in production to pull some of these numbers? The workload will be transferred from that old appliance directly to this one so the loads are identical.

 

[louisk]

If you measure from FreeNAS, you may not be able to determine which hosts consume more resources than others. This may or may not be important. I would think windows would have a way of measuring disk io for a block device, but that's probably because I'm used to FreeBSD.

 

[framewrangler]

There is only one host connected to my FreeNAS box at this time so I don't have to worry about multiple hosts. Thanks for your input thus far. I'll dig into my more specific questions now and report back with any remaining questions.

 

[jgreco]

On the FreeBSD side you can use "iostat" or "gstat" for per-disk monitoring, or "zpool iostat" for pool level monitoring (salt with options until the flavor is right).