Hardware recommendation needed

Status
Not open for further replies.

tazinblack

Explorer
Joined
Apr 12, 2013
Messages
77
Hi folks,

I'm looking for new hardware for a freenas installation.

These are my ideas:
  • suitable for 19" rack
  • 12 hotswap disk slots
  • redundant power supply
  • disks direct connected without raid controller or expander if possible
  • 2 - 4 network ports 1 Gbit/s (intel preferred)
  • ECC RAM
  • enough RAM sockets for 32 GB (expandable to 64 GB)
I would like to have 10 SATA disks at raid-z2 and two small fast SSDs for write cache since it will bis used as vmware storage connected by NFS. I'm not sure what's the best way connecting the disks. I have one installation using a raid controller only as jbod but there freenas is not able to show the disk serials. So this makes problems when you have to replace a disk. I don't know if this is a problem with the raid controller or the disks.

What's the best way to connect the disks for maximum performance and which hardware is suitable for that.

Can someone give some recommendation here?


Thanks
tazinblack
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
No. Just no.

We don't build servers for people. You put up your build and we critique it. If you want someone to give you a parts list, feel free to find other threads that have other builds.

Hint: If you ignore our stickies when you post your hardware we'll shove them in your face. So you might as well take our stickies to heart, because anything you ignore from the stickies we're going to stick back in your face. ;)
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
To get the redundant power and expandable to 64GB, you're probably looking at E5.

It is probably a little more than you need, but I've been eyeing the ASRock 2U units, they have a pair that look like they've been basically designed for ZFS. Sadly I have no actual experience with them. I am contemplating picking up a pair of them for ESXi nodes. The only thing they're missing for that use is an LSI2208 (has LSI2308 HBA) and of course the E5's are mostly ridiculously expensive except for maybe the 1620 or 2609.
 

tazinblack

Explorer
Joined
Apr 12, 2013
Messages
77
Sounds good. Think I would take E5-2609V2 since it has only 80W TDP. Or do you think it has not enough power.
Problem will be the hardware dealer for ASRock 2U.
What's the matter with LSI2308 I found it on the support list at freebsd?
I'm also not sure if a raidcontroller with cache and bbu and two disks would be better that two small mirrored ssds for zil.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
The 2609 is a contemptible CPU, just so you're aware. It is pathetically slow, lacks turbo boost, and lacks hyperthreading.

It is a great CPU to fill a socket while you save up for a real CPU but that's about it.

For an ESXi node, an LSI2308 HBA is kind of crappy. ESXi wants all writes to be written sync, which for a Direct Attach Storage (DAS) datastore means a RAID controller with BBU write cache. Otherwise it is kind of draggy slow.

I have the totally awesome Supermicro X9DR7-TF+ here in an SC846 chassis running ESXi with an E5-2697v2. It is very pleasant and has lots of room for expansion. Trying to stick one in a 2U for ESXi with FreeNAS is problematic though.

The ASRock 2U's are very tempting because with an add-on LSI2208+BBU, they're pretty hot. Even hotter would be seeing if I could get ASRock to build units with the EP2C602-2T2OS6/D16 board which would give you 4x 10Gbit connectivity and the 2308 AND still a single PCIe slot available while only needing a single CPU. I'm thinking dual-node ESXi cluster hotness.
 

tazinblack

Explorer
Joined
Apr 12, 2013
Messages
77
ok, as ESXi server I agree. But for a freenas used for storage for ESXi connected by NFS a E5-2697-V2 would be a bit of overkill if you have only 2 ESXi Servers connected and about 15 VMs <20 users.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
There again we have this odd thing where people characterize SAN storage by: 1) number of attached hosts, 2) number of VM's, or 3) number of users - or in this case ALL THREE. Makes no sense.

Honestly I can kill most storage arrays if I try. Key word is often "IOPS".

1) The number of attached hosts is meaningless so long as it doesn't exceed the capacity of the iSCSI or NFS server in terms of connections and resources.

2) The number of VM's is irrelevant; their IOPS usage is extremely relevant though. Even a large filer based on platters may be very IOPS-constrained. I can run hundreds of small VM's on a small filer as long as the IOPS rates are low to moderate. I can't run even a single VM with high IOPS rates in some cases.

3) Users are like VM's. Varies based on the load.

The 2697v2 is a bit of an overpriced toy, but I wanted an expandable ESXi node that could host a large FreeNAS, a 12 drive RAIDZ3, and have resources left over for significant VM's etc. Going about it the way I did, the 2697 is not a great choice for a high performance FreeNAS but that wasn't the design goal - the 2697 is nearly untouchable for overall performance and it lives within its 130W TDP. For pure FreeNAS use I'd go with the E5-2643v2.
 

tazinblack

Explorer
Joined
Apr 12, 2013
Messages
77
I totally agree with you. But this sometimes helps to imagine the workload.
In otter words: How much IOPS are possible on a freenas with i.e. 10 SATA 1 TB 7200 RPM disks running RAIDZ2 with i.e. 2643v2 CPU.
I think nobody can tell this either. The only thing that counts here is experience in sizing a freenas.

And if you are not so experienced with that there is only the chance of asking someone who has that experience.
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
IOPS are a very vague method for measuring performance. IOPS over the LAN versus IOPS to the pool aren't the same thing, and almost never actually are. In some cases, multiple LAN packets with data to be written might count as a single "IOP" to the pool.

Frankly, the second you want to start talking about IOPs or have any kind of care to the IOPS of your pool, you are instantly talking mirrored vdevs. PERIOD. There's no cheating this except with very very expensive hardware. And most people would be laughed at because the cost of the hardware necessary to get good IOPS with anything that isn't mirroed is so outrageously expensive you were probably better off financially to have gone mirrored.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
I totally agree with you. But this sometimes helps to imagine the workload.
In otter words: How much IOPS are possible on a freenas with i.e. 10 SATA 1 TB 7200 RPM disks running RAIDZ2 with i.e. 2643v2 CPU.
I think nobody can tell this either. The only thing that counts here is experience in sizing a freenas.

And if you are not so experienced with that there is only the chance of asking someone who has that experience.

I can almost never recall a time I've seen this help "imagine the workload," because invariably it is a sign that the people talking about their VM environment really have no idea how to characterize their workload. This is one reason that VM environments can end up being totally sucktacular; someone "heard" ZFS was "great" for this so they set up an 8GB 4-disk RAIDZ2 filer and fill it 80%, which follows one set of guidance available for FreeNAS but in reality they failed to identify that one of their VM's was a massive Exchange server. So now you have FAIL on both sides of the equation.

As for the above question, the answer is "assume approximately the same IOPS as what is possible on a single 1TB 7200 RPM disk," though in practice for a complex workload the actual number may well come out a little higher than that, and the CPU is basically irrelevant if we're talking any kind of E5.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
IOPS are a very vague method for measuring performance. IOPS over the LAN versus IOPS to the pool aren't the same thing, and almost never actually are. In some cases, multiple LAN packets with data to be written might count as a single "IOP" to the pool.

Frankly, the second you want to start talking about IOPs or have any kind of care to the IOPS of your pool, you are instantly talking mirrored vdevs. PERIOD. There's no cheating this except with very very expensive hardware. And most people would be laughed at because the cost of the hardware necessary to get good IOPS with anything that isn't mirroed is so outrageously expensive you were probably better off financially to have gone mirrored.

All wound up this morning?

IOPS to SAN is kind of like horsepower to cars. It isn't a perfect measurement but it is what we've got to work with. For spinny rust, anything less than a doubling of IOPS is probably not all that helpful to someone designing a system, mostly because the IOPS numbers for even a massive spinny rust pool can be relatively low. IOPS for sequential operations are high, of course, but for random seeks are bad. So I usually think of an "IOPS" on spinny rust as requiring a seek event; that's a pessimistic assumption but better than making the opposite assumption. On that assumption, you can take the speed of the drive, shave off the two zeros at the end, and the drive itself ought to be capable of at least that number of IOPS (maybe even 50% more).

So a RAIDZ2 of four 7200RPM drives might only be capable of ~~100 IOPS. Which is pessimistic, assuming a heavy seek workload.

But the thing that kind of gets people is that a mirrored vdev pool of 200 of those same drives may be capable of about 10,000 random IOPS. Which is really still not all that much. So then we're back to characterizing workload in order to identify whether maybe there's a lot of sequential IOPS, or maybe a small enough working set that we can use ARC/L2ARC, etc. Unfortunately, the real problem is that the metrics that virtualization systems tend to give you are overall IOPS, not specified as "seek-inducing" or "sequential." So for example on a pair of Seagate 5900 RPM NAS disks in mirror that I'm sure cannot go north of 100 random IOPS, I see it peaking over 1000 sometimes.
iops.JPG

So over the years I've become rather cynical and wary of discussions of IOPS, because in the end it is hard to produce meaningful and useful numbers. For a SAN, I can figure out both the lower and upper bounds for what a given array ought to be capable of, but actually understanding what a VM's workload is can be a frustrating and complex task, probably more suited to labbing out and actually trying various things, and deriving meaningful conclusions from that. Still, when comparing apples to apples, IOPS gives you something to compare.
 

tazinblack

Explorer
Joined
Apr 12, 2013
Messages
77
Sure you can't tell about the workload if you only know about the number of VMs and if there is LAN in between it makes it not easier.

Ok, so for the example from above: 10 SATA disks (~75 IOPS each) at RAIDZ2 would mean 8 data disks and 8 * 75 IOPS = 600IOPS for this constellation.
With 15K SAS disks this would be around 1400 IOPS.

That's the same calculation as it's used for any kind of storage since the physics underneath are always the same as long you have the same disks.

But what I don't know is the behavior when the ARC and maybe ZIL on SSDs come into play.
So the thing what people do if the behavior of the workload is only vaguely known and the functional interaction of the storage to build with the hypervisor is also unknown they ask for examples.

So maybe this is the one of the directions to go.
So maybe you can tell me a bit about how are your ESXi boxes run against freenas using NFS as connection.

Did it bring a lot of performance using any kind of ZIL (disks, SSDs, disks behind raid cache with BBU).
Or would you say ESXi and freenas is not a good combination at all, use a different kind of storage for that.

And then the next question would be something like how many disks do you have inside that box and how much RAM.
How much VMs do you run on that and how much users do you have.

Sure is that a very vague knowledge of the world but you have to start somewhere and learn from your mistakes and make it better next time.

So I hope this is not annoying you too much.
But it would be great to get some hints.

And by the way, thank you for this discussion!
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Ok, so for the example from above: 10 SATA disks (~75 IOPS each) at RAIDZ2 would mean 8 data disks and 8 * 75 IOPS = 600IOPS for this constellation.

[...]

Sure is that a very vague knowledge of the world but you have to start somewhere and learn from your mistakes and make it better next time.

And ... FAIL.

The IOPS behaviour of a RAIDZ vdev can be assumed to be capable of about the same number of IOPS as the slowest component device. It is absolutely NOT the aggregate IOPS of the component data disks.

This is also true for mirror vdevs, though theoretically ZFS ought to be capable of independently reading from both halves of a mirror (last I checked, it doesn't, and interleaves read requests for improved throughput).
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Or would you say ESXi and freenas is not a good combination at all, use a different kind of storage for that.

Depending on the cost and time you are willing to invest, this may be true. Many people are unprepared for the amount of time and/or money involved in building a good working server to host dozens of ESXi VMs. I've done contract work with many people and some of them ran 30-50VMs(mostly service functions like AD, email, etc.) and had no performance problems with a server that cost around $5k. The number of users wasn't a good metric for that individual because the services weren't really setup for a number of users. The workload from day to day within that company was the biggest factor. The person with 30-50VMs was not able to find a limit except that he could saturate all 4x1Gb links simultaneously, so he was very happy to have paid me for a few hours of my time.

The thing to keep in mind is that you need to build the system to work for your desired workload. Set some boundaries for what you are expecting and how far you are willing to go. For one user, 8GB sticks of RAM might be a good start, for others, 32GB sticks were the starting point. You have to look at many factors and put a kind of idea of what you think the absolute minimum you are likely to need, what you are expected to need, then what you might need for heavy load. Then, based on that, stop and do hardware shopping. Generally, once you want to run more than 1 or 2 VMs you are talking major performance penalties if you don't design the system around running VMs. And the second you decide to design the system around running VMs, the cost increase rapidly before tapering off

There are 2 kinds of people that regular show up here with needs/wants that involve VMs.

1. Like I said in the previous paragraph, the cost increases rapidly when you want to do more than 1 or 2 VMs. People get downright pissed when the build a $1500 server and then can't do 2 VMs with good speed. Many spend weeks or months with an underpowered and unacceptable hardware setup and try to tweak it to work. Almost nobody ever ends up happy. It's extremely frustrating for someone that doesn't know what they are doing and just wants the end product to work. For the actual financial costs of more than 1-2 VMs it's a line that once crossed means higher costs that most don't want and will try to tweak it to work. 99% give up in agony and frustration because there's no silver bullet to cover wide ranges of situations. These people often accept their crappy performance, bail on ZFS as a solution, or take very dangerous(for their data) options like setting sync=disabled.
2. Others already know they want to run dozen(s) of VMs and are looking at building servers with a purpose. At this point, most people find it just faster and easier to get help from someone like me that does contract work, or just walks away from ZFS when they read the dozens of threads that give no answers to the problems.

Normally, when someone wants to do VMs, I tell them not to consider <64GB of RAM, expect to need a ZIL and L2ARC appropriate for your server specs(note that you can have too much of both, so right-sizing is somewhat important otherwise you spend money on resource that are unusable) and to expect mirrored pairs for whatever total disk space you want. There's just no building it for "cheap". I've done builds that cost as little as $3000 and as much as $6-8k for the hardware. They almost always have some hardware to reuse, that helps the cost. A simple 4U 24 drive server case is an instant $1k, so just reusing a few parts can be a very big cost savings.

Now you see why I do consultation services.. too many people would rather pay someone some cash to give them a parts list that is appropriate for their need than try to play these games of buying too much or not enough hardware. In almost every case where the owner tried to do it himself and then came to me he spent more than $1000 on hardware that was unusable for his server and was literally shelved for future use. They don't like hearing that, but the solution to the problem isn't like the windows world... you can't just keep throwing more hardware at it. It's important that you throw the *right* hardware at it.
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Have I told you how much I've enjoyed watching you go from total n00b to ZFS expert?
 

cyberjock

Inactive Account
Joined
Mar 25, 2012
Messages
19,526
Have I told you how much I've enjoyed watching you go from total n00b to ZFS expert?

Nope. Do you really think that?
 

jgreco

Resident Grinch
Joined
May 29, 2011
Messages
18,680
Well, mostly, yes. You have a little bit too much inflexibility at times, but compare what you wrote above to what you were asking not even two years ago:

http://forums.freenas.org/index.php?threads/removing-a-vdev-while-keeping-data.6672/

I don't know how it couldn't be pretty clear that you've come a long way. I believe you've also gotten at least a little bit into virtualization, which is very good.

I've been called a bunch of things over the years, not all of them complimentary, but a well-regarded friend in the industry has described me as a generalist. This isn't entirely accurate, but I do have the knack of being able to do a lot of different things, because I'm not really super-invested in any one thing. That having been said, storing massive amounts of information and building appliances are both some of my stronger areas of expertise.

For most stuff, I can usually differentiate between bulls***ters and people that actually know what they're talking about, but especially in an area where I have some expertise, it is not horribly hard to identify those who really know (and probably know more than me), those who did homework and tried to learn, and those who are simply looking for information handouts. I might answer questions from that last class of people. I like to point people in the middle group in the right direction, sometimes with a giant shove.

I've given you a giant shove or two, one of which was off a cliff (ESXi/VT-d!). You flew. So I guessed right.

And I enjoy that process, it makes for a great opportunity to discuss fun tech with smart people who simply need some introduction.
 
Status
Not open for further replies.
Top