New Hyper V HA Build

[Steven Sedory]

Hey there,

I've built a few production SANs with freenas, all which have been virtually flawless for the past five years or so.

It's been a while though (about 2.5 years) since my last SAN, and I wanted to bounce my setup off the community. Hopefully someone out there can give me the green light.

We're going to use 2 or 3 cluster nodes on Server 2012 R2, probably Dell R630's w/ 2x 2.6 GHz 10 Cores and 256GB RAM each. They'll have dual SFP+ which we'll twinax to the SAN (we will save on power with no switch, which is doable in this small cluster size).

The SAN will probably be a R730xd with the 24x 2.5 HDD config.

We have about 40 VMs that we need to migrate over to this new cluster from a handful of separate inefficient hyper v nodes, most of which have their own local storage.

What I'd like to do is setup this new R730xd SAN with 15x 600GB 10K SAS drives, which I'll split up into 3x RAIDZ pools. We're going to use the remaing 9 bays for individual SSDs for a few customers that are paying for those.

We will use a couple Intel PCI 750's for the SLOG, and another for l2arc cache.

My main question is, would we be better off, performance wise, striping those three RAIDZ pools together? My thought was to put each of our three nodes on a segregated iSCSI portal pointing to each pool, to separate traffic. But I wonder if striping and connecting all three nodes to the same portal/volume, would be better.

There's obvously much more to say about this, but any input and feedback would be invaluable. Thanks all.

 

[Stux]

Depending on traffic, will the P750s have the necessary endurance? Might have to step up to their big brother

I thought SSD costs had more or less reached parity with 10K drives. Should you not investigate using cheap SSDs to build the array instead of expensive HDDs?

 

[Ericloewe]

I thought SSD costs had more or less reached parity with 10K drives.

For $/IOPS, yeah. $/TB, not quite, last I checked (10K RPM drives are not a particular interest of mine).

 

[Steven Sedory]

Thanks for the reply.

When you say endurance, what metric should I be looking at?

As for SSD, I am a little concerned when it comes to SSD's... Can you point to an equally resilient (to good ol 600GB 10k or 15k sas) SSD?

Also, to all yall out there, is there a good guide to estimating I/O that each of the pools can do, so I can estimate my max amount of VM's on a given pool?

 

[Stux]

Each vdev has the IOPS of a single device in the vdev.

HDs = 100+

Good enterprise ssd 10000+

Re sequential up

Mirrors have the read speed of a single device multiplied by the number of disks in the mirror. And write speed of a single disk.

Raid has n-p for read and write.

 

[Stux]

And Regarding resiliency, it really only has to be resilient enough to not exceed your redundancy.

But you're off the beaten path here

 

[Stux]

Regarding intel P750, the key is TB written per day. Every sync write will be written to the slog. If every write is sync, then if your writes to the array exceed that, you'll exceed the 750s capability.

This metric is the primary difference between the 750 and its big brother

 

[Steven Sedory]

Great, thanks. Understood on sync writes. So would that be the P3700 800GB, or? And that due to the higher seq read/right speeds?

Also for HBA, we were going to do 9300-8i. Would setting up a second for redundancy be overkill? If not, any guidance on configuring that?

 

[Steven Sedory]

Up to this point, thank you for your input, I'm going to start a separate post asking more specifically about the SAN.

 

[Stux]

Their are a few different options

http://www.intel.com.au/content/www/au/en/solid-state-drives/intel-ssd-dc-family-for-pcie.html

800GB is probably at least 10x larger than you need for a slog. So the 400GB might be better

 

[Steven Sedory]

Thanks

 

[Steven Sedory]

Each vdev has the IOPS of a single device in the vdev.

HDs = 100+

Good enterprise ssd 10000+

Re sequential up

Mirrors have the read speed of a single device multiplied by the number of disks in the mirror. And write speed of a single disk.

Raid has n-p for read and write.

I understand this, at just the disk level, but what about when you introduce 256GB RAM into the equation (Arc cahce) and the PCI card for L2Arc? Is that because the Arc mainly helps for ONLY quick writes to the pool (goes onto arc cahce, then writes to disk later), but not reads (has to pull from disk)?

I thought one of the big benefits to ZFS is the milt-teired storage abilities?

 

[Ericloewe]

ARC has absolutely nothing to do with writes. ARC is for reads only.

 

[Steven Sedory]

ARC has absolutely nothing to do with writes. ARC is for reads only.

So the only way to increase write speed to a pool is have a slog (when doing sync writes) and have lots of vdevs? (and of course faster drives in those vdevs, i.e. 15k and SSD)

 

[Stux]

It's complicated :)

You don't have to wait for a write to finish, it's thus instant, except for sync writes, they need to be written somewhere and you have to wait for them. Fast slog will mitigate that.

But, if you're sustained write speeds can't actually be written to your array (sprayed across the disk ) fast enough, over a transaction group (2-5 seconds) the write will eventually block to clear the incoming buffer.

Reads you have to wait for, unless they're in ARC or l2arc.

If your whole working set fits into arc/l2arc then you'd not have to worry about it.

 

[Steven Sedory]

It's complicated :)

You don't have to wait for a write to finish, it's thus instant, except for sync writes, they need to be written somewhere and you have to wait for them. Fast slog will mitigate that.

But, if you're sustained write speeds can't actually be written to your array (sprayed across the disk ) fast enough, over a transaction group (2-5 seconds) the write will eventually block to clear the incoming buffer.

Reads you have to wait for, unless they're in ARC or l2arc.

If your whole working set fits into arc/l2arc then you'd not have to worry about it.

Good stuff. Thanks for clearing that up.

 

[Steven Sedory]

You don't have to wait for a write to finish, it's thus instant

Can you elaborate? Sync writes aside...is that because the Arc Cache (RAM) is indeed used for that?

 

[Ericloewe]

No, ARC is not used for writes. Period.

Sync writes are written to ZIL and then flushed when appropriate. SLOG offloads the ZIL to a separate device.

 

[Steven Sedory]

Thanks for clarifying. Where do no sync writes go?

 

[Ericloewe]

RAM.