New SMB build for VMWare

[Jeremy Lea]

Hi all,

I was looking for comments on a new build. I am replacing a VMware Essentials Plus cluster, and was going to use their VSA software, but that was killed between when I had to put a quote in and when I got the money (and replaced with vSAN, which blows the budget). I then considered a Synology device, but the cost compared to FreeNAS is not good... I am very comfortable with FreeBSD and ZFS (reg@), but I tend to go with more mainstream options in case someone else has to take over admin duties from me. However, in this case I can't see the benefits.

I work with a small research group, where I am the defacto IT person. I can only replace hardware every six years or so, and it typically takes 18 months between when I have to put together a budget and when I actually get money. However, I can get money for upgrades (more RAM, new HDD, etc.). As a result I like to buy newest generation hardware and make sure it can be incrementally upgraded if needed. I have a nice budget, but I also have lots of things to do with it, so I don't want to waste money on things that won't be used.

We currently have a 3 machine VMware cluster that is a bit cobbled together. I have around 25 VMs, some of which are not active, and they are using ~1TB of disk space for the OS disks, about ~2TB of critical data files, ~1TB of database storage (in SQL Server and PostgreSQL) and ~4TB of backups of various types. I also have ~60TB of not so critical data in a Dell MD3000+2xMD1000, which I'm not going to touch. The storage is split between local storage in the hosts and the MD3000, with the VMs mostly on a shared LUN in the MD3000 to enable vMotion. I currently have 64GB of RAM in the cluster, which is my constraint for more VMs. I don't generally have HA requirements (we don't make money on IT), but a bunch of people notice if things go down for extended periods. Mostly the storage is quiet, but it does need to be fairly snappy when it is needed (and needs to still be considered snappy in six years time). Mostly, I just need to make sure that critical data never disappears ;-)

I plan on replacing two of the ESXi nodes in the cluster with diskless Haswell based machines (in a four node 2U machine with two nodes for VMware and two nodes for other compute tasks - still deciding on ASUS RS720Q-E8-RS12 or Intel H2312XXKR2/HNS2600KP). The one machine handling the MD3000 will stay as the third ESXi host, mostly because it will be to difficult to unravel the mess of 2TB raw disk mappings that I had to do to make that 60TB of data available on the network... I don't want a repeat of that.

So anyway, on to the FreeNAS build. I'm looking at a mostly Intel box:
Chassis: Intel R2224WTTYS+AXX1100PCRPS PSU
Controller: Intel RMS3JC080 (LSI SAS 3008 based) and Intel RES3TV360 SAS Expander
CPU: 2x Intel E5-2620v3 (maybe E5-2609v3?)
RAM: 2x Crucial 32GB DDR4 kits (4x8GB) (CT4K8G4VFS4213) (maybe 4x?)
OS: 2x 80GB SSD (probably Intel 530)
VM Storage: 8x480GB SSD (mirrored, stripped) for the VMs and the databases (Intel 530)
Data Storage: 16xWD10EFRX (1TB 2.5in Red drives), in two RAIDZ2 pools. I might switch some of these for 2TB (ST2000LM003) drives, and make two RAIDZ3 pools (9x1TB, 6x2TB).
Network: 2xNetgear S3300-28X (GS728TX), stacked, with the storage multi-pathed via two VLANs.

I don't think I'll need any SAS disks, and with the main IO load on the SSDs I don't think I'll be needing special ZIL/L2ARC devices. I'll probably dedup the VM storage, because it has a lot of copies of Windows. Currently the build price of the box is about $8000. The hardware is mostly available now, but it looks like there might be some lag because the chassis has just been released. Without disks, this is about $4,800, which is ~$800 more than a Synology RS3614RPxs which is what I was considering - but it is a much nicer box.

I will stick to backing up within the VMware environment, although I might consider using ZFS send to make my offsite backups. Also, I will use iSCSI for everything. I might present some of the storage (e.g. for SQL Server) directly to Windows rather than go via VMware.

Some questions:
1. Should I be considering Enterprise class SSDs (at ~twice the cost)?
2. Could I use the cheaper CPU? It's only ~$300 extra for the 2620s, but I feel like the CPU will be mostly idle anyway...
3. Should I put in more than 64GB of RAM?
4. Does anyone know of issues with this hardware? Generally I find FreeBSD works well on brand new server hardware, even if it doesn't support all the newest features.
5. I kind of like having everything virtual. Should I consider running ESXi on this machine with FreeNAS on top of that? I considered for a while buying two boxes like this and make on physical server and a passive virtual server on the other with hastd running to keep a backup. This would be more like the VSA architecture I originally planned.
6. Does anyone have any ideas on iSCSI<->Windows performance vs iSCSI<->ESXi<->Windows?
7. Should I just buy a TrueNAS box?

Any other thoughts are welcome.

-Jeremy

 

[Ericloewe]

(Small side note: HOLY CRAP! Five expanders for 400 bucks? SAS3? If this is legitimate, it's an incredible deal.)

1. You'll definitely want Intel S3500 or S3700s. The Intel 530 uses a SandForce controller - those things are on the unreliable side and next to useless for incompressible data.
2. The E5-2609 v3 sounds a bit slow.
3. More RAM will definitely help a lot. Throw as much RAM at the thing as you can.
4. You're missing the all-important motherboard in the description. SAS3 stuff is still not quite as mature as SAS2.
5. Virtual is generally a bad idea
6. I don't really understand what you mean. Do you mean CIFS vs iSCSI performance?
7. TrueNAS might be a good idea, since you can get something that you know for sure will work. I hear their prices are quite reasonable (though I've never asked).

 

[marbus90]

In these parts of the interwebz Supermicro is held high. IPMI comes standard with their Quadnode Chassis, also you get to choose between 3.5 and 2.5", SATA only, LSI 3008 or 3108, 2xGbe/2x10Gbase-T/2xGbe+Infiniband, some even have NVMe options. Almost all Supermicros come with the option to add 1-2 SATA DOMs for the OS, basically reliable fast USB-sticks with SATA interface which don't block the 2.5/3.5" bays. By the way, ixsystems can quote you those systems too.

FreeNAS does not support High Availability, that would be the TrueNAS part. But on my request ixsystems wasn't able to offer an supported upgrade-optimized system as you and I projected it. You might get away with underspeccing it and later adding stuff by yourself as the warranty is done...

Personally I'd always pick the 2620 for Turboboost up to 3.2GHz and afaik Hyperthreading too. Also it has 2.4GHz instead of 1.9GHz baseclock, which comes in handy with Samba etc. Very probably you can be good with a single CPU in each node and add one later. Just be sure to grab the 16GB Registered DIMMs instead of 8GB.

Virtualisation of FreeNAS is a bad idea. Personally I would pick Crucial MX100 512GB and overprovision them, but if you need more enterprisey stuff with capacitors, the Intel S3700 are a good option for primary storage. You might also get away with raidz2, omitting the need for 2 more SSDs. The biggest issue is HDD vs SSD, even a crappy older SSD gives you a nice boost - therefore raidz2 is fine.
With TrueNAS you'd need Dualcontroller SAS SSDs anyway.

 

[cyberjock]

That amazon link has to be a screwup. Intel says that item won't be released until Q2 of 2017... http://ark.intel.com/products/81278/Intel-RAID-Expander-RES3TV360

Deduping storage is NOT useful for VMs. The reason is that the blocks are deduped, not the files in the VM. So unless the files and such happen to align then dedup buys you nothing at all. It will also DRASTICALLY increase your RAM requirements and can seriously increase the latency of your server (which is the biggest problem for storing VMs on ZFS).

You shouldn't even buy 8GB DIMMs as they aren't dense enough. I'd recommend you go with 16GB DIMMs to start.

RAIDZ2 for VM storage is going to be a fail. You will have IOPS limitations as a result... even on SSDs. If you want to run VMs from ZFS storage you're going to need to look at mirrored vdevs.

There's a few other things I'd change, but see my following paragraph.

To be honest, for that kind of server you might want to look at TrueNAS with a support contract. Sounds like you're looking for something that will give you good life and it would suck to drop that kind of money on the hardware and then have problems that you can't fix. If you know you can't spend more than X dollars I'd call the sales guys and ask if what you want can be done for that kind of money. Often people don't realize how expensive ZFS storage for VMs are and when they see the number it blows their hair back. Then they go and build it themselves thinking they'll save some money. But that "saved" money goes right out the window the first time that storage has a problem and your entire enterprise goes down because none of the VMs can be started. Just keep in mind how painful it would be to the company (and potentially your employment) if things suddenly went sideways. If you want to do 5+ years of support I'd make sure you include it all up-front so you write the check and have 5+ years of support already in the bag.

Honestly, if you want to virtualize FreeNAS then you shouldn't be running it for VM storage. Virtualizing freenas is risky. If you want to go with TrueNAS, Virtualizing the OS isn't even an option. (They know better and they don't do risky things like that.. especially when you could hold their feet to the fire with a support contract.)

Just make sure you have this in perspective before you buy your hardware. ;)

 

[Ericloewe]

That amazon link has to be a screwup. Intel says that item won't be released until Q2 of 2017... http://ark.intel.com/products/81278/Intel-RAID-Expander-RES3TV360

Deduping storage is NOT useful for VMs. The reason is that the blocks are deduped, not the files in the VM. So unless the files and such happen to align then dedup buys you nothing at all. It will also DRASTICALLY increase your RAM requirements and can seriously increase the latency of your server (which is the biggest problem for storing VMs on ZFS).

You shouldn't even buy 8GB DIMMs as they aren't dense enough. I'd recommend you go with 16GB DIMMs to start.

RAIDZ2 for VM storage is going to be a fail. You will have IOPS limitations as a result... even on SSDs. If you want to run VMs from ZFS storage you're going to need to look at mirrored vdevs.

There's a few other things I'd change, but see my following paragraph.

To be honest, for that kind of server you might want to look at TrueNAS with a support contract. Sounds like you're looking for something that will give you good life and it would suck to drop that kind of money on the hardware and then have problems that you can't fix. If you know you can't spend more than X dollars I'd call the sales guys and ask if what you want can be done for that kind of money. Often people don't realize how expensive ZFS storage for VMs are and when they see the number it blows their hair back. Then they go and build it themselves thinking they'll save some money. But that "saved" money goes right out the window the first time that storage has a problem and your entire enterprise goes down because none of the VMs can be started. Just keep in mind how painful it would be to the company (and potentially your employment) if things suddenly went sideways. If you want to do 5+ years of support I'd make sure you include it all up-front so you write the check and have 5+ years of support already in the bag.

Honestly, if you want to virtualize FreeNAS then you shouldn't be running it for VM storage. Virtualizing freenas is risky. If you want to go with TrueNAS, Virtualizing the OS isn't even an option. (They know better and they don't do risky things like that.. especially when you could hold their feet to the fire with a support contract.)

Just make sure you have this in perspective before you buy your hardware. ;)

It's possible, but notice how it says "launched", the discontinuation date is the same as the launch date. 2017 is also rather far away for an announced launch (yet close for discontinuation...).
Other retailers seem to have similar prices, which is very weird - 5 SAS3 expanders for the price of one? Hell, I wouldn't be surprised if it was one expander and 5 cables - but ark mentions the five-pack.

Edit: The version with 8 external ports says launched in Q2 2014

 

[Robert Smith]

1. You'll definitely want Intel S3500 or S3700s. The Intel 530 uses a SandForce controller - those things are on the unreliable side and next to useless for incompressible data.

I have been reading a lot of SSD recommendations akin to “anything by Intel is fine.” Did a quick search after reading your post, and, oh boy, those (520, 530) are problematic models.

Amazing, just a few years ago SandForce were trumpeted as the best SSD controllers. Thighs change…

 

[cyberjock]

I've never thought that Sandforce was the best...

 

[Robert Smith]

I've never thought that Sandforce was the best...

2010-ish a lot of people did proclaim so.

 

[marbus90]

And 2012-ish a lot of people didn't proclaim that anymore. Reliable SLOG SSDs are S3700 with their high TBW, write consistency, capacitors to prevent data loss during power loss and reasonably speeds at 4k writes without a queuedepth. For the Dualcontroller TrueNAS the only option are SAS SSDs, the ultimate SLOG SSDs for that are either the HGST ZeusRAM or the HGST s840Z ZIL.

For critical VM datastores I'd tend towards either S3700 or the standard SAS SSDs. If it's home use, one might get away with consumer SSDs, as long as they are overprovisioned so that at least 10% of the SSDs nominal capacity is unused, yielding higher write speeds/IOPS and gaining higher consistency.

 

[Jeremy Lea]

(Small side note: HOLY CRAP! Five expanders for 400 bucks? SAS3? If this is legitimate, it's an incredible deal.)

1. You'll definitely want Intel S3500 or S3700s. The Intel 530 uses a SandForce controller - those things are on the unreliable side and next to useless for incompressible data.
2. The E5-2609 v3 sounds a bit slow.
3. More RAM will definitely help a lot. Throw as much RAM at the thing as you can.
4. You're missing the all-important motherboard in the description. SAS3 stuff is still not quite as mature as SAS2.
5. Virtual is generally a bad idea
6. I don't really understand what you mean. Do you mean CIFS vs iSCSI performance?
7. TrueNAS might be a good idea, since you can get something that you know for sure will work. I hear their prices are quite reasonable (though I've never asked).

Thanks for the input. I'm pretty sure the 5 pack is a typo, even though it is repeated all over the net...

Looks like it's enterprise class SSDs for the VMs. That probably means less SSDs and more HDDs.

I guess what I'll do is buy the E5-2620v3, and see what the CPU utilization looks like. If I can't make them sweat then I'll put those CPUs in my compute nodes and buy some slower ones for FreeNAS!

Yes, I should have said for 128GB I'd use 2x4x16GB (CT4K16G4RFD4213) - with both CPU sockets filled I'd like to have 8 sticks of DDR4. With only one socket, you only want 4 sticks if possible.

Sorry, the motherboard is an Intel S2600WTT (http://ark.intel.com/products/82156/Intel-Server-Board-S2600WTT). Intel C612, Dual Socket R3, 24 DIMMS, Dual 10Gb X540. Lots of nice features. The chassis has 24 2.5in bays in the front and two in the back. It can also take two fixed SSDs internally, and a USB stick and SATA DOM. The chassis can also be upgraded to support 4 NVMe bays (don't ask for a price on that!). The R2312WTTYS chassis has the same motherboard, but 12 3.5in bays in the front, although the board only has 10 SATA/SAS ports without a RAID module. I prefer to have more spindles...

It looks like it is possible to use SAS 6Gb stuff with this chassis, but it looks like it requires some special cables, or maybe even swapping out the backplanes. The older chassis (R2224GZ4GCSAS) is also quite nice, but doesn't have the rear mount bays, and has 4x1Gb not 2x10Gb, v2 CPU, DDR3, etc.

Sorry, I wasn't very clear on the question about iSCSI. The question was should I use the VMware iSCSI stack and then create a VMDK on that, or use the Windows iSCSI stack and avoid the VMware storage layer completely. The FreeNAS machine will only serve iSCSI, no Samba/CIFS. I did some more reading on this, and it seems most people recommend using either a VMDK or a raw device mapping (RDM), so as to allow more use of VMware features.

Regards,
-Jeremy

 

[jgreco]

Deduping storage is NOT useful for VMs. The reason is that the blocks are deduped, not the files in the VM.

This is a questionable statement, and is generally false for most larger virtual machine deployments which can benefit from dedupe, but getting it right is complicated and maybe more effort/expertise than is justifiable. The win is when stuff fits in ARC/L2ARC that wouldn't otherwise have been there. In an environment with a lot of VM's from a common image, it'd be very desirable.

Getting stuff to fit into ARC is a massive win. I was seeing 600 MBytes/sec read performance on a FreeBSD VM hosted on ESXi over a single iSCSI session to a test FreeNAS box ... when the data was in ARC. The usual problem with ARC is that there's never enough of it. L2ARC cut that in half to about 300 MBytes/sec, and going to the pool cut it again to about 130MBytes/sec.

The other problem, though, is divergence. Over time, as patches and updates are performed within a VM, you get less shared data, and increased fragmentation, which increases the trajectory at which performance feels like it's tanking.

As for the OP's other issues.

I'm not particularly convinced that dual CPU is useful except to gain more I/O capacity or memory capacity. I've been finding that this E5-1650 v3 is a fantastic choice for our new VM storage engine here.

 

[Jeremy Lea]

In these parts of the interwebz Supermicro is held high. IPMI comes standard with their Quadnode Chassis, also you get to choose between 3.5 and 2.5", SATA only, LSI 3008 or 3108, 2xGbe/2x10Gbase-T/2xGbe+Infiniband, some even have NVMe options. Almost all Supermicros come with the option to add 1-2 SATA DOMs for the OS, basically reliable fast USB-sticks with SATA interface which don't block the 2.5/3.5" bays. By the way, ixsystems can quote you those systems too.

The most comparable Supermicro system is the 2028U-TRT+, which is spec'ed about the same, expect that it does not support the 2 OS drive bays at the back. Looking online, that would be about ~$500 cheaper than the Intel box, which is significant. However, the sites that sell it all list it as requiring that it be sold was a fully assembled system, so I might have to take their prices on components.

On the other hand, Intel hardware has good documentation. Supermicro don't list all sorts of details about their boxes, like connector types. Not knowing what is coming in the box is fine if you're assembling lots of servers, because you can keep spare cables, and know what to buy the second/third time. For example, I can't see any listed SAS expander for this chassis (although the Intel PCI based one would work - but will the plugs fit...)

(On the compute node side, the closest Supermicro is the 6028TP-HTR, which is ~$1000 more than the ASUS/Intel and also poorly documented)

Regards,
-Jeremy

 

[jgreco]

The 2028U-TRT+ is sold as a complete system by Supermicro due to the fact that it's a little finickier than your typical Supermicro.

All the Supermicro systems that I know of are documented as to what's available, though sometimes you actually have to look up the part numbers or go digging a little on the website.

On the other hand, Intel hardware has good documentation. Supermicro don't list all sorts of details about their boxes, like connector types.

Which is clearly not true, as you can drill down to each part.

For example, the integrated motherboard has

http://www.supermicro.com/products/motherboard/Xeon/C600/X10DRU-i_.cfm

10 SATA3 ports, which in the default configuration of the TRT+ are wired to the first ten 2.5" bays.

The included backplane is a BPN-SAS3-216A which has an entire manual

http://www.supermicro.com/manuals/other/BPN-SAS-216A.pdf

just like the motherboard does, if you go looking for it. But most of the information is right there directly on the TRT's page if you just look for it - especially under the parts list.

Supermicro would install a SAS expander as part of the system if that was part of the ordered system.

Having built numerous Supermicro systems over the years, "not knowing what is coming in the box" is NOT one of Supermicro's failings. I will admit that sometimes I cheat when speccing a parts list because I don't want to work Really Hard(tm). For example, for these two servers I'm playing with, I wanted a WIO-based single CPU box and knew I wanted to base it on the X10SRW-F, so I looked at the servers that integrated it, picked the SYS-5028R-WR, which is essentially the exact server I wanted, except for 3.5" disks, then went and looked at the parts list to make sure that we put the correct riser cards on the purchase requisition and that there weren't any other unexpected gotchas. And that's the lazy way to substitute in a SC216BE26-R920WB chassis in for that. ;-)

 

[marbus90]

The 2028U-TRT+ is really a bit over the top. The 2028R-E1CR24L seems better fitting towards your needs for a storage server (and comes with 2x10Gbe via X540 too). Also you don't need to worry about cables. Just select the CPU, RAM and disks, done. MCP-220-82609-0N is the part number for the rear 2x2.5in hotswap bays.

On another note, I'd rather pick these Enterprise SAS HDDs for your HDD pool: Seagate ST2000NX0263, raidz2 for them is still okay. SAS drives with SAS expanders is just a little bit more thrustworthy. Also Reds aren't certified to be run in such big arrays, which means that you can experience serious performance and reliability issues because all disks are shaking each others heads, leading to misalignment and (recoverable) read errors.
SSDs would be the Toshiba PX03SNF080, at 800GB instead of 480GB you are good with 6 instead of 8 SSDs. Yeah, I know, they don't come cheap compared to the consumer-oriented low price SATA stuff. Also would leave you full 6 bays free to add another HDD raidz2 vdev.

The prices I found for the Asus chassis is in the 2900$ area, whereas the basic Supermicro 6028TR-HTR with 3.5" and Gbe is in the same range. For 2x X540 10Gbe onboard look at 6028TP-HTTR for 3.5" disks or 2028TP-HTTR for 2.5", they should run for ~3800-3900$. Intel was around 4000$ with all compute nodes, Gbe only.
If you'd like mirrored ESXi boot and/or SAS connectivity, the 2028TP-HC0TR is the better option for you. It comes with LSI 3008 controllers for each of the nodes, connecting without expander to the 6 individual bays (which btw use the same trays as your storage system), which adds around 300-350$ to the total for the Quadnode barebone. Without expander you're also good to go with plain SATA disks.

A thing which all Supermicros have in common: support for http://www.supermicro.com/products/nfo/SATADOM.cfm modules. No SSDs or Hotswap bays needed, plugs onto the mobo and done. The storage system would even support two of them, so you can and should use two DOM's for the FreeNAS and mirror them (possible with FreeNAS 9.3 in software). The Quadnode 2028TP-HC0TR would also support these: http://www.supermicro.nl/products/nfo/mSATA.cfm so you could also run two flash modules per node there, however they're both connecting to the Intel SATA ports and therefore not provide redundancy for ESXi boot.

Bear in mind that most switches are still only supporting 10Gbe via SFP+ modules, so you would need to pick XS708E or XS712T switches for the server interconnection. They have 8 or 12 10Gbase-T ports and 1 or 2 SFP+ uplinks shared with the respective RJ45 ports. The 12port model seems a good idea to run 10Gbe for the servers with 20Gbe aggregated uplink.

 

[Jeremy Lea]

This is a questionable statement, and is generally false for most larger virtual machine deployments which can benefit from dedupe, but getting it right is complicated and maybe more effort/expertise than is justifiable. The win is when stuff fits in ARC/L2ARC that wouldn't otherwise have been there. In an environment with a lot of VM's from a common image, it'd be very desirable.

Getting stuff to fit into ARC is a massive win. I was seeing 600 MBytes/sec read performance on a FreeBSD VM hosted on ESXi over a single iSCSI session to a test FreeNAS box ... when the data was in ARC. The usual problem with ARC is that there's never enough of it. L2ARC cut that in half to about 300 MBytes/sec, and going to the pool cut it again to about 130MBytes/sec.

The other problem, though, is divergence. Over time, as patches and updates are performed within a VM, you get less shared data, and increased fragmentation, which increases the trajectory at which performance feels like it's tanking.

The effort is already required to make sure that block sizes/offsets match up and down the stack, otherwise performance is going to suck.

I did some reading and calculations. Even after about 3 years in production my Windows VMs still have ~90% in common (on the OS disks), based on a rough analysis of the dedupe ratio achieved in the backups. The other OSs are similar, because I keep them all at the same version. That means ~400GB of data is actually ~40GB, and most machines do not touch all of that under most situations, so this is well within the size the should stay in the ARC. However, I also found that the reported 1-5GB of RAM per 1TB of deduped files is also way off, since it is based on 128KB blocks, and the VM dedupe would be 4KB blocks. In this case the theoretical memory use could be about 80GB per TB, and will only use 1/4 of the ARC, so the ARC size would need to be ~320GB. However, if the dedupe ratio was close to 10, then it would be closer to 8GB of RAM, which would be a good number. I think I might experiment with this during hardware burn in...

I'm not particularly convinced that dual CPU is useful except to gain more I/O capacity or memory capacity. I've been finding that this E5-1650 v3 is a fantastic choice for our new VM storage engine here.

I hadn't thought of using a single socket chip. As I said above, I was going to get the 2620s and see how they worked, and then put them in the compute modules if I couldn't make them hot. But I was already going to only use a single socket in the compute machines (the other two of the quad machine box, not the ESXi hosts)... The E5-1650v3 looks like a good choice for those machines. Maybe I'll end up doing a bunch of CPU shuffling!

Regards,
-Jeremy

 

[Jeremy Lea]

Bear in mind that most switches are still only supporting 10Gbe via SFP+ modules, so you would need to pick XS708E or XS712T switches for the server interconnection. They have 8 or 12 10Gbase-T ports and 1 or 2 SFP+ uplinks shared with the respective RJ45 ports. The 12port model seems a good idea to run 10Gbe for the servers with 20Gbe aggregated uplink.

The GS728TX has 2 SFP+ slots and 2 10Gb RJ-45 connectors. Any or all can be used to stack them. My plan was to use SFP+ direct attach cables to the ESXi boxes, Cat 6a to the FreeNAS and one Cat 6a cable for stacking. With LACP (supported across stacked switches), and the right VLAN config, I should be able to make the ESXi boxes multipath IO to the FreeNAS box, and still have full redundancy for when the Netgear switches randomly reboot. The third ESXi host will get a 4x1GB to increase its bandwidth, and the compute nodes (the other two machines in the quad box) don't need bandwidth (to the storage - they might one day need direct bandwidth to one another for MPI).

The 2028U-TRT+ is really a bit over the top. The 2028R-E1CR24L seems better fitting towards your needs for a storage server (and comes with 2x10Gbe via X540 too). Also you don't need to worry about cables. Just select the CPU, RAM and disks, done. MCP-220-82609-0N is the part number for the rear 2x2.5in hotswap bays.

I hadn't seen that box. Supermicro has so many machines, it's hard not to miss some! That does look like a good deal, and is a better match in many ways to the Intel box. I couldn't get a good price for it online (it says coming soon on the Supermicro site), but it looks to be ~$400-800 cheaper, which will buy a bunch of RAM.

On another note, I'd rather pick these Enterprise SAS HDDs for your HDD pool: Seagate ST2000NX0263, raidz2 for them is still okay. SAS drives with SAS expanders is just a little bit more thrustworthy. Also Reds aren't certified to be run in such big arrays, which means that you can experience serious performance and reliability issues because all disks are shaking each others heads, leading to misalignment and (recoverable) read errors.
SSDs would be the Toshiba PX03SNF080, at 800GB instead of 480GB you are good with 6 instead of 8 SSDs. Yeah, I know, they don't come cheap compared to the consumer-oriented low price SATA stuff. Also would leave you full 6 bays free to add another HDD raidz2 vdev.

Those would all be great, and only add about $12,000 to the cost... But don't worry, my rack is aligned with the earth's magnetic field, so I won't be getting magnetically inducted precession on the platters, which means I should be able to manage 8 or more HDDs. I'll also add an extra EEPROM to the sacrifice. Seriously, WD has no idea how much vibration 5 or 8 disks have, because that is much more a function of the overall environment...

The prices I found for the Asus chassis is in the 2900$ area, whereas the basic Supermicro 6028TR-HTR with 3.5" and Gbe is in the same range. For 2x X540 10Gbe onboard look at 6028TP-HTTR for 3.5" disks or 2028TP-HTTR for 2.5", they should run for ~3800-3900$. Intel was around 4000$ with all compute nodes, Gbe only.
If you'd like mirrored ESXi boot and/or SAS connectivity, the 2028TP-HC0TR is the better option for you. It comes with LSI 3008 controllers for each of the nodes, connecting without expander to the 6 individual bays (which btw use the same trays as your storage system), which adds around 300-350$ to the total for the Quadnode barebone. Without expander you're also good to go with plain SATA disks.

A thing which all Supermicros have in common: support for http://www.supermicro.com/products/nfo/SATADOM.cfm modules. No SSDs or Hotswap bays needed, plugs onto the mobo and done. The storage system would even support two of them, so you can and should use two DOM's for the FreeNAS and mirror them (possible with FreeNAS 9.3 in software). The Quadnode 2028TP-HC0TR would also support these: http://www.supermicro.nl/products/nfo/mSATA.cfm so you could also run two flash modules per node there, however they're both connecting to the Intel SATA ports and therefore not provide redundancy for ESXi boot.

Oops, didn't notice the first price I found was in Euros. The Intel box is around $3500, but the SFP+ networking card is expensive. The ASUS has a well priced dual SFP+ network card with a custom form factor, leaving the one slot open for later use (like the HBA for my MD3000). I've spent four years with having to much local storage in ESXi boxes - I'm not wasting money on that again (now the VSA is gone). ESXi also doesn't need redundant boot, especially if you don't need true HA support. The ASUS supports an M.2 SSD, which will also work, and will be great for local swap/scratch space. The Intel boxes all support SATA DOM also. An interesting feature of the Intel box is that it would be possible to route the drive bays out the back using a bridge board and a pass through SAS2 connector, so you could kind of treat it as a JBOD expansion chassis, which would be an interesting hack. That might also be possible with the ASUS box, using a SATA breakout cable (although the M.2 SSD blocks access to the third HDD bay on each node).

Regards,
-Jeremy

 

[marbus90]

The 6028TP-HTTR and 2028TP-HTTR have per node 2x 10Gbe via RJ45 onboard. It might come out cheaper that way. Also it's downards comatible to 1Gbe ports on the switches. If you want to add a single 10Gbe SFP+ port, you could use the AOC-PTG-i1S which fits in the 0 slot, means it wouldn't use the PCIe x16 slot.

Thats an interesting feature of the Intel chassis, but personally I'd keep to the hardware I know - Supermicro and HP Procurve/Proliant ;)

Cheap entry-level Datacenter SSDs would be the Micron M500DC. If you don't shy away from using hdparm, you could pick consumer-grade SSDs like the Crucial MX200 or Samsung 850, underprovision them to 80% of their capacity and enjoy better performance consistency and eventually longer lifespan due to more free cells for wear-leveling.
I'd still keep to the Intel S3700 or S3710 series for SLOGs and for the SSD pool S3610 if you're fine with ~28k write IOPS instead of 39k on the S3710 SSDs. Doesn't matter if you pick 480GB or 800GB, price per GB is slightly lower on the 6x800GB model with fewer bays used, performance on 8x480GB will be better tho. Still, 2.4TB vs. <2TB - the choice is up to you.