BUILD ESX FreeNAS Box Hardware & Setup Questions

[joel3452]

So I am looking to build an ESX server that will host Freenas, Windows 7, Linux in a home environment. It will primary be used as a media storage server for 2-3 concurrent users.

Motherboard: Supermicro X11SSH-CTF ($418.90)
CPU: Intel Xeon E3-1230 V5 3.4GHz Quad-Core Processor ($271.66 @ Newegg)
CPU Cooler: Cooler Master Hyper 212 EVO 82.9 CFM Sleeve Bearing CPU Cooler ($24.99 @ Newegg)
Memory: Crucial 32GB Kit (2 x 16GB) DDR4-2400 ECC UDIMM VLP or 2* Crucial 16GB DDR4-2400 ECC UDIMM VLP depending on in stock
2* Storage: Sandisk Extreme Pro 240GB 2.5" Solid State Drive ($119.99 @ Newegg)
8* Storage: Western Digital Red 6TB 3.5" 5400RPM Internal Hard Drive ($234.52 @ Newegg)
Case: Fractal Design Define R5 (Black) ATX Mid Tower Case ($119.98 @ Newegg)
Power Supply: SeaSonic 660W 80+ Platinum Certified Fully-Modular ATX Power Supply ($104.99 @ Newegg)

I have read many of the forum posts about ESX & Freenas and the general guidelines seem to be:
- Pass through the LSI controller (in IT mode) so Freenas can have the hard drives natively
- Don't over-commit the Vcpu
- Reserve all of Freenas's memory (will probably lock it as 16 GB to start)
- Create a backup of the Freenas config in-case ESX doesn't boot

So here is my general idea for the build:
A1. Assemble the machine
A2. Test the memory using memtest86+ for 12-24 hours
A3. Flash the LSI controller to IT mode
A4. Load an OS on a temp drive and use something like Hard Disk Sentinel to do full read/write on the drives several times
A5. Create Raid 1 array of the 2 SSD using the Intel 236 controller
A6. Install ESX to raid 1 of SSD
A7. Create Freenas VM on the SSD datastore and passthrough the LSI controller with 2 Vcpu, 16 GB memory reserved, VMXNET3 nic, LSI passed through
A8. Install Freenas as normal creating a single Raidz2 using the 8 hard drives

So my general questions:
1. Can you help me find the best 32GB memory kit for this board? The part numbers on Supermicro's website for tested memory don't show much on amazon or newegg.
2. Is that the best CPU for the project? All of the CPU's above that one seem to have onboard video which won't get used or slightly higher clocks.
3. Any major differences between the X11SSH-CTF and X11SSL-CF besides the 10Gb NIC and two extra SATA?
4. Any recommendation for good SSD's for ESX install/Datastore
5. What is the best easiest ways to backup my ESX and Freenas configurations?
6. Do the Supermicro boards come with the SAS breakout cables?

Thanks and let me know if there is any feedback or other things I didn't think of or missed.
Edited to make some questions clearer hopefully and cleaned up parts links.
Edited again to add suggested memory.

 

[snaptec]

1. registered wont work at all I think
2. depends on the workload



Gesendet von iPhone mit Tapatalk

 

[Dice]

1. outrageous numbering system.
1. no
2. depends.
3. is that a question?
4. that seems like a statement rather than a question.
5. it is in the manual.
4. some effort on formulating questions.
3. some effort in making lists useful.

 

[Stux]

2. Its the best bang for the buck at the top of the performance chart. Its the cheapest way to get the most skylake cores for ESXi.

 

[Ericloewe]

registered wont work at all I think

It sure as hell won't. LGA115x systems only support UDIMMs, no exceptions.

 

[joel3452]

Is there a good recommended 32 GB ECC kit for that motherboard available in US? I normally use Amazon and Newegg for parts, but didn't see much on either besides Kingston ECC ram which I believe some people in the forum don't recommend.

 

[Ericloewe]

Is there a good recommended 32 GB ECC kit for that motherboard available in US? I normally use Amazon and Newegg for parts, but didn't see much on either besides Kingston ECC ram which I believe some people in the forum don't recommend.

Why do you want a "kit"? Buy the amount of DIMMs you need (2).
That said, Crucial.com works well, pricing is reasonable and you're very likely to end up with DIMMs from the QVL. Be sure to check the DIMMs for Micron part numbers if you buy Crucial.

 

[joel3452]

Looking at crucial's site for that motherboard it looks like they are mostly out of stock except the VLP versions: http://www.crucial.com/usa/en/compatible-upgrade-for//x11ssh-ctf
The VLP version should work the same except they are a bit shorter correct for low profile motherboard right?

 

[Ericloewe]

Looking at crucial's site for that motherboard it looks like they are mostly out of stock except the VLP versions: http://www.crucial.com/usa/en/compatible-upgrade-for//x11ssh-ctf
The VLP version should work the same except they are a bit shorter correct for low profile motherboard right?

Yeah, the PCB is just a bit shorter. It'll bother you for all of 60 seconds over the lifetime of the server.

 

[joel3452]

With the memory straightened out does everything else look okay in the hardware/config idea?

 

[Dice]

4. depending on the quality and speed of storage solution, any consumer SSD will basically do.

 

[CookiesLikeWhoa]

5: VMware Snapshots in FreeNAS will do nicely for taking snapshots of your VM's.
6: Negative. It will come with SAS cables but not breakout ones.

Another thing you can do as well, though up to you, is install ESXI on a thumb drive, pass the SSD's through to FreeNAS, make mirrored Vdev out of them and serve them up as an iSCSI target for ESXI. That way you'll have some protection on your ESXI data store while still getting the performance you want. Only downside is you'll need to make sure FreeNAS powers up before the rest of your VM's do.

 

[Mlovelace]

First of all to be pedantic, ESX and ESXi are not the same hypervisor. VMware stopped development of ESX with version 4.1, so unless you are going to run an extremely out dated hypervisor, you're planning an ESXi build.

Secondly, installing ESXi onto a mirror'd pair of SSDs is a waste. You would be better served installing ESXi on a USB thumb drive, and spending the money that would have bought two SSD instead on a single larger and faster SSD to use as a datastore and or as a Host cache device. On boot ESXi loads into memory and runs in memory, so the performance of the hypervisor is dependent upon the hardware specs of the Host, not the boot device. Simply keep a current backup of the ESXi config, and if the USB thumb drive dies, install on a new one, upload the config you'll be backup inside of 5 min. Here's a link on backing up your ESXi config.

 

[joel3452]

Mlovelace, yes I was referring to ESXI, specifically I will probably install 6 update 2 as that is the current version and the one I have been playing with on a spare machine at work.

Why would using a raid 1 of SSD be a waste though? When you install ESX on single hard drives it lets you use the rest of the storage as a datastore. When I did a test build on a spare machine installing to a single SSD, ESXI took up ~1 GB of space and then the rest of the SSD was a default datastore. Does this not work the same if I made a raid 1 using C236's chipset?

So I was figuring if I do a raid 1 of the SSD, not only is my ESX install have some protection but my datastore where the Freenas, Windows, and Linux VM live will have some protection as well as pretty good speed. Let me know I was misunderstanding you.

Cookieslikewhoa,
I don't know if I could pass through the SSD's from the C236 ports. I think it would also add complications to the startup as at least with my planned deployment each VM should be able to run without any of the others as dependencies. Not sure if the benefits of Freenas managing the raid 1 would be that significant over using the C236's raid 1.

 

[CookiesLikeWhoa]

You should be able to pass the SATA ports through.

Monitoring the SSD's. It would give you notifications about the health of the drives and let you know when one is on its way out. Also scrubs to ensure the data is good on them.

Realistically the FreeNAS VM shouldn't be going down that much. But it's up to you.

 

[Mlovelace]

Why would using a raid 1 of SSD be a waste though? When you install ESX on single hard drives it lets you use the rest of the storage as a datastore. When I did a test build on a spare machine installing to a single SSD, ESXI took up ~1 GB of space and then the rest of the SSD was a default datastore. Does this not work the same if I made a raid 1 using C236's chipset?

So I was figuring if I do a raid 1 of the SSD, not only is my ESX install have some protection but my datastore where the Freenas, Windows, and Linux VM live will have some protection as well as pretty good speed. Let me know I was misunderstanding you.

C236 is a Intel-chipset containing RSTe (rapid storage technology). The chipset is supported, even sata-ports, but not this "raid-controller". Basically, it is a fake- (aka software-bios, pseudo) raid, and these are generally, for all intents and purposes, not supported by ESXi. Even if you create a raid 1 with the C236, the ESXi installer will see individual disks. If your heart is set on running a raid 1 boot, then get good true hardware raid-controller for ESXi, with on-board cache and power-loss protection.

 

[joel3452]

Well that is a bummer to hear that the C236 is a software raid. So what you would suggest is install ESX on flash drive, then have two pools under Freenas. One running the SSDs and the other running the hard drives? Where does Freenas get installed in this setup? Do you pass through a second Flash drive to the Freenas VM?

I mean if its recommended to put the datastore in Freenas, I wonder if it is even worth it to use the SSD's for a datastore since this is only for home use. How does ESXI handle having a datastore in another VM? That sounds like it might make things much more complicated.

Or should I just boot ESXI off a flash drive and use a single SSD for my datastore and then do manual backups of the VMs occasionally? I honestly care about the data that will be in Freenas the most. Most of the VMs that I will be running I can rebuild in a weekend if needed.

 

[Mlovelace]

If the data on the freeNAS is important what is your plan to back up that data, let alone, the ESXi VMs? There isn't a level of array redundancy in the world that replaces having a backup.

What is the intended use of the Win 7 and linux VM? You may consider running freeNAS on bare metal and using the virtual box jail for the windows 7 and linux VMs... why over complicate things. If you're running freeNAS bare metal you can take those two SSDs make a separate mirror pool specifically for jails, E.g. virtual box jail.

 

[joel3452]

Well my uses for the other VMs: For the Windows VM, I would like to use it basically as another machine that I will be able to remote into and do work. Or to leave tasks running that take a while like queue movie encodes, etc. The Linux VM, not as sure about. I wouldn't mind moving my GSN3 VM from my workstation to the ESXI machine for when I do practice Cisco labs.

I imagine most of the things that I want to do in the Windows and Linux VM's I could probably do in jails or virtualbox. But sometimes when you google something simple like deluge Freebsd jail and see its a ton of stuff and some people with massive complications. Versus if I have a linux vm, its an apt-get install away to have deluge. There is also the practical part of me that wants to be able to use more of the 4 core + HT cpu that will probably be 95% idle for Freenas except when I do large operations.

As for the backups...My current setup is an old desktop converted into a server running Unraid without about 18TB of storage. Most of my data is write once read occasionally like all of movies I ripped, tv box sets I converted from dvd to x264, etc. There is really only 100-200 GB of stuff that is "valuable" like picture and such. And that data I have a copy on an external portable hard drive I encrypted with Veracrypt. So my plan after I setup the new server is to decommission the old machine and use the mix of old drives (mostly 2TB Greens) in it as full on cold storage. By that I mean encrypt each drive with Veracrypt and then lock them in desk drawer at work so I have some offsite backup. I do have good (for the US) Internet at my house of 300/20Mb so I might start backing up that 100-200 GB of stuff to there as well using something like Crashplan or similar.

 

[joel3452]

Could I just use a single SSD as my ESXI install + datastore? What are the dangers to the Freenas VM if that SSD would fail? Could I just boot up the machine off a flash drive with Freenas install on it and import the pool?

or could I use a IBM ServeRaid M1015 and create a Raid1 of 2 ssd's and use that for the ESXi install + datastore. And then store all of the VM's on that? I realize I wouldn't get the ZFS snapshots or things, but the VM performance should be quite fast and fairly reliable right?