FreeNAS + ESXI Lab Build Log

[Jailer]

You don't need mini SAS to SATA cables with the X10SL7-F. The SAS controller has SATA connectors on the board already.

 

[Chris Moore]

2x MiniSAS to SATA, https://www.amazon.com/Cable-Matter...F8&qid=1519584902&sr=1-4&keywords=SAS+to+sata

For the other hardware you have selected, I don't see where you need these.

Any reason the higher model # CPU is cheaper than the 1231? Why is there such a wide price range on the various models?

I can only guess that is has something do do with supply and demand, but I have often been confused by the pricing myself. The only answer I could come up with comparing them on ark.intel.com is that the 1270 is already marked as 'Discontinued' after having been launched in 2013 where the 1231 is still marked as 'Launched' after being introduced in 2014. The specs are very similar though.

 

[Maelos]

Jailer, as it seems we have similar builds, how did you decide on your CPU and RAM? If I may ask, where did you end up purchasing them?

I looked at the picture again and whoops, you are right, no need for SAS to SATA adapters.
https://www.supermicro.com/products/motherboard/xeon/c220/x10sl7-f.cfm

Also, another link I will build into my resource/newbie guide - https://ark.intel.com/products/series/59137/Intel-Xeon-Processor-E3-Family

E3 1290 for $190 with 3.7GHz, um, yes please! https://www.ebay.com/sch/i.html?_odkw=Intel+Xeon+1150&_osacat=0&_from=R40&_trksid=p2045573.m570.l1313.TR2.TRC0.A0.H0.XIntel®+Xeon®+Processor+E3-1290.TRS1&_nkw=Intel®+Xeon®+Processor+E3-1290&_sacat=0 - DOH! LGA1155

https://ark.intel.com/products/80913/Intel-Xeon-Processor-E3-1286-v3-8M-Cache-3_70-GHz - FCLGA1150 is the socket type, but apparently it works

May very well be a hoax, the others are around $500-->>Here is an Intel® Xeon® Processor E3-1281 for $189 https://www.ebay.com/itm/Intel-Xeon...298475&hash=item4b3a42af7f:g:IboAAOSws5BagGus (THIS IS AN ENGINEER SAMPLE) Hmm...

Another, 1271 - https://www.ebay.com/itm/Intel-Xeon...552085&hash=item2850a0b8bb:g:mhoAAOSwWEZafgoY

Xeon E3 v3 family - https://ark.intel.com/products/series/78581/Intel-Xeon-Processor-E3-v3-Family
1281 https://ark.intel.com/products/80907/Intel-Xeon-Processor-E3-1281-v3-8M-Cache-3_70-GHz
1231 https://ark.intel.com/products/80910/Intel-Xeon-Processor-E3-1231-v3-8M-Cache-3_40-GHz

Almost identicle but the 81 has a higher clock speed, an "idle states" feature, anmd is 2 more W on TDP.

 

[Jailer]

When I built my server I was transcoding my DVD/Blu Ray collection in a Virtualbox VM. That's why I went with the Xeon. The RAM was bought 16GB at a time and honestly I probably could have gotten by with just the initial 16GB for my use case. RAM and CPU were purchased through newegg.

With your plan to run VM's I would consider an X11 series motherboard and newer CPU just for the option of being able to upgrade to 64GB of RAM if needed. You may not need it now but your needs may change once you get things built.

 

[Maelos]

The RAM for X11 is just absurd though. I don't plan on running many VMs, maybe just a few light ones permanently, and others for labbing purposes. The price is double for DDR4 for the same amount of RAM.

 

[Chris Moore]

The RAM for X11 is just absurd though. I don't plan on running many VMs, maybe just a few light ones permanently, and others for labbing purposes. The price is double for DDR4 for the same amount of RAM.

That is the reason I have been pointing people at used server gear on eBay. I would prefer to have new, but the price is just like you said.
You can get a second hand server with 128GB of RAM for less than the cost of a new server with 16GB or RAM.

 

[Maelos]

Just as you stated in the first page. I will keep browsing for deals but at least now I know what I want. Take a look at...https://www.ebay.com/ulk/itm/222851352573

The ECC RAM alone is worth 3x what they are asking

 

[Chris Moore]

Just as you stated in the first page. I will keep browsing for deals but at least now I know what I want. Take a look at...
https://rover.ebay.com/rover/0/0/0?mpre=https://www.ebay.com/ulk/itm/222851352573
The ECC RAM alone is worth 3x what they are asking

It is an auction though, so it could bid up significantly.
HDDs might be worth something to sell on eBay and the CPU & RAM are good to reuse, but I would recycle the rest.
I that model system board is only good for the 1U chassis because it only has two card slots, so forget any expansion.
Also, I hate the cramped, thin chassis and the high RPM (loud) fans that are usually in them. They are alright in the datacenter where I don't hear them, but I don't want them in my house.
I notice the seller has the CPU heatsink mounted sideways.

 

[Stux]

Beware the maximum ram.

X10SL7 has 32GB limitation, which imo is only barely enough for an ESXi/FreeNAS AIO that you actually want to do lots of things with.

VM hypervisors are almost always ram constrained.

So, more important than generation is class, Ie are you going with the consumer LGA115X CPUs etc, with their ram and PCIe limits, or the server grade LGA2011 style CPUs with plenty of lanes and slots and RDIMM support?

This used to be a simple E3 vs E5 naming scheme... then intel messed it up.

Or an embedded system?

Thus guide is worth looking at:
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

 

[Maelos]

Chris,

I'd buy that, boot it up to see how loud it was, record it for future reference. I'd then take the RAM (and other valuable parts) and recycle it or use it at work as a teaching board.

Stux,

read through that guide and used it as a reference for much of my research. It is an interesting idea, but I'm not sure when I will need more than 32 GB of RAM. I've got 24 GB in my esxi lab right now and never have come close, though it was not used much. From what I have read, additional non Windows VMs are not that resource intensive. This platform is a FreeNAS and webserver box first, learning lab second. I already have a lot of other things I can learn from, but having some room to play would be valued. There could be some value, say $400, in getting an X11 for the extra RAM capacity. I may not even start with that much, but having the flexibility would be nice. As for the multisocket and more flexible 2012 boards, I'm not sure I would ever utilize it. It's nice to imagime I would, but I don't use my lab enough as it is.

Do you have any other arguments for the 2011 option? Stu, your builds have been some of the best guides to my build and purpose, so thank you for that. I will keep scouring eBay for deals and I look forward to hearing more from you.

--Adding http://www.supermicro.com/products/motherboard/xeon/c600/x9da7.cfm as a link from research started by another post.

 

[Stux]

Yep, just as long as you are aware of the limitations, and go in with eyes open.

The non-2011 boards are limited in RAM, PCIe lanes, and number of cores. Generally the 2011 boards provide the full capability of that generation, ie 40 lanes, 8-12 dimm slots, per CPU socket. And high core counts. And even the option of dual socket (which I think is rarely needed for a FreeNAS install)

But you pay for it in dollars, size and power. Unless you buy discarded enterprise gear, which can be very affordable, and @Chris Moore seems to have a knack for finding ;)

 

[Maelos]

After having looked around I am again considering Chris's suggestion of https://www.ebay.com/itm/Supermicro...-2-6ghz-8-Core-128gb-24-Bay-JBOD/232656106862. It takes an X9 board with 2 CPU sockets, large cap of RAM, and is an all in one to boot. If I wanted to upgrade later my only limitations would be the form factor (mATX). I like that I would get the chasis (not needed initially, but bonus), plenty of cores even if at a lower speed than desired for VMs (not even really sure if I'd need it). I'm also a little worried about the noise from the psus. I will keep looking and thinking on this. I have already learned quite a bit and am enjoying the process.

X10, single socket E3v3 1270 4c/8t 3.5/3.9GHz ,max 32GB, w/ RAM and HBA, $1053, https://www.theserverstore.com/Supermicro-4U-Server-W-X10SLM-F_p_660.html

http://cpuboss.com/cpus/Intel-Xeon-E5-2650-vs-Intel-Xeon-E3-1270

https://community.spiceworks.com/topic/347799-more-cores-or-more-ghz

 

[Chris Moore]

The commentary from Spiceworks is indicative of lower clock and more cores, the way I understand it, and that dual 8 core system I pointed you at gives you 16 real cores with 32 threads (counting hyperthreading) which should easily give you enough horses to run 8 to 16 VMs at a reasonable speed. Please take my word for it, you don't want to try using systems clocked lower than that. We had someone accept a contract without looking close enough and we got a whole shipment of about a dozen systems with dual 8 core setups, but running at 1.7GHz. They are miserably slow. Having core count is great, but the speed still matters. Honest.
The other thing, with spinning disks, you need a bunch to get fast access to the data. We have systems where I work that are configure with 800GB disks in large quantities, to get the spindle count up, because that increases the IOPS.

 

[Maelos]

My worry is that the 2.6 GHz may be too slow, although the quadruple amount of threads would be nice. As for the disks, I am not as worried about performance yet. I see the same thing you mentioned for speed at work (data center operations). It's all about the IO speed. Perhaps at a later date I can snag a bunch of smaller disks to play with ISCSI and ESXI, but for now I just need to figure out what CPU requirements best fit what I plan to run.

How much of a difference is it between 2.6 and 3.5GHz? You have mentioned 1.7GHz in a couple threads, but is the same 0.9GHz jump as noticable? I know this will be application/software specific, which is something I will have to go back to the drawing board to better figure out my requirements. I do enjoy learning all of this though and all of your feedback is invaluable. One of the reasons I am here is because there does not seem to be nearly as much mentoring in the IT field as I would like, and doing it myself seems to be the best, and advised, way to learn.

 

[Chris Moore]

The reason I like a higher clock speed is for the times when you need to use it interactively. One of the servers I manage has dual six core at 3.4GHz. That gives 24 threads and there's only 8 VMs running normally. That gives me enough resources that the VMs all feel pretty responsive, almost like using a dedicated system. At 2.6GHz it feels like remote desktop over a 1gig NIC and is totally usable, when you get to the VM on a 1.7GHz system, then it's like remote desktop over dialup, sluggish and waiting for the screen to refresh.
That's all subjective and your personal perspective may be different.

Sent from my SAMSUNG-SGH-I537 using Tapatalk

 

[Maelos]

While that description helps a lot, what VMs are those? What is the usage/load they are getting? You mention dual six core - how far is that from single four core in performance, qualitative or not, and how far is it from dual octa with more or less clock? I am going to ask around and think of some more build possibilities. I don't mean to appear indecisive, even if that is what I am. I am learning a lot and trying to find the best value while also looking forward. While an X11 system with dual 2011 and near endless RAM sounds great, it may be more responsible to make an X9 or X10 series that costs only a 1/4 and then upgrade the parts as needed/available.

My plan would to have the following VMs run permanently

1. FreeNAS, possibly with Plex Jail unless advised to run elsewhere
2. Linux VM(s) with Apache (website), databases for websites, cPanel, game server(s), VOIP server(s)
   
3. Nagios XI (use at work, would like to learn more), possibly with Gearman and additional VMs for "distributed" workers
   
4. Confluence (could go on Linux VM)
   
5. Grafana? (could go on Linux VM?)

Semi Permanent (have yet to decide)

1. Sophos UTM (somewhat of a maybe)
2. AlienVault SIEM (somewhat of a maybe)
3. pfSense? I have a Cisco 5505 which I have not dabbled in extensively yet. I may want to be able to play with both

As needed for lab work

• Additional Linux distros such as Kali, CentOS, Debian, etc for testing/labbing/learning
• Windows Environment - been told 2x Windows PCs and 2x Window's Servers is a good mix for labbing
• JIRA
• VMWare vCenter

So it looks like 8+ with room for 4 more VMs as needed. That being said, FreeNAS would likely dominate the need for resources. This is also a preliminary list and could grow or shrink as I learn. I am getting a lot of ideas from https://www.reddit.com/r/homelab/wiki/software and work, please suggest if you think of anything.

If I have time later today I may put up a few builds covering the various ranges.

X11, multi socket (2011)(assuming more RAM capacity)
X11, single socket
X10, multi socket (assuming more RAM capacity)

X10, single socket
X9, multi socket (assuming more RAM capacity)

• $1100 with ship - https://www.ebay.com/itm/Supermicro...-2-6ghz-8-Core-128gb-24-Bay-JBOD/232656106862?rmvSB=true
• E5-2650 v2 is roughly $150 a peice
• 16x $23/8GB = $368
• X9DRI-F $250
• Subtotal is roughly $900 without chassis ($200ish), drive trays ($100ish), PSUs ($?), Backplane ($?)

X9, single socket

 

[Chris Moore]

, what VMs are those?

The first system I mentioned is actually older, it is running socket 1366 processors (model x5690) and has 196GB of RAM. With that, it is running Windows Server and Hyper-V with some Windows server 2016 and Windows 10 mixed (8 or 9 VMs) but in a lab environment where they are not being heavily used. The 2.6GHz system (using Xeon E5-2630 v2 CPUs) is running a few (how many changes) Windows 7 VMs, also in a low use lab. It has plenty of RAM and it really only feels a little sluggish but I don't mind it much. Those 1.7GHz systems are so bad I have been trying to get rid of them even though they are only about 3 years old and it is still way early (for us) to be replacing them. We normally run servers for 5 to 7 years at work, but I plan to run mine at home to around 10 years.

how far is that from single four core in performance, qualitative or not, and how far is it from dual octa with more or less clock?

The best resource I know for comparison of processors is this:
Single chip:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2630+v2+@+2.60GHz
Dual chip:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2630+v2+@+2.60GHz&cpuCount=2
They have results in there for single and dual chip systems, higher CPU Mark usually makes for better performance, and as you can see, doubling the number of physical processors does not exactly double performance.
Be sure to take into account the processor version when you are searching because there is a difference between a v1, v2, v3 etc. even within the same model number.
For example:
Xeon E3-1270 v1:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1270+@+3.40GHz
Xeon E3-1270 v2:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1270+V2+@+3.50GHz
Xeon E3-1270 v3:
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E3-1270+v3+@+3.50GHz
Sometimes, when the version changes, the socket changes too, so it can become very complicated.

PS. The system I pointed out on eBay has Xeon E5-2650 v2 processors in a dual configuration that would have a CPU Mark score of 19079 which is about double the CPU Mark score of the single Xeon E3-1270 v3, even though the E3-1270 v3 has a higher clock speed.

https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E5-2650+v2+@+2.60GHz&cpuCount=2

 

[Maelos]

https://www.cpubenchmark.net/cpu_value_available.html#multicpu - considering that the 2650 is #4 on that list...not bad. Hm...
The final factor would be noise - https://forums.servethehome.com/index.php?threads/supermicro-noise-levels.1004/. If I carry over some of the fans from my early build and do a bit of modding (may be possible?), I could dramatically cut this down. I plan to try to do something similar to my Cisco 2800 Router which is obnoxious.
Ohh, I found that the 846 chassis has space for internal non-hotswap drives, including 2.5 - https://www.supermicro.com/manuals/chassis/4U/SC846.pdf

 

[Chris Moore]

https://www.cpubenchmark.net/cpu_value_available.html#multicpu - considering that the 2650 is #4 on that list...not bad. Hm...

I assure you that I am not trying to send you the wrong direction. There may be better hardware available, but it will likely cost much more.

 

[Maelos]

I assure you that I am not trying to send you the wrong direction. There may be better hardware available, but it will likely cost much more.

Chris, I would never have joined the forum if I did not come seeking your guidance. I trust you, I am just nitpicking. The deal that you have linked is probably the best deal I can get, even if I were to plan out other setups, but I do the planning to learn too. Pleases don't take offense. I may go forward with purchasing one of these soon enough.

More questions
My downstairs closet is across from the AC/Heat. If I was to go with this board and its server grade fans/PSUs, is it possible that I can better soundproof the room to reduce the noise?
Also, I am just now reading about vibration being a problem for non-enterprise disks. Will this be the case for non NAS drives, or even non-pro NAS drives such as the shucked White/Red WD drives I plan to use.

https://www.ebay.com/itm/Supermicro...329114?hash=item25ee36591a:g:ImMAAOSwpvZaStPU - 920W Quieter, Plat power supply about $100 a piece as a future upgrade