My Dream System (I think)

[joeschmuck]

You know, I'm looking for another project to mess around with and I'm thinking of building a server with which I can run ESXi on and then FreeNAS and Sophos and some goofing around VMs on top of that, or maybe when FreeNAS 10 comes out, if Bhyve works really well, Sophos in that as well as other VMs, I don't need ESXi unless it's just that much better. With that in mind I'm liking the X11SSM-F-O, 64GB RAM, Xeon E3-1230 v5 Skylake 3.4GHz and that will only cost just under $1300. OUCH! But this should give me the freedom to move my FreeNAS and it's 6 hard drives to this platform, and my Sophos system. I have a dual Ethernet port card I would need to add as well and it could be a sweet learning platform. The cost of the RAM is the real kicker to be honest so I guess half of that to start with.

I think I will first take my spare computer and load ESXi on it again and see if I can make it work in general for what I want, just to see if I can control it and understand what I'm getting into before I toss a lot of money at it.

Even though I'll talk myself out of purchasing that system, Here is to Dreaming!

 

[jgreco]

ESXi really is the bee's knees of virtualization platforms, because VMware has put a hell of a lot of effort into supporting so many different platforms.

On the other hand, they just let go the Workstation and Fusion teams, http://arstechnica.com/information-...rkstation-team-culled-in-company-restructure/

If you read the user comments for that article, you'll likely get an impression that the VMware is well-liked because it is very well developed, but that VMware is a bit of a dinosaur in this new era of freeware virtualization solutions - all of which are getting better at a rapid clip. Many of the things that make ESXi shine are actually part of vSphere, and not available for free. This will probably eventually doom VMware as people stop paying for the luxury and instead opt for cheap functionality, or ditch and move entirely to the "Cloud" (which is basically Xen hypervisor on someone else's machines). But in the meantime, ESXi is the Cadillac.

Don't forget that if and when you opt to go that route, some of us do have experience running FreeNAS on hypervisors. It is extremely nice to have a single machine running multiple tasks and sharing the hardware. It's expected to work reasonably well as long as you don't stupid it into fail.

 

[gpsguy]

Yeah, I read that yesterday.

I *think* ESXi is a safe bet for the near future. Companies with deep pockets will continue to buy VMware products.

On the other hand, they just let go the Workstation and Fusion teams

Since you already use VMware Workstation, you should feel right at home, once you launch the vSphere client and connect to your ESXi server.

I think I will first take my spare computer and load ESXi on it again and see if I can make it work in general for what I want, just to see if I can control it and understand what I'm getting into before I toss a lot of money at it.

 

[joeschmuck]

Yes, I do use VMware Workstation and I like it quite a bit overall but it does have some limitations. I think my experience using ESXi would work much better using dedicated hardware. I forgot about vSphere client costing extra money. I don't think it comes with VMware Workstation but I should check into that. Free is good, well I guess I already paid for it :)

 

[gpsguy]

The (fat) vSphere client which you use to manage the free (or paid) ESXi server is free.

The installation of ESXi is somewhat akin to FreeNAS. Once it's installed on the server, there is very little configuration that's done on the server itself. Setup an IP address, etc. and get it on the network.

Once that's done, open the URL from your Windows machine and download the vSphere client. After installing it on your Windows machine, connect to the server via IP address. The client UI, is very similar to what you are used to, with VMware Workstation.

The web version of the client, as well as all the bells and whistles on license on the platform cost $$$. Now and then, I'm tempted to buy vSphere Essentials ($667), which includes 3 years of maintenance (upgrades). This version, feature wise, doesn't offer much more than the free version. The big benefit is support for the backup API. The money doesn't stop me from doing it, it's just the (lack of) spare time.

Another option is to get VMware Advantage for $200/year. One of the benefits is EVALexperience which gives you a "365 day evaluation license for personal use".

 

[joeschmuck]

I pulled the trigger, just like the Internet Radio, I went and after doing some research and figuring that I need a new toy to occupy some more time, I purchased the following hardware:

X11SSM-F-O from NewEgg for $211.98 delivered.
E3-1230 v5 Skylake 3.4GHz CPU from SuperBiiz for $247.99.
32GB RAM in two Samsung M391A2K43BB1-CPB 16GB sticks from SuperBiiz for $114.99 each/ $229.98 total.
SuperBiiz also factored in an addtional almost $18 for S/H and a signature fee. And I would have purchased everything from NewEgg however the CPU and especially the RAM were significantly higher for the same exact stuff.
My total is: $708.42

I have spare parts to work with for now and I'm certain all of this will end up in my Cooler Master HAF 912 case. Once all this is stable, what the heck will I do with my current FreeNAS build? Of course it won't have the hard drives installed but it's been one very solid system. I could give it to my youngest son and put in a few mirrored drives for him but he might screw it all up, especially if I wanted to use it as an offsite backup (he lives about 12 hours south of me). Eh, maybe I could sell if for a few hundred bucks, or maybe piece it out. I really need to get rid of some of these old computer parts laying around. Maybe E-Bay LOL.

 

[adamjs83]

The web version of the client, as well as all the bells and whistles on license on the platform cost $$$. Now and then, I'm tempted to buy vSphere Essentials ($667), which includes 3 years of maintenance (upgrades). This version, feature wise, doesn't offer much more than the free version. The big benefit is support for the backup API. The money doesn't stop me from doing it, it's just the (lack of) spare time.

Before you pull the trigger on that make sure you check out VMUG Advantage. It's a roughly $200/year subscription to the full catalogue of VMware products for home and lab use.

 

[joeschmuck]

I am going to look at the free options first because free is good. When I build this system, I will just take a few extra hard drives and create a FreeNAS system for playing and also work on my Sophos, a second IP address for my home since my ISP does provide two of those and this way I can get some practical time while retaining my current systems operational.

 

[Spearfoot]

@joeschmuck, I built an 'all-in-one' system such as you describe (ESXi v6.0 w/ FreeNAS 9.3 running as a VM) last Spring and it works great! (See my signature for more details). I wouldn't recommend a system like this in an enterprise environment, but it works great as a developer's lab.

One of the key things about building such a beast is that you must use VT-d to pass the disk controller through to the FreeNAS VM. I don't know details about the motherboard you've chosen, other than to note that it has 8 x SATA 3.0 ports at 6Gbps -- nice! However, if you can't pass these through to the FreeNAS VM individually, then you may end up needing an HBA that you can dedicate to FreeNAS. An IBM M1015 can be had on eBay for ~$100.

I used a SuperMicro X10SL7 for my all-in-one. It has a built-in LSI 2308 HBA, which I dedicated to the FreeNAS VM. I used the mobo's SATA ports for a pair of SSD's set up as ESXi datastores, where I installed FreeNAS using the 'mirrored' option. ESXi 6.0 boots from a USB stick.

A young feller named Ben Bryan has a pretty good article about setting this up:

https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/

 

[jgreco]

No worries, @joeschmuck is not likely to go too far off the recommended path, and he's got a bunch of virtualization geeks here in the forum who'll straighten out any problems.

 

[joeschmuck]

No worries, @joeschmuck is not likely to go too far off the recommended path, and he's got a bunch of virtualization geeks here in the forum who'll straighten out any problems. :)

No doubt you guys will keep me in line and I always appreciate the advice, but of course I'll try to do all that I can on my own first. It's the only way to learn in my opinion. Eventually I'll hit a wall and I'll need to ask for help. I was going through the motherboard manual, holy cow there are a lot of options that will require some internet searching to figure out exactly what they are, even if I know I won't use them. Well hopefully next weekend I'll have all my parts in hand so I can play with it and at least check out the BIOS, get ESXi loaded and then start the RAM testing.

One of the key things about building such a beast is that you must use VT-d to pass the disk controller through to the FreeNAS VM.

I've read that before and the motherboard I selected does support VT-d, I just hope it actually works. The 8 SATA ports was highly desired since this was the plan in the first place and although I currently have 6 drives in my pool, I will eventually replace them with either 4 or 5 higher capacity drives when it's time to replace the pool. The other two SATA ports are for the ESXi boot device and then a separate data store for various VMs (I have a 120GB SSD and a 256GB SSD itching for a place to reside).

 

[Ericloewe]

Catch: It's been historically painful to try to pass the PCH SATA ports through. Passing only some of them through is outright impossible, I believe, so you'd need an LSI SAS controller and pass that through to FreeNAS.

We should also compare notes on the X11SSM-F. I think I'll finally have time to assemble the new server this week...

Also, that sounds like a bargain on the X11SSM-F. Most retailers in Europe are aiming at the 300 buck mark. Are all X11 boards around that price range in the US (strong USD comes to mind)?

 

[joeschmuck]

Catch: It's been historically painful to try to pass the PCH SATA ports through. Passing only some of them through is outright impossible, I believe, so you'd need an LSI SAS controller and pass that through to FreeNAS.

We will find out. If I need to spend a little more money, yes it will suck because I'd rather buy more RAM but I'll deal with it.

 

[jgreco]

Also, that sounds like a bargain on the X11SSM-F. Most retailers in Europe are aiming at the 300 buck mark. Are all X11 boards around that price range in the US (strong USD comes to mind)?

I'm seeing less than $180 for the X11SSM-B (bulk pack, no IPMI) and less than $190 for the X11SSM-F from a distributor.

 

[jgreco]

No doubt you guys will keep me in line and I always appreciate the advice, but of course I'll try to do all that I can on my own first. It's the only way to learn in my opinion.

Yeah, well, I'll beat you senseless if you don't read the virtualization stickies first, eh?

 

[joeschmuck]

Yeah, well, I'll beat you senseless if you don't read the virtualization stickies first, eh?

Stickies? Why would I do things the easy way?

 

[anodos]

Good to see that you're joining the dark side (Intel).

 

[joeschmuck]

Well it certainly isn't the cheap side, that's for sure.

 

[jgreco]

Catch: It's been historically painful to try to pass the PCH SATA ports through. Passing only some of them through is outright impossible, I believe,

Yeah, the X9 stuff wasn't a problem, the PCH ports on the X9 Cougar Point and Patsburg just magically worked for passthrough, and I was very disappointed that the X10 Wellsburg stuff didn't.

Passing through only some of them would be dependent on how they present. For example, the Cougar Point shows up as an "Intel Corporation Cougar Point 6 port SATA AHCI Controller", and my guess for X11 is that as long as they're all integrated on the PCH they'd also show up as a single controller. The Sunrise Point (C236) is supporting 8 ports, and it looks likely it's one big controller.

Anyways, to make a long tirade short, the point I kinda wanted to make was that if it's possible, it might be better for you to use the PCH ports for NAS drives, and pick up a RAID controller for ESXi. The ports on the LSI are slower than the native Intel ports.

 

[jgreco]

Stickies? Why would I do things the easy way?