My Dream System (I think)

[Rand]

True. But that would mean - address it to Supermicro and ignore it until they come up with a solution :)

 

[jgreco]

Fair enough.

 

[joeschmuck]

Making the IPMI a dedicated port did not solve the Shutdown issue. @jgreco as you said, if it don't power off, I can live with that as well and I'm certain to submit a ticket with Supermicro since they said it's compatible on their website. I'm sure it will get fixed in time.

I've also just tried to install ESXi 5.5 and it will not install, even with it being a bare board. It could be the new hardware isn't compatible. So that was a bust but at least I tried it.

EDIT: Installed Ubuntu and was able to shut the computer down without issue. ESXi has some issue with ACPI. Maybe there is some setting I need to change, hard to believe they have it all messed up.

 

[jgreco]

If anyone ever wonders why I'm perfectly happy buying "older" gear, this is an ideal example of why. Newer is often not better for useful values of "better"...

 

[Rand]

Not sure if you have seen those ... similar problem with disks not showing up at various places: (ignore if you know them)
https://forums.freenas.org/index.ph...ide-vmware-esxi-6-in-a-dell-13g-server.35320/
https://forums.freenas.org/index.ph...ssthrough-with-x8dt6-f-wont-show-disks.25198/

 

[joeschmuck]

@Rand I have not seen those before but I think I've already figured all that out on my own. I actually learn better by failures like this.

If anyone ever wonders why I'm perfectly happy buying "older" gear, this is an ideal example of why. Newer is often not better for useful values of "better"...

Yup, I knew there would be issues, just didn't expect ESXi to actually be one of those, but hey, the board hasn't been out very long.

 

[jgreco]

(mutters something probably inappropriate to the forum)

 

[joeschmuck]

So an update...
FreeNAS runs very well, I can access my hard drives via smartctl so that is a big plus that the HBA is working properly.
Sophos has been setup and I plan to test shortly.
ESXi still will not power down the system when you select shutdown however the VMs do all power down completely. And I still need to work on having an UPS trigger ESXi to shutdown too, but no rush.

 

[Rand]

Which HBA is in use now? I assume its not that 4-port SATA adapter?

 

[joeschmuck]

Yup, the 4 port adapter works perfectly fine as a pass-thru but ESXi does not support it as a card to place a datastore on it as you know they only want high end cards of course. The 4 port card belongs in my main computer so I can have SATA III speed for all my SSDs but the SATA II ports on the motherboard are doing just fine. I'm not certain which SATA adapter is best for ESXi use and then FreeBSD use as well. Definitely something I will need to investigate. If it turns out that the adapter I currently am using is compatible with FreeBSD 10 or 11, then I might just buy a second card, cheap for $38. But I need to check into the compatibility, certainly ESXi has no issues with it as a pass-thru card.

 

[joeschmuck]

A little research indicates that the achi(4) driver for the SATA port add-on card I am using for testing was introduced in FreeBSD 8 and is still used in FreeBSD 11 (current). I doubt this driver will go away soon since FreeBSD is an OS and these type of drivers tend to stick around for along time since people need them. Either way I plan to purchase one more of these add-0n cards so I can connect all six of my FreeNAS hard drives into this machine. Things are coming together nicely actually and maybe 2 more weeks of testing will be the key.

I need to figure out what I want to do for a high speed datastore, I could use my 120GB or 256GB SSD for any high speed VMs and then retain use of the 2.5" 1TB laptop hard drive for the ESXi boot and all other VMs that are not high speed needy, like FreeNAS and two copies of Sophos. Eh, I need to iron that out still. The big thing is the SATA add-on card will be just fine at least through FreeBSD 11 and if it stops being support at some point in time, I can just buy something new but for now it's only costing me $38 delivered since I already had a card on hand and I can live with my main computer running a little slower (I can't even tell).

 

[joeschmuck]

ESXi now shuts down like it should. The problem was ESXi 6.0U1a. And update to 6.0U1b fixed the problem.

 

[jgreco]

Oh geez there's a U1b now.

 

[Rand]

So how are things coming? :)

 

[joeschmuck]

Pretty well actually. I have ESXi running well, FreeNAS and Sophos both running on top of it. My current system consists of the X11SSM-F, 32 GB RAM, Intel Dual Port NIC card, two IOCREST four port SATA add-on cards, 256GB SSD (boot device), 120GB SSD (VM storage), 1TB 2.5" HDD (VM Storage), and my six WD Red drives forming my FreeNAS pool.

I have already placed my order for an additional 32GB RAM and it should arrive Friday.

So when I started this endevour I had no idea how difficult it would be to get things to work properly and I still haven't configured everything yet. Part of the issues had to do with the X11SSM-F and Skylake CPU being so new that ESXi didn't work properly until only a recent VMware upgrade. Other issues were my learning how to make ESXi do what I want. I started my installation using two old hard drives and created a FreeNAS system using a mirrored pair and placed some files on it. Then I tested booting and shutting down and rebooting the FreeNAS VM to ensure it actually shuts down properly before ESXi turns it off.

I have put a fair amount of time into this rig and testing it to ensure it operates and my data is as safe as it can be.

As for the hardware configuration... The 256GB SSD is my OS drive and hold ESXi, the FreeNAS VM (8GB), and Sophos VM (80GB). Additionally I have a 120GB SSD and 1TB HDD for additional VM storage. All these drives are connected to the MB SATA connectors. The six WD hard drives are connected to the two IOCrest four port add-on cards which are in pass-thru in ESXi. This means I don't need specific drivers for these add-on cards to make FreeNAS work. I have a single Intel Dual Port NIC installed as well which I'm using for the Sophos interface, well for the WAN connection.

Throughput for my FreeNAS is similar to my standalone system while using a common Ethernet port, the throughput moves up and down from about 80Mb/sec to 105Mb/sec and I'm certain this have to do with the E1000 NIC emulation going on. To be honest, I'm actually fine with the performance.

Sophos appears to be working fine as well.

So once I get the rest of my RAM and run MemTest86 for a few days (yet again), then I will finalize the build, button up the case, cross my fingers that it all stays working.

Well that is my update.

 

[BigDave]

Mark, you are a true Bridge Builder,
congrats on your success and thanks for
taking the time to document your experiences.

 

[Rand]

Very nice, congrats:)

-Whats the reason you go for 64 Gigs Ram?
-Any thoughts invested in a backup strategy yet? (For VMs as I assume you have a plan for data:))
This is a bit tricky with passthrough since the easy "snapshot / in place copying" will not work for VMs with pass through... On the other hand its only the boot drive & config that needs to be saved and thats probably easily recreated...

I am still waiting for the mainboard so only theoretical musings on my end;)

 

[jgreco]

This is a bit tricky with passthrough since the easy "snapshot / in place copying" will not work for VMs with pass through... On the other hand its only the boot drive & config that needs to be saved and thats probably easily recreated...

Or, better yet, easily backed up. There's no particular need to get OCD about backing up FreeNAS as long as you have the install media you used and the config file (this is also part of why I don't care for the online update thing)

 

[Rand]

O/c, config to be backuped and used for reinstallation :)

 

[jgreco]

So when I started this endevour I had no idea how difficult it would be to get things to work properly and I still haven't configured everything yet. Part of the issues had to do with the X11SSM-F and Skylake CPU being so new that ESXi didn't work properly until only a recent VMware upgrade. Other issues were my learning how to make ESXi do what I want.

Gives you a new appreciation for all the layers of complexity going on, doesn't it. Also some better insight into why it is so easy for people to fail, especially if they're not as thorough as you appear to be.