My Dream System (I think)

[jgreco]

Let us know how it goes, of course.

 

[joeschmuck]

Here is likely my main final update on my Dream System.

So I started this thread basically 6 months ago with a dream to reduce the number of computers I have running various programs and using ESXi to do that. With the help of everyone in this thread I was able to pull this off and learn quite a bit more. Special thanks goes out to @jgreco for always being my sounding board and keeping my feet stuck in reality. Additionally he provided a RAID card which completed the build, and yet again I learned something new, this time about RAID cards. And a thanks to everyone else who provided assistance or even questioned what the heck I was doing in the first place because lets be real, in this forum we do not promote use of ESXi for FreeNAS and while I agree with that somewhat, well I think we have an excellent thread here for anyone else who wants to dive in providing they understand the proper discipline to run a system like this.

My final system configuration is:
ESXi 6.0U2
Intel E3-1230v5 (3.4GHz) Skylake CPU
Supermicro X11SSM-F
64 GB Samsung DDR4 ECC 2133 MHz RAM
Dell PERC H310 flashed in IR mode
Two IOCREST SI-PEX40062 4 port SATA PCI-E (in pass-thru for NAS Drives)
Two 256 GB SSD Boot Drives and Datastore1 (RAID1 using the PERC H310)
One 120GB SSD for Datastores
Six WD Red WD20EFRX NAS Hard Drives (RAIDZ2, 7.3TB usable space)
All wrapped up in a Cooler Master HAF 912 case
APC Back-UPS Pro BR1000G

VMs:
FreeNAS 9.10 Stable
Sophos
Ubuntu (always running BOINC in the background)

The changes I've made to the system recently are as follows:
Removed an Intel Dual NIC card and a second instance of Sophos which I was running a second WAN IP from but because I rarely used it, I disconnected it. I may reconnect it in the future but first I will see if I could simulate a second MAC address so I could use the same NIC port I'm using for the primary WAN connection. I'm suspect that it can be done but I just need to investigate it more.

Added the Dell PERC H310 and reinstalled my entire system from scratch (except for the VMs) so I could proof the instructions I used in this thread.

Added a second 256GB SSD (Crucial M4) and connected those to the H310 as the boot devices and first datastore.

Removed the 1TB laptop hard drive.

Things to do in the Future:
Well I need more storage for my datastores and I'm contemplating if I should reinstall the 1TB alone, buy a second 1TB laptop drive, or just pop in a pair of 2TB 3.5" drives in a RAID0 configuration. I have a pair of 2TB drives now but they are old as dirt and I had considered using FreeNAS as storage for extra VMs but now I'm doing something I'm just not comfortable doing just yet and the VMs (Windoze 7 and Windoze 10) likely won't be very snappy. A SSD would be the right choice but unless a 1TB SSD comes on the market for $100 anytime soon, I'm S.O.L.

One other thing which I need to do it get some automation into backing up my system to FreeNAS. With the RAID1 for the boot SSDs I at least have full redundancy and I have of course ESXi on it, FreeNAS, and Sophos, all the main VMs. I have enough room for Ubuntu but I placed it on the 120GB SSD for now. But since you never know when I will do something stupid to my system, having a backup is important.

Total cost of this system was astronomical and I don't even want to think about it, however when compared to a very good quality NAS, it's still cheaper. And I have my first ASUS M5A78L-M/USB3 w/16GB ECC RAM just sitting on a shelf, no longer my Test System, as an added expense in the beginning. I need to figure out how I can re-purpose this because I'm apt to let it sit and collect dust. Maybe I could sell it on Ebay and recoup a few bucks. Or maybe I will put it back into a case and make it a test system for when FreeNAS 10 comes out. Yea, I'll hang on to it for a few more months and test FreeNAS 10 with it.

Conclusion:
If someone really wants to create an ESXi system and run FreeNAS on it, I say "Go For It", just be aware of the risks and the discipline required. This was a great experience for me and I would do it again, but some things would be done differently of course. Right now I'm not using any of the motherboard SATA ports, what a waste and a better motherboard selection would have benefited me here. If I could get the SATA ports into pass-thru mode with ESXi then I could remove the SATA add-on cards I have, but with my current motherboard that is not possible. I'm not sure if that can be corrected with a newer BIOS (coming out soon) but I would like to see that (trust me, I'll try it). And if you are going to be going down this path, plan everything out that you can ahead of time. Read this thread, see the challenges I had and you will likely have a less challenging experience than I did, but I did learn from the errors of my ways.

 

[Mirfster]

Great thread and a ton of great info in here. Thanks for all your work and notes!

And I have my first ASUS M5A78L-M/USB3 w/16GB ECC RAM just sitting on a shelf, no longer my Test System

What about maybe thinking of using this as part of a JBOD build that you can connect to the ESXi and pass-through to FreeNAS? ;)

 

[jgreco]

And a thanks to everyone else who provided assistance or even questioned what the heck I was doing in the first place because lets be real, in this forum we do not promote use of ESXi for FreeNAS and while I agree with that somewhat, well I think we have an excellent thread here for anyone else who wants to dive in providing they understand the proper discipline to run a system like this.

And by the way, this, right here, is why we warn against virtualization. What @joeschmuck has done is a recipe for success. Not a guarantee of success. If you take the time to learn the things you need to know, carefully experiment and come to understand what you're doing, and master the tools that you're using, I'd say you have a really good chance of making things work. If users consistently did this, virtualization would never have been a problem here. However, just as with system builds, too often people are interested in getting to the finish line. If you take all your hardware bits and assemble them in a half hour on your kitchen table, then load FreeNAS on it and call it a day, it might "work" but you won't have done a good job, and you may have problems in the future. If you do that same thing with software, you get the same bad results.

Just as with carefully building and testing your hardware platform, and making sure everything's done right, the software side with virtualization must be carefully and thoughtfully implemented.

Some have said that I am a massive hypocrite because even as I wrote the anti-virtualization stickies, I was successfully running virtualized FreeNAS. Yes. Hell yes, even. I am not willing to give anyone a bullet list HOWTO document to do something that requires technical expertise (even mastery). I want people to be scared because there's a huge potential downside. Data loss. Destruction. Death, skulls and bones.

Anyone who is mature enough to own their self-made mess should of course be allowed to do as they please. If you go the full @joeschmuck route and you are having problems with some particular angle, there's at least several of us who will be happy to step in and help you find the path. It can be a great learning experience along the way too.

 

[jgreco]

By the way, I would like to point out that @joeschmuck 's process of building this and getting to a stable, happy point seems to have been approximately six months. That says something, at least to me. Take heed, those of you who want something quick and easy. It's neither quick nor easy if done right.

 

[Spearfoot]

Dell PERC H310 flashed in IR mode
...
Two 256 GB SSD Boot Drives and Datastore1 (RAID1 using the PERC H310)
One 120GB SSD for Datastores
...

I humbly admit I've never installed ESXi to anything but a single USB stick. Somehow I got the idea ESXi needed a separate boot drive that couldn't also be used as a datastore. Doooohhhh!

Your post has me thinking I ought to re-flash one of my recently-acquired Dell H200 HBAs to IR mode and configure a pair of SSDs as my ESXi boot drive and datastore for the FreeNAS VM.

Currently I boot ESXi from a USB stick and have a mirrored installation of the FreeNAS VM on a pair of small SSDs configured as separate datastores.

 

[jgreco]

I humbly admit I've never installed ESXi to anything but a single USB stick. Somehow I got the idea ESXi needed a separate boot drive that couldn't also be used as a datastore. Doooohhhh!

ESXi will happily install to USB flash, but will not use USB flash as a datastore. The USB will not be redundant.

ESXi will also happily install to supported RAID1. A cheap non-caching RAID controller like the IR-mode card will be insanely slow for writes to HDD, because ESXi requires sync writes.

Your post has me thinking I ought to re-flash one of my recently-acquired Dell H200 HBAs to IR mode and configure a pair of SSDs as my ESXi boot drive and datastore for the FreeNAS VM.

Currently I boot ESXi from a USB stick and have a mirrored installation of the FreeNAS VM on a pair of small SSDs configured as separate datastores.

Yeah. So the SSD's with an IR-mode card will not be blazingly fast, but will be pretty reasonable. I've been on the edge for awhile as to the value of the high end cache-BBU RAID controllers. They're frickin' fantastic but for the same price I could probably afford to go more heavily with SSD and get acceptable results on an IR-mode card. Be sure to read

https://forums.freenas.org/index.php?threads/esxi-home-server-freenas.43527/page-3#post-289705

Obviously, with a RAID controller, the system will boot regardless of whether one of the SSD's dies, and I believe you can also have a spare configured even on the IR cards, so you can put three SSD's in, two active, one standby, and have it magically and automatically replace a failed unit. And will continue to boot without any interruption.

For whatever it is worth, you might also consider trawling around fleabay for an LSI 9260 or one of the OEM's along with the BBU or supercap unit. Those are more expensive. The current hypervisor configuration here involves:

Three WD Red 1TB 2.5" (two mirror, one standby)
Five Intel 535 480GB or 850 Evo 500GB (two mirror, two in another mirror, one standby)

This is eight drive slots and provides 2TB of usable datastore, half as HDD, half as SSD, with automatic spare. The 9260 or 9360 turbocharge things for both HDD and SSD, but that's less noticeable with SSD because they're already fast. I have posted numbers in the past but can't find them now.

 

[joeschmuck]

The Dell H310 controller isn't very fast but it does what I need it to do. I have a lot to think about with respect to some high speed VMs, maybe I'll actually use my motherboard SATA connectors to hook up a hard drive, these seem to be pretty fast.

As for why my boot device is in a RAID1 configuration, it's because when I go on travel, I need to ensure that I have done everything I can to make the internet not die. I can't fix it while I'm out of town and I'll hear the grief from the family so mitigating this as much as I can is the main reason. And it doesn't hurt that if I have a failure of a drive that a simple replacement will make everything happy again. Using USB flash drives are fine if that is what you want to use.

And also, this is not a top of the line system which I built, it's one that is more than enough for my current needs.

And thanks for the kind words @jgreco. It's appreciated.

 

[maglin]

Congrats joeschmuck on a successful journey. I've enjoyed reading your thread.


Sent from my iPhone using Tapatalk

 

[joeschmuck]

I've enjoyed reading your thread.

Well one thing is for certain, we got off topic a few times and had a few laughs in the process. I mean, 17 pages? It's not all ESXi, that is for sure but it was an Off Topic thread in the first place.

 

[jgreco]

thanks for the kind words

No need to be insulting, you old geezer.

 

[joeschmuck]

Hey, we are in another thread, lets fight there. And who's calling who old? ;)

 

[jgreco]

Hey, we are in another thread, lets fight there. And who's calling who old? ;)

I'm calling you old, you old codger. Look, you have wrinkles and all-white hair.

 

[joeschmuck]

Busted.

 

[gpsguy]

My *guess* is that I might be 6 years old than you are.:(

And who's calling who old?

 

[Mlovelace]

ESXi will happily install to USB flash, but will not use USB flash as a datastore. The USB will not be redundant.

ESXi will also happily install to supported RAID1.

Dell through it's DRAC and HP through it's iLO4 will allow you to mirror USB/SD devices for exactly this purpose. It's also disappointing that Supermicro hasn't put forward ESXi custom images like other vendors. It's easy enough to build your own custom image, but there are a lot of supermicro servers running VMware, you'd think they put more effort into that customer base.

 

[jgreco]

Dell through it's DRAC and HP through it's iLO4 will allow you to mirror USB/SD devices for exactly this purpose.

That's vaguely nice but I have to wonder how good the support for failed device alerting, etc., is. I'm always the skeptic.

It's also disappointing that Supermicro hasn't put forward ESXi custom images like other vendors.

For what purpose? It's been somewhat unusual, at least here, where ESXi hasn't worked out of the box, and usually when that happens it is because I've got some unusual device in the box. One of Supermicro's strengths has been that they have a tendency to build the hardware that people want, rather than making hardware people have to adapt to. I'm thinking especially of cases such as the X9SCL+-F where they created a version of the X9SCL that specifically had dual 82574L's, to accommodate the folks running older ESXi 4 that didn't have support for the 82579.

It's easy enough to build your own custom image, but there are a lot of supermicro servers running VMware, you'd think they put more effort into that customer base.

I can't recall having needed a custom image. Ever. I've made a few, especially in the early days, but things work well by default in most cases.

 

[Mlovelace]

That's vaguely nice but I have to wonder how good the support for failed device alerting, etc., is. I'm always the skeptic.

The iLO emails that the device is in a degraded state. I've had to replace one. After you move the logs and scratch space the only writes that go to the USB are updates. There really isn't a down side to it, especially if the raid 1 boot is killing the ssd performance.

 

[joeschmuck]

My *guess* is that I might be 6 years old than you are.:(

So you are 106? Wow, that is old. I was 20 when Alan Turing and I were working on his big break through. Okay, maybe I'm exaggerating just a bit. All I can say is that I just missed the Free Love movement.

 

[joeschmuck]

I had to make a custom image to include a NIC driver that use to be in ESXi 5 but was dropped from ESXi 6. I'm no longer using that NIC so I dropped the custom image.