My Dream System (I think)

[joeschmuck]

TBH, I am finding my experiment fun. While it may be a bit off the "proven" track I will eventually post (in a new thread so not to hijack)

Don't worry about hijacking this thread, if it's still ESXi related, I don't care, but you might enjoy having your own thread and I'll leave that up to you.

 

[joeschmuck]

Okay, easy enough. ESXi lacks any native support for mirroring or other software redundancy options. Your cheapest solution is an M1015 in *IR* (not IT) mode. Or any other well-supported RAID controller under ESXi. Then you get a cheap extra 120GB SSD, and mirror that with your original 120, and then boot off RAID1 SSD. You can of course do that with your 256GB instead if preferred. Or both!

You can of course RAID1 whatever you like, but HDD on the low end RAID controllers is really poorly performing because there's no write cache. SSD isn't so terrible. Was just discussing that last week with someone;

https://forums.freenas.org/index.php?threads/esxi-home-server-freenas.43527/page-3#post-289705

In general, I feel that you should probably stick with the slightly better quality SSD's like the Intel 535's and the Samsung 850 Evo's which do seem to work just fine on the backside of a RAID controller, but I'm a notorious cheapskate and I've been working with the PNY CS1311 960GB and SanDisk Ultra II 960GB SSD's these past few months. I didn't do extensive testing with them on the LSI's ... they're destined for some Synology DiskStation NAS units.

Anyways, the expense of getting a good-performing HDD RAID controller (which has write cache) is so great that it should be tempting just to avoid that and get SSD instead, unless you need a pile of space. For a small system, the cheap RAID plus SSD is *probably* going to be cheaper. Though you can probably prove me wrong if you eBay intelligently.

For cheap backups on ESXi, look at ghettoVCB. But you'll be getting seriously under the hood of ESXi if you do that. As in, be comfortable modifying the firmware of an appliance class device that's the virtualization equivalent of FreeNAS. Which I suggest you can and should do.

I'll give all that a look. I was thinking my motherboard would support mirroring the boot device but I haven't looked into it yet. I personally like the Samsung 850 EVO, I have one and it works great on my main computer. I will check out ghettoVCB.

Thanks!

 

[jgreco]

I'll give all that a look. I was thinking my motherboard would support mirroring the boot device but I haven't looked into it yet. I personally like the Samsung 850 EVO, I have one and it works great on my main computer. I will check out ghettoVCB.

Thanks!

Doesn't matter if the motherboard supports mirroring a boot device. ESXi won't.

I endorse the 850 Evo's as a fine choice that sits solidly between the too-cheap stuff and the ridiculous-expensive stuff.

A lot of the stuff we do here isn't super intensive on the I/O subsystem, so I find it very attractive to look at the consumer grade drives that lack the massive PBW ratings of the data center class drives. I also figure that if I blow through the drive's endurance and blow out an inexpensive $150 SSD (which is what we were pickin' Intel 535 480's for) in a year or two, the replacement will be a fraction of the price.

One suggestion, though, do consider getting two different types of drives if you get SSD to RAID1 together, particularly shooting for a different controller technology.

 

[joeschmuck]

Doesn't matter if the motherboard supports mirroring a boot device. ESXi won't.

So I'm a bit behind the power curve here and I need some explanation unless I got this right...

The issue is that my X11SSM-F board Intel C236 PCH isn't a real hardware RAID controller, is that true? It's really a software controller and that is why it won't work? I hope that isn't the case because out of all those onboard SATA ports, I'll end up using maybe one of them for datastores if I'm lucky. What a waste.

 

[Ericloewe]

The issue is that my X11SSM-F board Intel C236 PCH isn't a real hardware RAID controller, is that true? It's really a software controller and that is why it won't work? I hope that isn't the case because out of all those onboard SATA ports, I'll end up using maybe one of them for datastores if I'm lucky. What a waste.

It's purely software RAID. The bad part is that it's backed by Intel's SATA driver, not by any sort of PCH firmware running on the CPU. The PCH just shows the OS a bunch of drives and tells it what they're supposed to be.

 

[joeschmuck]

So now I need to evaluate if I want to spend about $120 for a used M1015 and drive connectors. Oh well, it's only money.

 

[Mirfster]

For what its worth, this is what I did:

• Bought one of these: "Syba Dual mSATA SSD to SATA III RAID 2.5" Enclosure Components SY-ADA40090"
• Bought two of these: "Samsung mSATA 128 GB SSD Laptop Solid State Drive MZ-MPD128E"
• Configured it as Raid1
• Installed ESXi 6.0 on it
• Created two 32 GB Virtual Hard Drives on it
• Installed FreeNas to the two Virtual Drives (in Mirror)
  • This is more of a "Feel Good" thing since it if the actual Syba controller goes, so does the ESXi and FreeNas... ;)

I didn't want to use up a 3.5" Bay for it and my C2100 fits two 2.5" on the inside natively on the fan shroud over the RAM and CPUs. I drilled 4 more holes so it will accommodate a 3rd 2.5" which is the Syba. Add a Molex to SATA power connector and I was in business...

 

[joeschmuck]

For what its worth, this is what I did:

• Bought one of these: "Syba Dual mSATA SSD to SATA III RAID 2.5" Enclosure Components SY-ADA40090"
• Bought two of these: "Samsung mSATA 128 GB SSD Laptop Solid State Drive MZ-MPD128E"
• Configured it as Raid1
• Installed ESXi 6.0 on it
• Created two 32 GB Virtual Hard Drives on it
• Installed FreeNas to the two Virtual Drives (in Mirror)
  • This is more of a "Feel Good" thing since it if the actual Syba controller goes, so does the ESXi and FreeNas... ;)

I didn't want to use up a 3.5" Bay for it and my C2100 fits two 2.5" on the inside natively on the fan shroud over the RAM and CPUs. I drilled 4 more holes so it will accommodate a 3rd 2.5" which is the Syba. Add a Molex to SATA power connector and I was in business...

That is a good way to go, wish I had thought of that in the beginning. I'm still inclined to go with the M1015 just because I can use my current older SSDs and not purchase new hardware. I have a pair of 256GB SSDs and pair of 120GB SSDs that can be used but before I can do that, I need to either upgrade my main computer to a new 1TB SSD or thin out what I have stored on my second 256GB SSD, or install a traditional hard drive. LOL, who am I kidding, I'm not replacing my main computers SSD with a traditional hard dive, that would be insane!

 

[Mirfster]

Thank, I have been running my main ESXi Server in a similar fashion, but using a "StarTech.com Dual mSATA SSD to 2.5-Inch SATA RAID Adapter Converter" and a couple of 32 GB MSatas for years so I figured why not.

 

[jgreco]

So now I need to evaluate if I want to spend about $120 for a used M1015 and drive connectors. Oh well, it's only money.

Yeah, looks like maybe you hosed yourself with the "two IOCREST SI-PEX40062 4 port SATA PCI-E (in pass-thru for NAS Drives)" ... if you'd gotten an M1015 to begin with, you could probably turn it around: flash the M1015 back to IR mode, and then pass thru the mainboard SATA controller to FreeNAS.

 

[Dice]

• Bought one of these: "Syba Dual mSATA SSD to SATA III RAID 2.5" Enclosure Components SY-ADA40090"

• Bought two of these: "Samsung mSATA 128 GB SSD Laptop Solid State Drive MZ-MPD128E"

This was a really neat solution I must say!

My config is abit different, I've chosen to rely on backups for the VM's and 'good memory' in terms of ESXi configuration (not learnt how to back it up yet, the gui does not leave any simple clues).
I use my motherboards SATA controller for datastore/ESXi and have forwarded HBA to all storage/FreeNAS.
In other words, I use a single 250gb EVO 850 for it all.

After reading about your solutions and the level of backup committments that @Spearfoot has put into his system, I feel a sudden need for revising my setup :p

Cheers /

 

[Rand]

...pass thru the mainboard SATA controller to FreeNAS.

I thought that was not working any more?

 

[jgreco]

This was a really neat solution I must say!

There doesn't appear to be any way to detect a failing module through ESXi if you do that, so it's not really that neat a solution.

My config is abit different, I've chosen to rely on backups for the VM's and 'good memory' in terms of ESXi configuration (not learnt how to back it up yet, the gui does not leave any simple clues).

In most cases, ESXi doesn't have a particularly complex configuration. Some people think that it's useful to back up the configuration of a hypervisor with vicfg-cfgbackup, but there are plenty of gotchas, and that's probably only going to be useful for restoring on the same hardware and version of ESXi. Everyone I know seems to think it a waste of time.

For free ESXi, the most powerful thing is probably to manage your configuration via the CLI and maintain a record of what you've done. For example, when we reload a hypervisor here, we have a bunch of shell script that does all the right stuff to bring a hypervisor up and online. If you have vCenter, then templates could be a better idea.

 

[Mirfster]

There doesn't appear to be any way to detect a failing module through ESXi if you do that, so it's not really that neat a solution.

Agreed, just didn't want to give up a PCIe slot for a Raid Controller and wanted a quasi feeling of some kind of redundancy. Hence my mentioning of:

This is more of a "Feel Good" thing since it if the actual Syba controller goes, so does the ESXi and FreeNas... ;)

I would not rely on them for a system that was remote and needed some type of notification on; nor for anything more than just to house ESXi and FreeNas (k, maybe some ISOs too at most). But it works fine for my use-case right now.

 

[joeschmuck]

Yeah, looks like maybe you hosed yourself with the "two IOCREST SI-PEX40062 4 port SATA PCI-E (in pass-thru for NAS Drives)" ... if you'd gotten an M1015 to begin with, you could probably turn it around: flash the M1015 back to IR mode, and then pass thru the mainboard SATA controller to FreeNAS.

You can't pass-thru the MB SATA ports on the X11SSM-F board, I tried, otherwise I'd have used only the one IOCrest board I previously had as the boot drive and the MB SATA ports for FreeNAS. Now if I buy the M1015 and it works to boot the system, well you know I'm going to try to pass-thru the MB SATA ports again. ESXi has been updated since my original tests and if it doesn't work, well at least I tried. If it does work then I can remove the two IOCrest boards and still be happy. I know I'll find other uses for them.

 

[Mirfster]

You can't pass-thru the MB SATA ports on the X11SSM-F board

Interesting, I just looked at my C2100 and it appears as if I could (if desired). Not that I have actually tried and verified this though. Wonder if this is isolated to that board?

 

[Ericloewe]

Interesting, I just looked at my C2100 and it appears as if I could (if desired). Not that I have actually tried and verified this though. Wonder if this is isolated to that board?

View attachment 12083

Well, ICH10 has essentially nothing in common with any PCH. Current PCHs integrate many essential system components. ICH10 is just a glorified I/O hub that Intel repurposed for the X58 platform, despite it being something like two or three years old already.

And, despite being nominally available for pass-through, people pretty much uniformly say it doesn't work well.

 

[Spearfoot]

Interesting, I just looked at my C2100 and it appears as if I could (if desired). Not that I have actually tried and verified this though. Wonder if this is isolated to that board?
View attachment 12083

Very interesting...VMware doesn't give me the option of passing through the Intel SATA controller on my newer X10SL7 board but it does on my older X8SIE system:

 

[joeschmuck]

Interesting, I just looked at my C2100 and it appears as if I could (if desired). Not that I have actually tried and verified this though. Wonder if this is isolated to that board?

Mine does not display the MB SATA ports at this time. Since my server is in the basement, I won't be messing around with it until I bring it up to my office.

 

[Mirfster]

ICH10 is just a glorified I/O hub that Intel repurposed for the X58 platform, despite it being something like two or three years old already.

Thanks for the clarification. Guess there is some small perks to running old stuff. Not that I am passing through the SATA ports though. ;)