BUILD ESXi Home Server + FreeNAS

[Gunndy]

All the upgraded parts should be in next week, but I'm still not quite sure how I am going to do the storage.

I'm thinking that I'm going to load/boot ESXi from a USB stick

Use the 9211-4i controller to create 2 x 120GB RAID1 SSD Datastore
Use the 9211-8i controller to passthru to FreeNAS VM and create vpool of 4 x 2TB Mirrored vdevs (Gives me the option to upgrade 2 drives at a time to increase storage)

I cant decide if I really want to RAID1 the SSD drives for a single datastore or use them solo for 2 datastores and back the .vmdk files up to the NAS storage. Is there a better solution that I'm not considering?

 

[joeschmuck]

To be honest, I'm not even sure which parts you are purchasing now, it's a long thread.

I cant decide if I really want to RAID1 the SSD drives for a single datastore or use them solo for 2 datastores and back the .vmdk files up to the NAS storage. Is there a better solution that I'm not considering?

What is the purpose of the SSDs? Are these expected to be the boot drive for ESXi and the data store for the FreeNAS VM?

If it were me, and I can't speak to your own desires, but I'd not use the 9211-4i controller and I'd plug in both SSDs into the motherboard, install ESXi on it. This would result in two SSD's which can hold future VMs, just not mirrored. I'd put my FreeNAS VM on the first device but honestly it just doesn't matter. And if you do have a failure of the SSD to either boot ESXi or FreeNAS, because you kept your configuration backup files, it's a fairly painless recovery, it's more annoying than anything else. And at this point in time I would state that since you are building a new system, play with it, break it and repair it, take detailed notes on how to fix it when it does break. It's no fun searching for help on the internet in the time of need but if you took the time to create a document with steps on how to recover from a failure, well life will be better.

But if you do desire the boot device for ESXi to be a mirror, that is fine. You can add additional hard drives/SSDs to your system as datastores to expand your VM capacity. You also could use the FreeNAS storage for your other VMs however there is a risk of if FreeNAS isn't running, then the other VMs would not be available.

 

[jgreco]

All the upgraded parts should be in next week, but I'm still not quite sure how I am going to do the storage.

I'm thinking that I'm going to load/boot ESXi from a USB stick

Use the 9211-4i controller to create 2 x 120GB RAID1 SSD Datastore
Use the 9211-8i controller to passthru to FreeNAS VM and create vpool of 4 x 2TB Mirrored vdevs (Gives me the option to upgrade 2 drives at a time to increase storage)

I cant decide if I really want to RAID1 the SSD drives for a single datastore or use them solo for 2 datastores and back the .vmdk files up to the NAS storage. Is there a better solution that I'm not considering?

Well, that depends on the value of your VM's. Also remember that backing up VMDK's doesn't necessarily guarantee high quality (usable/recoverable) backups. SSD is a lot cheaper than five years ago. I've been picking up low quality consumer grade SSD's like the CS1311 and SanDisk Ultra II 960's for around $220. These aren't suitable to high traffic, lots-of-write-activity datastores, but at some point you have to decide your time is more valuable than a little extra $$$ and that it is desirable to get fast storage at a modest price.

Best Buy's selling the CS1311 960GB's for $210 today.

 

[jgreco]

If it were me, and I can't speak to your own desires, but I'd not use the 9211-4i controller and I'd plug in both SSDs into the motherboard, install ESXi on it. This would result in two SSD's which can hold future VMs, just not mirrored. I'd put my FreeNAS VM on the first device but honestly it just doesn't matter. And if you do have a failure of the SSD to either boot ESXi or FreeNAS, because you kept your configuration backup files, it's a fairly painless recovery, it's more annoying than anything else. And at this point in time I would state that since you are building a new system, play with it, break it and repair it, take detailed notes on how to fix it when it does break. It's no fun searching for help on the internet in the time of need but if you took the time to create a document with steps on how to recover from a failure, well life will be better.

The moment you start hanging lots of stuff off a single machine, it seems like you should be able to afford to spend a little more to make that a high quality, low pain machine. The value of a 120GB SSD is probably like $40, so I'm going to disagree with @joeschmuck here and say you should just mirror the little buggers. If you need more VM space, go get yourself two cheap 240GB SSD's at $60 each and mirror them as a second datastore. The speed of local SSD storage on a hypervisor is awesome, generally much better than FreeNAS, and more pleasant to use.

 

[joeschmuck]

so I'm going to disagree with @joeschmuck here

I'm fine with that and completely understand why.

 

[Gunndy]

The moment you start hanging lots of stuff off a single machine, it seems like you should be able to afford to spend a little more to make that a high quality, low pain machine. The value of a 120GB SSD is probably like $40, so I'm going to disagree with @joeschmuck here and say you should just mirror the little buggers. If you need more VM space, go get yourself two cheap 240GB SSD's at $60 each and mirror them as a second datastore. The speed of local SSD storage on a hypervisor is awesome, generally much better than FreeNAS, and more pleasant to use.

This was kinda what I was considering. If I only use 2 ports on the 9211-4i at this time for mirroring the 2 x 120GB Kingston SV300 SDDs as Datastore1.....I can get two more SSDs at a later time if/when I need more high speed Datastore space and add those in mirrored as well. I also have a 3TB Seagate Barracuda laying around I could add to the on-board controller as a High capacity/low speed datastore for non-important VMs that I'm playing with or learning on. Obviously some VMs are easier to rebuilt than others

My main concern is that I've read about really bad performance with ESXi datastores set up as RAID on a 9211 (no on-board cache or battery). It appears that this is mainly an issue for spindle disks and SSDs don't have this same issue, but I can't find any definitive proof. Either way I'll test datastore performance myself as RAID1 vs Solo on the 9211 while building and make my choice based upon those numbers vs others experiences.

 

[jgreco]

This was kinda what I was considering. If I only use 2 ports on the 9211-4i at this time for mirroring the 2 x 120GB Kingston SV300 SDDs as Datastore1.....I can get two more SSDs at a later time if/when I need more high speed Datastore space and add those in mirrored as well. I also have a 3TB Seagate Barracuda laying around I could add to the on-board controller as a High capacity/low speed datastore for non-important VMs that I'm playing with or learning on. Obviously some VMs are easier to rebuilt than others

My main concern is that I've read about really bad performance with ESXi datastores set up as RAID on a 9211 (no on-board cache or battery). It appears that this is mainly an issue for spindle disks and SSDs don't have this same issue, but I can't find any definitive proof. Either way I'll test datastore performance myself as RAID1 vs Solo on the 9211 while building and make my choice based upon those numbers vs others experiences.

Throughput on a 9211 in IR mode is generally "not great". I've provided examples in the past, I think maybe in this thread with @Black Ninja but I'm not seeing it... and crap most of our LSI 2008 IR boxes have been recycled or upgraded. Wait. There's one. Let me twiddle around here a bit... okay, 8GB VM on an LSI 2008 IR mode with a pair of SSD's in RAID1 ... 171MBytes/sec sequential write, 299MBytes/sec sequential read. There's no HDD on the 2008 but I can tell you it'd be like 10MBytes/sec write or something really low like that. That's just basic dd testing to the raw virtual disk.

 

[Gunndy]

Throughput on a 9211 in IR mode is generally "not great". I've provided examples in the past, I think maybe in this thread with @Black Ninja but I'm not seeing it... and crap most of our LSI 2008 IR boxes have been recycled or upgraded. Wait. There's one. Let me twiddle around here a bit... okay, 8GB VM on an LSI 2008 IR mode with a pair of SSD's in RAID1 ... 171MBytes/sec sequential write, 299MBytes/sec sequential read. There's no HDD on the 2008 but I can tell you it'd be like 10MBytes/sec write or something really low like that. That's just basic dd testing to the raw virtual disk.

Thanks a ton....those numbers seem decent. Any chance you know what the throughput was/is for just one of those SSDs?

And by the way, here is a recap of the updated server build:

CPU: Intel Xeon E5-2670 2.6GHz 8-Core Processor
CPU: Intel Xeon E5-2670 2.6GHz 8-Core Processor
CPU Cooler: Noctua NH-D9L 46.4 CFM CPU Cooler
CPU Cooler: Noctua NH-D9L 46.4 CFM CPU Cooler
Motherboard: ASRock ASRock EP2C602-4L/D16 SSI EEB Dual-CPU LGA2011 Motherboard
Memory: Kingston 32GB (4 x 8GB) DDR3-1600 Memory
Memory: Kingston 32GB (4 x 8GB) DDR3-1600 Memory
Storage: Kingston SSDNow V300 Series 120GB 2.5" Solid State Drive
Storage: Kingston SSDNow V300 Series 120GB 2.5" Solid State Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Red 2TB 3.5" 5400RPM Internal Hard Drive
Storage: Western Digital Caviar Green 3TB 3.5" 5400RPM Internal Hard Drive
Power Supply: EVGA 850W 80+ Bronze Certified Semi-Modular ATX Power Supply
Other: LSI Internal SATA/SAS 9211-8i 6Gb/s PCI-Express 2.0 RAID Controller Card
Other: LSI Internal SATA/SAS 9211-4i 6Gb/s PCI-Express 2.0 RAID Controller Card
Other: Rosewill RSV-L4412 - 4U Rackmount Server Chassis, 12 SATA / SAS Hot-swap Drives

 

[Dice]

Power Supply: EVGA 850W 80+ Bronze Certified Semi-Modular ATX Power Supply

Has this option been discussed in the thread?
Bronze....?
When building such a killer machine, at least a Gold certified PSU.
I've seen Platinum ATX PSU's dropping significantly in price recently.
Cheers /

 

[jgreco]

Thanks a ton....those numbers seem decent. Any chance you know what the throughput was/is for just one of those SSDs?

Not really. They're ~5 year old SATA 6Gbps SSD's, so figure "closing in on full 6Gbps speed" but probably not *right* there.

 

[Robert Trevellyan]

I also have a 3TB Seagate Barracuda laying around

Which model?

 

[Gunndy]

ST3000DM001 with updated firmware to eliminate the "click of death" caused by constant head parking. I know its not a 24/7 drive, but it would also really only be used for VMs that are not powered on full time. I like to tinker / load all kinds of OSs and try them out. Also, will have 2-3 Windows 7 VMs that are powered off most time and used for testing/experimenting with Active Directory management a little more.

I've worked in IT for 20 years and have been mostly dealing with workstation management / application packaging. I only have an intermediate knowledge of many of the infrastructure components in an Enterprise Network and want to get better. That is one of the main reasons I'm wanting to build an ESXi home lab / server.

I'd like to expand my personal consulting business to include a more robust "Soup to Nuts" portfolio.

 

[Robert Trevellyan]

ST3000DM001

That model has a terrible reputation - probably best not to use it for anything.

 

[Gunndy]

That model has a terrible reputation - probably best not to use it for anything.

I've been using another of the same model as a storage drive on my main gaming/production rig for 3 years without issues. I think applying the firmware fix for the "click of death" is key. And even if the drive does go bad, as I said, I'm not going to host any VMs that I care about loosing on it and it saves me from having to spend another $100 right now on another WDRed.

Keep in mind that while this is a rather monster server build, MANY of the parts are repurposed stuff I already have. The only thing I've really bought new are the MB, 1 extra XEON E5-2670 (and cooler), and 32 GB Memory, and an LSI 9211-41 controller. Everything else I already have running in my existing FreeNas box which is an I3-4130 with 32MB Ram and 12TB storage (currently 8 x 2TB RaidZ2) or laying around.

Going to then re-purpose the 4130 CPU and motherboard into a standalone pfSense router.

 

[joeschmuck]

I've worked in IT for 20 years

That statement brought up a bad memory, not something that happened long ago but it happened last Thursday, and nothing against you... So last Thursday at 9AM I got a call from my companies IT service desk and they were going to remove an application I didn't want on my system. The IT professional on the other end said it was a piece of cake, take only a few minutes. Now I had planned to leave work by 9:30AM that day because I had all my time in and I was taking off for the long weekend. So the guy jumps into the Add/Remove Programs and couldn't find it listed (duh, I already checked that), then he starts running a few things and starts to delete a few things, then goes into the MSCONFIG and resets everything to Normal Boot, and he never thought to investigate what other IT folks had turned off in the past. Reboot and dead. Windows fails to boot. The guy first says that I should bring the laptop in to IT to fix it when I remind him that there is no local IT department where I work at a remote site. Next he says I could boot from a Windows CD and fix it, then I remind him that the drive is encrypted, like all company laptops are and that I won't have access to the C: drive. He tells me he's sorry and will ship me a replacement laptop. I ask about my data on the drive and don't get the feeling he has a clue on how to recover all my work. At this point in time I'm very angry at his incompetence, so much for it being a piece of cake. I will have lost some data and scripts if he cannot recover my data but thankfully I back up the really important data nightly to the company server, so I lost about 4 hours of work and unfortunately some URLs to some military sites which had recently changed IP addresses. And to be honest, most of the IT people I work with are very good and have a clue what they are doing and always in the past have stopped short or doing something bad. One time the person knew there would be a logon issue after a reboot and set me up with a local account to get past that. If he didn't think about that possible issue then I'd have shipped back a laptop that time too.

Edit: Sorry for ranting... Just hit that spot.

 

[jgreco]

then goes into the MSCONFIG and resets everything to Normal Boot, and he never thought to investigate what other IT folks had turned off in the past. Reboot and dead. Windows fails to boot. The guy first says that I should bring the laptop in to IT to fix it when I remind him that there is no local IT department where I work at a remote site. Next he says I could boot from a Windows CD and fix it, then I remind him that the drive is encrypted, like all company laptops are and that I won't have access to the C: drive. He tells me he's sorry and will ship me a replacement laptop.

There's usually this relationship between how inconvenienced someone is by a problem of their own creation and how careful they are in avoiding them (whether now or in the future).

I have a tendency to be able to work on equipment a continent away because I very carefully think through the consequences. I find that many of the younger guys in IT aren't able to predict the consequences of their actions as well, which often leads to such things.

 

[Gunndy]

There's usually this relationship between how inconvenienced someone is by a problem of their own creation and how careful they are in avoiding them (whether now or in the future).

I have a tendency to be able to work on equipment a continent away because I very carefully think through the consequences. I find that many of the younger guys in IT aren't able to predict the consequences of their actions as well, which often leads to such things.

I agree....I know I've been told quite a few times at work by the "younger" crowd that I overthink things alot. And yet, I haven't been the one responsible for a system outtage at work in over a decade.

 

[joeschmuck]

Yea, I feel this was a fairly new person since I've never heard of him before and his photo looks like he just got out of high school, and I know most of the IT folks. He will learn in time but I'd prefer it if he doesn't learn it at the expense of a remote user, where it's very difficult to correct the issue in any reasonable time.

EDIT: Sorry to take this off topic.

 

[Dice]

Edit: Sorry for ranting... Just hit that spot.

*extremely tempted to bring up a massive <wall of rants> regarding the local IT-department's inexcusable lack of respect of user data*

 

[cyberjock]

If you plan to use the storage for ESXi VMs, your only viable option is mirrors. So either go RAIDZ2 and don't use the storage for VMs, or go with mirrors and pay the 50% penalty for redundancy. :(