BUILD ESX FreeNAS Box Hardware & Setup Questions

[Dice]

At this point, I feel additional research and >reading< should be priority.
(the forum has up until quite recently been resenting threads characterized by freenasnewbie+ESXi. One tagline on the topic is: >if you need help, you shouldn't<)

 

[snaptec]

Could I just use a single SSD as my ESXI install + datastore? What are the dangers to the Freenas VM if that SSD would fail? Could I just boot up the machine off a flash drive with Freenas install on it and import the pool?

Maybe, its just like pulling the power plug on the FN. Normally it comes up and everything is ok, but it also can boot and the pool/data is corrupted..

 

[joel3452]

I appreciate everyone's advice thus far. I did take your advice Dice and do lots more reading over the weekend. I think my final design will be similar to what I originally wanted with a raid 1 running off of a IBM ServeRaid M1015 for the ESXi datastore for the VM storage and the LSI passed through to Freenas.

For the initial setup and such though I am going to do a bare metal and then later export the configuration into ESXi. I also think I need to just play around with different configurations and settings to see what works best for my setup.

I do think I will need to get some cables though for the intial setup. Its hard to tell from Supermicro's site if the motherboard comes with the SAS to 4 sata cables for the LSI controller. It looks like it will come with several SATA cables, but not the SAS cables. Would these type of cables be the correct ones: Cable Matters Internal Mini-SAS to 4x SATA ?

 

[Mlovelace]

Would these type of cables be the correct ones: Cable Matters Internal Mini-SAS to 4x SATA ?

No, the motherboard you're looking at uses the mini-SAS SFF8643 connectors you'd want something like these https://www.amazon.com/dp/B00VAM1R90/?tag=ozlp-20.

 

[joel3452]

No, the motherboard you're looking at uses the mini-SAS SFF8643 connectors you'd want something like these https://www.amazon.com/dp/B00VAM1R90/?tag=ozlp-20.

Thanks for the correction. I couldn't tell from the small picture and didn't see the SFF port number on the specs sheet. I will order 2* CableCreation Internal HD Mini SAS (SFF-8643 Host) - 4x SATA (Target)
to go along with the order.

And then it looks like the IBM M1015 uses a SFF-8087.

 

[wblock]

or could I use a IBM ServeRaid M1015 and create a Raid1 of 2 ssd's and use that for the ESXi install + datastore. And then store all of the VM's on that? I realize I wouldn't get the ZFS snapshots or things, but the VM performance should be quite fast and fairly reliable right?

If the M1015 is set to IT mode, ZFS will happily create a mirror of two SSDs, and you will have the ZFS features on that mirror.

 

[joel3452]

But what performance impact would that have since then it would need to be shared out through iSCSI/NFS?

Also where would the Freenas VM be installed? I know you can install ESXi on a flash drive, but you would still need to have a hard drive/dom for the VM datastore.

So it seems like there are two options
1. ESXI - Flash Drive, Freenas Datastore - DOM/HD, Share SSD for other datastores.
2. M1015 set to raid 1, install ESXI, and all data stores. Have raid redundancy, but lose ZFS snapshots/checksums, but ESXI and Freenas VM have redundancy.

 

[Spearfoot]

If the M1015 is set to IT mode, ZFS will happily create a mirror of two SSDs, and you will have the ZFS features on that mirror.

But @joel3452 is talking about using a RAID1 array on the M1015 HBA as the ESXi boot device and datastore for the FreeNAS VM (and possibly other VMs). I do this with both of my FreeNAS systems; in my case, I used Dell H200s flashed to P20 IR mode and created a RAID1 array of matching Intel DC S3500 SSDs.

 

[Spearfoot]

But what performance impact would that have since then it would need to be shared out through iSCSI/NFS?

Also where would the Freenas VM be installed? I know you can install ESXi on a flash drive, but you would still need to have a hard drive/dom for the VM datastore.

So it seems like there are two options
1. ESXI - Flash Drive, Freenas Datastore - DOM/HD, Share SSD for other datastores.
2. M1015 set to raid 1, install ESXI, and all data stores. Have raid redundancy, but lose ZFS snapshots/checksums, but ESXI and Freenas VM have redundancy.

This is getting a little confusing...

To virtualize FreeNAS under ESXi, it must be installed on a datastore that is not controlled by the FreeNAS VM. And you must have a HBA you can pass through to the FreeNAS VM, either the motherboard's SATA controller or a built-in or slot-based HBA.

You can install ESXi on a USB flash drive... but ESXi won't allow you to use a USB flash drive as a datastore. So to go this route, you install and boot ESXi from a USB flash drive and create a local SSD (or HDD) datastore where you then install your FreeNAS VM. Use two SSDs and you can mirror your FreeNAS VM during installation. Once you get FreeNAS running, you can create additional FreeNAS-based datastores based on your FreeNAS pool(s) and install as many VMs there as you please.

If you want redundancy for your ESXi boot & FreeNAS VM datastore, you can set up a RAID1 (or fancier) array on a separate RAID controller supported by ESXi. LSI-based RAID controllers fit the bill for this purpose and, as I mentioned above, I use this approach in my virtualized FreeNAS systems (see 'my systems' below). But note that this controller is separate from the controller you're passing through to the FreeNAS VM.

Will your FreeNAS-based datastore be slow compared to SSD? Probably so. But you can still get pretty good performance, especially if you use a separate datastore network in your All-in-One. See this thread for benchmarks:
Physical vs. Virtual FreeNAS + iSCSI vs. NFS: performance testing results.

There! Clear as mud, right? Anyway... the point is that there is more than one way to skin this cat, but you have to know what you're doing. Which brings us back to the standard advice we give to novice virtualizers-of-FreeNAS: if you have to ask how to do it, you probably shouldn't even be trying to do it. :)

 

[joel3452]

Yep that clear as mud explanation is exactly why I want to keep ESXi storage separate from Freenas storage.

That is why I plan on passing through the LSI3008 to Freenas and getting a used M1015 off Ebay to create the raid1 of SSDs for ESXi install + Datastores. Basically the same setup as your current system except with a M1015 as they seem recommended and fairly cheap used.

And yes I completely understand the logic of don't try if you have to ask. I am going to go slow and careful and keep all of my data on my current server and then convert it to backup space after the new guy is ready for prime-time. I also am going to install Freenas and configure it on a Flash drive that I shove in the lockbox at home just in-case ESXi ever decides to cause issues.

Edit:
What are you thoughts on the SanDisk Extreme Pro 2.5" 240GB SATA III MLC ? They seemed to be a fairly well regarded MLC flash drive and are supposed to have consistent performance rather than the extreme variations of the newer TLC drives which are fast for the few GB of a write.

 

[Spearfoot]

What are you thoughts on the SanDisk Extreme Pro 2.5" 240GB SATA III MLC ? They seemed to be a fairly well regarded MLC flash drive and are supposed to have consistent performance rather than the extreme variations of the newer TLC drives which are fast for the few GB of a write.

Dunno anything about 'em. But with new Intel DC 3500's on eBay for reasonable prices, why buy consumer-grade SSDs?

 

[joel3452]

Is the specific one you linked to where you got yours? I am hesitant to buy SSD's and flash memory off of ebay because of the large amount of fakes going around. The prices are definitely great for a 240GB Intel enterprise ssd. The same guy goes for $230 on Newegg.

 

[Spearfoot]

Is the specific one you linked to where you got yours? I am hesitant to buy SSD's and flash memory off of ebay because of the large amount of fakes going around. The prices are definitely great for a 240GB Intel enterprise ssd. The same guy goes for $230 on Newegg.

Yeah, I understand your concerns about eBay sellers... That's why I only buy from USA-based sellers with large numbers of feedbacks and a very high positive feedback rating. I haven't purchased anything from this particular seller, but he has 5,627 ratings and 100% positive feedback.

 

[Magius]

Speaking of Intel SSDs on Ebay, just this weekend I fell into a deal on a DC S3700, 100GB, for $51. At that price, I figured it doesn't even matter if I "need" a SLOG or not, I'd be foolish not to have one :) If you keep your eyes out and use the "best offer" feature, you can snag some great stuff on Ebay.

 

[joel3452]

That is a real good price for a S3700. Was it a used drive pulled from a data center? Did you pull up the Intel toolbox to see how much data was written to it?

Edit: Those S3500's on Ebay are very tempting. They do seem almost too good to be true though compared to Amazon/newegg/etc pricing.

 

[Magius]

That is a real good price for a S3700. Was it a used drive pulled from a data center? Did you pull up the Intel toolbox to see how much data was written to it?

Yes, it was a used drive, "working pull from server" or whatever in Ebay language. It just shipped yesterday, and of course as soon as I get it I'll be looking at its history.

With something like an s3500 I would have been a little more concerned about endurance, as it might have taken a beating in the datacenter and used up most of its, what, 50-75 TBW rating? With an s3700 though, they're rated for over 1 PBW, and I cant imagine it getting beaten up so bad that it's anywhere near that number...? I'll come back and let you know, especially if I'm proven wrong :)

 

[joel3452]

According to the spec sheet its Endurance Rating (Lifetime Writes): 10 drive writes per day for 5 years. So that should be in the neighborhood of 1.825 Petabyte. I don't know if Intel does its math to include write amplification or not. Either way, should have lots of life as long as its wasn't some dedicated database drive that was being abused 24/7.

 

[Ulises]

5: VMware Snapshots in FreeNAS will do nicely for taking snapshots of your VM's.
6: Negative. It will come with SAS cables but not breakout ones.

Another thing you can do as well, though up to you, is install ESXI on a thumb drive, pass the SSD's through to FreeNAS, make mirrored Vdev out of them and serve them up as an iSCSI target for ESXI. That way you'll have some protection on your ESXI data store while still getting the performance you want. Only downside is you'll need to make sure FreeNAS powers up before the rest of your VM's do.

Unless someone has already noted, you have no datastores to install freenas by doing this.

 

[Ulises]

Basically, create a hardware raid for your datastores, install esxi onto a thumbdrive, then install freenas on your previously created raid, and then you know, passthrough your storage controller for freenas and configure all your zfs storage.

 

[joel3452]

Yep that is the general plan after I run it on bare metal for a while to make sure everything looks good and goes smoothly.