Will this configuration work?

[Redemption]

I built a server with the following specs;

Supermicro X11SPI-TF
Intel(R) Xeon(R) Silver 4210 CPU @ 2.20GHz
190 Gigs DDR4 Registered Memory
MegaRAID SAS Raid controller for ESXI
9211-8i 6Gbps HBA LSI
6 x 1TB SSD drives
4 x 14TB Seagate Exos

I installed EXSI, used the MegaRAID controller, and created a Raid-10 Array out of 4 SSD drives. I installed a few VMs, including Truenas. I attached the remaining drives directly to the motherboard. From the memory, I dedicated 100 Gig of RAM for TrueNas.

I did a Passthrough and passed the Lewisburg Sata AHCI controller to TrueNas VM. The plan is to create a Mirror with the two SSD drives strictly to Install TrueNas apps. Will that work, or should I use the 9211 instead?

I am also debating whether I should do another mirror for the 14TB drives or use a RaidZ2. Either way, I lose half the drives, but I think I get a little performance boost with the mirror on reads.

The use case will be some VMS, including a Domain controller. I expect to run the typical apps, plex, sonarr, radarr etc. I will be doing video Editing; there will be transfers of large data files here and there. The server will be connected to a 10-gig switch.

Thank for the help in advance!

 

[Redemption]

Did I ask foolish questions? or perhaps did not ask the right questions? Lots of views and 0 replies :(

 

[HoneyBadger]

Perhaps just didn't get the right view yet? ;)

Your ESXi configuration is good, in that you have a separate storage controller exclusively for the VMware datastore, and plan to attach the 4x1T SSD's there.

Do you plan on running your more performance intensive VMs (domain controller, etc) on the MegaRAID-backed VMFS datastore, or loop back to TrueNAS?

I did a Passthrough and passed the Lewisburg Sata AHCI controller to TrueNas VM. The plan is to create a Mirror with the two SSD drives strictly to Install TrueNas apps. Will that work, or should I use the 9211 instead?

What model are the SSDs? LSI SAS controllers are picky about the models that they will allow TRIM to work with, so using the SATA controller may be better here from compatibility and latency.

I am also debating whether I should do another mirror for the 14TB drives or use a RaidZ2. Either way, I lose half the drives, but I think I get a little performance boost with the mirror on reads.

If you plan to put VMs on the 14TB drives and use it as a "loopback" datastore then definitely mirrors for the performance. I'd lean towards them anyways as it will make for faster resilvers, but it does put you at the risk of a two drive failure taking out the pool.

For your large-file dataset, use recordsize=1M to squeeze out a bit more efficiency there, since you aren't likely to ever work with files smaller than that.

 

[jgreco]

Did I ask foolish questions? or perhaps did not ask the right questions? Lots of views and 0 replies :(

There's a limited number of us who do both virtualization stuff and FreeNAS.

Perhaps just didn't get the right view yet? ;)

Alternatively I would say you failed to wait a sufficient amount of time for an answer. See above about limited number of us doing virtualization. 35 minutes in between posts is ... insufficient.

MegaRAID SAS Raid controller for ESXI
9211-8i 6Gbps HBA LSI

I dedicated 100 Gig of RAM for TrueNas.

Plus making some good choices up front reduces the number of bits of meaningful critique you're likely to get from the community, though @HoneyBadger naturally managed to dredge up a few points.

One possible point of concern is that ESXi 7 with the 9211-8i controller has some quirks, including the dreaded Doorbell handshake issue, which @Spearfoot has noticed (speculated?) only seems to affect the 2008 based chipset, and ESXi sometimes has boot issues with the 6Gbps LSI controllers installed, even if configured for passthru.

Please make sure to avail yourself of the existing resources, which may include

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

"Absolutely must virtualize FreeNAS!" ... a guide to not completely losing your data.

[---- 2018/02/27: This is still as relevant as ever. As PCIe-Passthru has matured, fewer problems are reported. I've updated some specific things known to be problematic ----] [---- 2014/12/24: Note, there is another post discussing how to deploy a small FreeNAS VM instance for basic file...

www.truenas.com

10 Gig Networking Primer

This document is now available in the Resources section! Use the tabs above to navigate to the document itself. The current version also follows inside the spoiler tag: Also, if you'd like to repost this elsewhere, I'd appreciate it if you would have the courtesy to ask. And to credit the...

www.truenas.com

 

[HoneyBadger]

Alternatively I would say you failed to wait a sufficient amount of time for an answer. See above about limited number of us doing virtualization. 35 minutes in between posts is ... insufficient.

To be fair, he waited a day and 35 minutes, but I'll attribute that to an improper RFC 2324 implementation on your end. ;)

One possible point of concern is that ESXi 7 with the 9211-8i controller has some quirks, including the dreaded Doorbell handshake issue, which @Spearfoot has noticed (speculated?) only seems to affect the 2008 based chipset, and ESXi sometimes has boot issues with the 6Gbps LSI controllers installed, even if configured for passthru.

That's a good point to raise. The boot issues would impact ESXi 7 or later where the SAS2008 is deprecated, but the doorbell handshake seemed to be present as far back as ESXi 6.7 and FreeNAS 11.x based on reports in that thread. Not sure if that's a case where it might have been resolved in some combination of newer HBA firmware, ESXi software, or upstream BSD in the mps driver.

 

[Redemption]

Haha, you both Rock! Sorry for being impatient, just want to start throwing data at this puppy.

After doing some research, I had decided on not using the LSI, VMware is reporting it as deprecated. It still works but may not in future. So I took the card out. The SSD are Samsung branded, Included a picture.

I plan on running a domain controller but VM's will be on the MegaRAID-backed VMFS datastore. I will be running only APPS on TrueNas, no vm's. I think I am good then, thank you all for the help!!!

 

[jgreco]

I had decided on not using the LSI, VMware is reporting it as deprecated.

A 6Gbps card I'm guessing. Most of the 12Gbps cards seem to be fully supported, though sometimes with strange firmware versions required. VMware deprecated Linux driver compatibility and kicked 6Gbps cards out of the supported list at the same time. 12Gbps LSI RAID cards should work fine if needed.

 

[Redemption]

Good to know, for now I will save the money. Should the need arise, at least there is an option! Thanks for the help.