Is that a SATA SSD? If it is, that is going to be a performance bottleneck. Your best speed from a SATA SSD is around 450MB/s, after overhead.with an SSD slog and L2ARC (stored as VMDK on my 960 evo SSD)
Is that the same 960 that you are also using for you SLOG and to host the VMs?If I copy a large file (7GB) from a test VM which is on the 960 evo (to avoid read speed bottlenecks)
dd if=/dev/zero of=/dev/zvol/VMs/ESXi-iSCSI-DS1 bs=7G count=1 oflag=direct
I’m new to FreeNAS and am looking into a build of my own, and really like the idea of an AIO server that will closely follow your build – in particular I’m looking at the Supermicro 5028D-TN4T with 64GB RAM (https://www.supermicro.com/products/system/midtower/5028/sys-5028d-tn4t.cfm). Initially I think I'll probably only be running about half a dozen VMs at any one time.
A couple of questions that I had:
You passed through the AHCI controller to FreeNAS – I have room for 4 HDDs and 2 SSDs. Assuming I used a M2 nvme boot disk, and used this for swap/L2ARC as in your initial build, could the 2 SSDs be used as a separate pool in FreeNAS for NFS or iSCSI VM datastores?
If so, do you think that would reduce or remove the need for a dedicated slog device?
Alternatively, with a large enough M2 boot drive, could I just store the VMs on the M2 and remove the need for sharing VM datastores through FreeNAS? Are there any downsides to this?
Lastly, with 4 HDDs, I’m looking at 8TB drives but I’m torn between a mirrored pool or RaidZ2. Scouring the forums there are very differing opinions - I’m leaning toward RaidZ2 for the added redundancy, but am concerned about resilvering times for the 8TB drives. Do you have any thoughts one way or the other?
Definitely still have a lot of learning to do, but just thought I’d get your opinion on this particular setup before I jump into it headfirst!
Cheers again for this great guide!
The RAIDz2 pool will give you roughly the IOPS of a single drive where the pool with two mirror vdevs will give you roughly the IOPS of two drives. IOPS is a function of the number of vdevs in the pool. More vdevs generally equates to more IOPS and in that case, more mirror pairs makes the pool 'faster' although it is not really raw speed that I am referring to. I have a pool of 16 drives to give me 8 mirror pairs so I can have better random I/O. I would suggest that you should go with more drives.Lastly, with 4 HDDs, I’m looking at 8TB drives but I’m torn between a mirrored pool or RaidZ2. Scouring the forums there are very differing opinions - I’m leaning toward RaidZ2 for the added redundancy, but am concerned about resilvering times for the 8TB drives.
Absolutely amazing, what a guide, thanks Stux. I've been tinkering with this stuff for years and learned a lot from reading this.
I have one question though, why did you share both via nfs and iscsi from freenas to ESXi? I know nfs was named storage and iscsi vm but is there a reason behind this choice of nfs vs iscsi? Or was it just to show examples of what is possible?
That would work, but your VMs would not be ZFS protected. No redundancy, no bitrot protection and susceptible to single point Of failure. But VMs are not irreplaceable data. So store the data on ZFS.