Virtualizing FreeNAS over ESXi using NVMe locally, not iSCSI targeting. Is NFS the way to go?

[Ryan Messner]

So I'm totally reworking our virtual and storage environments and we're small enough that I can run most everything I need on two pools by using 2x 2TB Intel P3500's

Currently we use iSCSI over teamed 4x1GbE to connect ESXi hosts and spinning disks on an Equallogic, let's be honest it's hot garbage.

In the process of the refresh, I feel like given our size and for the performance gains, it's better to use the NVMe locally and replicate to off site.

I've done some reading about NFS vs iSCSI with VMFS but every application I've encountered focuses on LAN bandwidth constraints and ensuing CPU overhead, etc. which has led me to this: If I set these guys up to run NFS and put VMFS on top of (next to) it, am I going to regret it? Or are those pitfalls strictly due to

For simplicity's sake, I want to add these P3500's to FreeNAS purely for the sake of replication and then have them mounted in ESXi to run VDI's and Servers. I then to back up my pools (if you can call them that) over WAN to the DR site.

Potentially stupid followup question if iSCSI is truly the way to go, can I map PCIe devices internally via iSCSI?

 

[tvsjr]

I've done some reading about NFS vs iSCSI with VMFS but every application I've encountered focuses on LAN bandwidth constraints and ensuing CPU overhead, etc. which has led me to this: If I set these guys up to run NFS and put VMFS on top of (next to) it, am I going to regret it? Or are those pitfalls strictly due to

For simplicity's sake, I want to add these P3500's to FreeNAS purely for the sake of replication and then have them mounted in ESXi to run VDI's and Servers. I then to back up my pools (if you can call them that) over WAN to the DR site.

I'm not clear what you're trying to do. FreeNAS needs full control of the devices being used and the storage controllers. The typical use case would be to build a FreeNAS server, configure pools (with many caveats that you should read up on), and expose that storage to ESXi via NFS and/or iSCSI.

Potentially stupid followup question if iSCSI is truly the way to go, can I map PCIe devices internally via iSCSI?

Err? iSCSI is a protocol... you can't directly map a PCIe device (or any other device) via it. You would need to use a storage subsystem of some type (FreeNAS or others) to create a storage array and to provide an iSCSI target, with the target pointed to that device.

Some clarification would be very helpful.

 

[Ryan Messner]

I'm not clear what you're trying to do. FreeNAS needs full control of the devices being used and the storage controllers. The typical use case would be to build a FreeNAS server, configure pools (with many caveats that you should read up on), and expose that storage to ESXi via NFS and/or iSCSI.

I basically want to load ESXi on some boot media, then spin up FreeNAS with hardware passthrough for the P3500's and create pools. I then want to use those pools on that existing ESXi host where I will spin up additional Virtual desktops and servers to run on the pools.

Err? iSCSI is a protocol... you can't directly map a PCIe device (or any other device) via it. You would need to use a storage subsystem of some type (FreeNAS or others) to create a storage array and to provide an iSCSI target, with the target pointed to that device.

Some clarification would be very helpful.

I knew this was a stupid question in principle, hence the caveat. Thanks regardless for the confirmation.

 

[Spearfoot]

Sounds like you want an AIO (All-in-One), i.e., boot ESXi from a local datastore, boot FreeNAS from the same (or other local) datastore, passthrough the P3500s to the FreeNAS VM, then use FreeNAS to create add'l datastores on a dataset created on the P3500s. I'm not sure if you can do that; it depends on whether or not you can pass-through the P3500's to the FreeNAS VM. I've built AIO systems, but I've only ever passed an HBA through to the FreeNAS VM.

 

[Ericloewe]

I basically want to load ESXi on some boot media, then spin up FreeNAS with hardware passthrough for the P3500's and create pools. I then want to use those pools on that existing ESXi host where I will spin up additional Virtual desktops and servers to run on the pools.

Holy mother of layering violations, that's a genuine Very Bad Idea (TM).

That idea can fail in so many ways that it's best avoided.

If you want to do that, wait for FreeNAS 10 and run your VMs in bhyve.

; it depends on whether or not you can pass-through the P3500's to the FreeNAS VM.

That's the easy part, they're regular PCIe devices.