The HP N40L I believe only supports 8GB of RAM.
Minor correction: The N40L supports 16GB the same way many Atoms support 8GB... undocumented-but-works. 16GB is big enough to be ESXi-useful and you can probably find some trite VM's that you could run alongside FreeNAS.
You shouldn't be using less than 6GB for FreeNAS, so why do you want to do this at all? (I think this is why jgreco is asking the question).
Right line of thinking, wrong specifics.
Problem 1:
ESXi requires a datastore on which to maintain VM data files and disk images, and that cannot be self-hosted on a virtualized NAS. Chicken-and-egg. ESXi will not use a USB disk or USB flash for datastore, so my question was summarized precisely by what I asked. The N40L has five bays total. It *is* possible to add another controller card such as a BR10i or M1015 or even a plain SATA controller for ESXi to use for datastores, and there's space in the N40L case for an extra 2.5" disk or two if you use tape. However, in general, the N40L will be quite small to host 5 full size disks PLUS extra hardware for ESXi datastores.
Problem 2:
HP MicroServer does not support PCI passthrough, which is a damn shame.
Fundamentally, FreeNAS is a real nice system and a real nice concept. However, the USB flash thing has its ups and downs. For a home user, it's probably pretty great. For us, it's real inconvenient to have a USB flash on a system that's half a continent away, and I've seen some failures where the flash has somehow become corrupt... it's great that the data on the FreeNAS server is well protected, but the FreeNAS system itself is poorly protected against failures. Updating it remotely (remember the 1GB->2GB size bump?) or replacing possibly failed devices is a pain.
That's where ESXi could really shine. Stick in an inexpensive ESXi-supported RAID controller. Throw in some small SSD's. Suddenly you have awesome-fast boot (no more minutes to load FreeNAS over USBv1!) AND it is redundant AND you can do installs and upgrades easily. With ESXi you basically have a large supply of USB keys that you can switch around without touching the hardware, and you can even run more than one at a time, for that annoying (and inevitable) case where you forgot to set something up and you wish you could have both the old and new server running at the same time so you could see just what you did last time.
That plus the other positives of virtualization make me understand why someone would try to do this. However, from a practical point of view, the Microserver is a poor platform for ESXi. It is lacking many of the features that would make for an awesome hypervisor platform.
Anyways, we're an ESXi4 shop here, so I have no comment on RDM (an ESXi5 feature) other than I'd think it could be made to work.
But we've seen a lot of people come through here with tears for their shattered data eaten by their questionable virtualization platform. I'm pretty convinced that the only safe and sane way to virtualize FreeNAS is by starting out with a hardware platform that is absolutely positively designed for virtualization, which means a modern Dell/IBM/HP/Supermicro with a proven and tested server-grade VT-d implementation, the correct Xeon CPU, and the correct PCI-e hardware. noobsauce80 and I spent a little time trying to get ESXi+FreeNAS up and running on something that theoretically supported it, which turned into kind of a horrifying scenario where it kind-of sort-of seemed to work (worked fine, then didn't work, etc). It was bad. I see strong, compelling reasons for making sure that not only the storage devices but also the storage controllers are owned by the FreeNAS kernel - passing the controller itself in seems to be best.
Me, I'm paranoid, and so I've designed a FreeNAS server that also happened to be fully supported by ESXi. Lots of experimentation had already convinced me that our N36L is a waste of watts, because for only ~10 watts more a Xeon E3-1230 platform provides better performance while also being nearly idle serving files under load. So our old 1U storage servers have been getting upgraded to X9SCI-LN4F's with 32GB, E3-1230, and a M1015 in crossflashed IR mode to give ESXi a RAID1 datastore for boot and FreeNAS VM. The difference is that you can actually load up some heavier VM's on the unused capacity of the Xeon and get virtualization-style efficiencies.
So I definitely understand why people want to run ESXi. However, as much as I might like virtualization, the fact remains that noobsauce80, myself, and many others have seen both the relatively few successes (usually with higher end enterprise grade server gear) and the many failures, most especially including the many people who have entirely lost their pools when something went awry.
And I tell you all of this so that you have the full context of why I'm about to say this:
If you have an N40L, it makes a great (if somewhat underpowered) FreeNAS box. Set it up, stick it in a corner, and leave it the hell alone. But don't make it more complicated. It is a poor ESXi platform.