Looking for advice on completing my new FreeNAS build considering FreeNAS on top of ESXi

[Mannekino]

Dear FreeNAS community,

I'm a bit of a pickle and reaching the limits of my current FreeNAS knowledge. For the past years I've been using a HP MicroServer N40L but it's time for a nice upgrade. I feel like the parts I currently have for my new NAS build are a bit too powerful for just FreeNAS and I would like to also create VMs for different purposes. I am considering two options:

• Running FreeNAS on bare metal and using the VM option in the newest version of FreeNAS
• Installing VMware ESXi and run FreeNAS in a virtual machine with direct access to my storage drives (if think this is possible but not 100% sure yet with my hardware)

Before I go any further he is the list of parts I currently have for my new build:

• Motherboard: ASRock H87M-ITX (with 4 x SATA3 6.0 Gb/s ports)
  
• CPU: Intel Core i7-4785T
• CPU cooler: Noctua NH-L9x65
  
• Memory: Corsair Vengeance Low Profile 16 GB Dual Channel DDR3 Memory Kit
• Hard drives: 4 x WD Red NAS WD40EFRX (4 TB)
  
• Chassis: Fractal Design Node 304 Black
• PSU: Seasonic FOCUS PLUS Gold 550

As you can see from the specs the motherboard only has 4 SATA ports and I will put 4 hard drives in. Right now I'm running FreeNAS from a USB key on my HP MicroServer N40L and this has worked just fine. However I am thinking to maybe use the PCIe slot I have for my new build for an small NVMe drive to run either ESXi on or FreeNAS itself if I chose to go with FreeNAS on bare metal.

Here are my questions:

1. Is it possible with my hardware to run FreeNAS on top of ESXi and can I give FreeNAS direct access to my hard drives (I assume this is the recommened method)?
2. If this is possible would you recommend I get a PCIe adapter with an NVMe drive? If so, how much storage would I need? Could for example an Intel 32 GB Optane drive work?
3. Do you guys have any recommendations for a PCIe adapter?
4. Should I get a M.2 SATA or NVMe drive? I am correct to assume that a SATA M.2 drive would require a PCIe card with a SATA controller on it whereas an NVMe drive is just a PCIe device that either ESXi or FreeNAS could boot from?
5. Could I use the M.2 drive as both a boot drive for ESXi and also as a SLOG device? I assume if I chose to run FreeNAS directely on bare metal configuring the M.2 drive as a SLOG device would be easier?
6. I watched a review of the new VM functionality in FreeNAS is this feature mature enough yet to use? Would recommend this over an ESXi setup?
7. If I decide not to go the ESXi route but to just run FreeNAS directly on bare metal would you still recommend to get a M.2 drive to install FreeNAS on or should it run fine from a USB 3 key?

Hopefully these questions make sense I'm still trying to orient myself.

Kind regards,

Mannekino

 

[Chris Moore]

Before I go any further he is the list of parts I have for my new build:

Is this what you already have, not what you plan to buy? I am guessing you are reusing equipment you already had from something else?

 

[Mannekino]

Is this what you already have, not what you plan to buy? I am guessing you are reusing equipment you already had from something else?

Sorry I just edited my post. I already had most of the parts but I just ordered the hard drives, chassis and CPU cooler. The motherboard, CPU and memory I had already.

I hope my questions aren't too much of a mess. I feel like my final decision to get a PCIe adapter with an M.2 drive depends on wether or not I will go the ESXi route or FreeNAS on bare metal route. If I go the bare metal route I think running FreeNAS on a USB key would be fine. I'm reading up on using an SSD for SLOG + L2ARC and the benefits this provides but my current knowledge level is too low.

 

[Chris Moore]

Is it possible with my hardware to run FreeNAS on top of ESXi and can I give FreeNAS direct access to my hard drives (I assume this is the recommened method)?

That hardware is not good for either ESXi or FreeNAS. That might make a fair desktop computer for a child that doesn't play video games.

If this is possible would you recommend I get a PCIe adapter with an NVMe drive? If so, how much storage would I need? Could for example an Intel 32 GB Optane drive work?

I would suggest a different system board, processor and memory. What you have is not appropriate to the task.

Despite your statement that you have used FreeNAS for some time, I think you may have overlooked or never looked into certain aspects as you were using a micro server. You should review some of the basics here:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

FreeNAS® Quick Hardware Guide
https://forums.freenas.org/index.php?resources/freenas®-quick-hardware-guide.7/

Hardware Recommendations Guide Rev 1e) 2017-05-06
https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

Then, you might want to review this build using ESXi to virtualize FreeNAS:

Build Report: Node 304 + X10SDV-TLN4F [ESXi/FreeNAS AIO]
https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

And this information about SLOG and how it accelerates performance of sync writes:

Testing the benefits of SLOG
https://forums.freenas.org/index.php?threads/testing-the-benefits-of-slog-using-a-ram-disk.56561

The ZFS ZIL and SLOG Demystified
http://www.freenas.org/blog/zfs-zil-and-slog-demystified/

 

[Mannekino]

Well, that was a bit harsh but I understand your sentiment. Unfortunatly I don't have the budget to get a server grade board with ECC memory and a different processor. I think my current motherboard, CPU and memory should do just fine for a home NAS. I feel like this is a pretty decent motherboard with a good chipset (H87) and a 35W TDP Core i7 should provide very nice performance.

 

[Chris Moore]

Well, that was a bit harsh but I understand your sentiment. Unfortunatly I don't have the budget to get a server grade board with ECC memory and a different processor. I think my current motherboard, CPU and memory should do just fine for a home NAS. I feel like this is a pretty decent motherboard with a good chipset (H87) and a 35W TDP Core i7 should provide very nice performance.

It might be good enough if all you want is to use it as a NAS, but you are asking about virtualization. I guess you need to be a bit more specific about what you are thinking to virtualize but a 2.2GHz processor is not a good candidate for much of anything. You have shot for low power without regard to capability.

 

[Mannekino]

Hi Chris,

Thank you for your input so far. Lets simplify a bit and forgot about ESXi for now. I'm not attached to that setup it was just an idea I had. I would be totally fine with running this as just a FreeNAS server. With regard to virtualization: I would be primarily using it for learning and testing. So running Debian/Ubuntu virtual machines and I want to learn Docker also for development purposes and testing out system administration software (think about monitoring tools). If this can be done within FreeNAS VM functionality I would be happy to give that a try.

Right now I'm considering two options:

1. Working with what I have and a small upgrade (PCIe SSD)

As stated in my original post I have one PCIe slot avaiable and I believe it would be a good idea to somehow utilize that. The best way I can think of is use a NVMe drive with a PCIe adapter and install FreeNAS on that drive. I haven't done the reading yet but I'm thinking I might be able to use the same drive for L2ARC and SLOG also. I'm going to read the links you provided this week and I already started on the "The ZFS ZIL and SLOG Demystified" article. Would something like this PCIe to M.2 NVMe adapter combined with a small sized (for example 32 or 64 GB) NVMe SSD work with FreeNAS as a boot drive (and maybe L2ARC plus SLOG)?

2. Possibly getting new hardware (motherboard, CPU, memory and PSU)

I understand that getting components from the recommend hardware list is preferred. I am trying to make the best of what I have and to be honest I don't think it is as bad as you make it out to be, but maybe I'm wrong. One option I'm considering is building a computer out of the parts I listed above (motherboard, CPU, memory and PSU). This weekend I will get a new chassis for my current home PC and a new PSU (that were I'm also getting the Seasonic PSU from). I could use that chassis and PSU combined with the parts listed above to build a new PC and try to sell it (I also have a AMD R9 380 laying around which could be used for this). The proceeds of that sale can be put towards new parts for my NAS.

Thanks.

 

[Chris Moore]

I would be totally fine with running this as just a FreeNAS server. With regard to virtualization: I would be primarily using it for learning and testing. So running Debian/Ubuntu virtual machines and I want to learn Docker also for development purposes and testing out system administration software (think about monitoring tools). If this can be done within FreeNAS VM functionality I would be happy to give that a try.

Sure, it can be done, but I think that CPU is way under-powered for it. You might still get some of what you want to do done, but you may find that you need more memory also. You will also need to set the system tunables to restrict how much memory FreeNAS uses for ARC so you can have memory to dedicate to the VMs.

As stated in my original post I have one PCIe slot avaiable and I believe it would be a good idea to somehow utilize that. The best way I can think of is use a NVMe drive with a PCIe adapter and install FreeNAS

How would installing FreeNAS on a NVMe drive help at all? That is one of the reasons I say you need to go back to square one and learn again how FreeNAS works. Having a fast boot drive gives you nothing in FreeNAS.

I haven't done the reading yet but I'm thinking I might be able to use the same drive for L2ARC and SLOG also.

You need to do some reading because the boot drive can't be used for anything, NOTHING other than booting the system and storing log files. You would need a physically separate device for each L2ARC and SLOG and one of the reasons I said you hardware was totally unsuitable is the fact that it doesn't have enough connectivity to connect all the devices you need.

I understand that getting components from the recommend hardware list is preferred.

There is a reason for that, because it would actually work.

I am trying to make the best of what I have and to be honest I don't think it is as bad as you make it out to be, but maybe I'm wrong.

Give it a try. The worst that happens, it has really poor performance or you loose all your data.

 

[Mannekino]

Give it a try. The worst that happens, it has really poor performance or you loose all your data.

You got me worried there for a minute, I thought maybe it would catch fire and burn my house down.

But jokes aside, I'm getting the message you think I'm a fool for even trying this. I've been running FreeNAS for about 4-5 years now on my HP MicroServer N40L with 8 GB memory and 3 x 2 TB storage. I'm still on the same USB key. I did do a reinstall twice I believe and imported my ZFS storage pool without issues. I don't think I'm wrong in thinking my new build would be a major upgrade from my MicroServer.

I will read all the links you provided before I make my decision on what to do.

 

[RodyMcAmp]

On an alternative note. This really should work pretty well, your vm’s may not be blazing fast but esxi rocks. I have set up home servers for testing with less power than this. I would start with making sure you could pass through your drives to the freenas vm. (Hint, make sure you reserve the ram for freenas. it won’t work as dynamic.)


The things I am going to recommend are not recommended as Chris pointed out. You should have server hardware, however Freenas will work just fine without it.

I think you are going to need at least 1 data store drive so you might need some kind of hba to pass though. So assuming you want to use the nvme drive that would work.

1. Install esxi on an usb drive. (or you could install it on your nvme drive and still use the rest of the space for your data store. )
2. Install freenas on your data store reserve all the ram you you want. 8 to 10 gb should work. Make sure you can pass your sata controller through to freenas.
3. This will leave 6-8 gb ram for vms, if your nvme drive is large enough you can install your vms on that fast storage.
4. If your nvme drive is to small you could setup nfs data stores on freenas and have esxi use that storage for VM storage.

This will be plenty fast for a few vm's. freenas only needs 2 cores for basic file sharing, a bit more if you want to run plugins.

You could easily run a windows vm with 4 gb ram or several small vms for experimentation.

 

[RodyMcAmp]

https://b3n.org/freenas-9-3-on-vmware-esxi-6-0-guide/
this guide is out of date but worth a read most of the steps still apply on 11.2

 

[Mannekino]

Hi RodyMcAmp,

Thank you for your reply. I just finished up reading the presentation linked in this thread (Slideshow explaining VDev, zpool, ZIL and L2ARC for noobs!) it was very useful and a nice refresh course with a great deal of new information for me.

Some conclusions I came to after reading it, hopefully I am not wrong:

1. Using my PCIe slot for an NVMe SSD for L2ARC or SLOG is not useful at all for me.
2. L2ARC on an NVMe SSD won't give me any performance increase, it actually could cause a decrease in performance.
3. SLOG on a NVMe SSD won't do anything for me since I won't be using NFS or use my FreeNAS as network storage for VMs. SLOG will actually introduce way more risk without a BBU (I realize now how ridiculous it was to consider using this).
4. Using the NVMe SSD could be very interesting for FreeNAS Jails and VMs.

So I decided to test this scenario first:

1. Buy a high quality USB 3.1 (Gen 1) drive for FreeNAS to run on.
2. Buy a PCIe to M.2 NVMe SSD adapter and something like a a 256 GB NVMe SSD.
3. Install FreeNAS to the USB drive.
4. Create one VDev plus zpool with the NVMe SSD as a pyshical drive and one VDev plus zpool with my four WD REDs as physical drives.
5. Use the SSD-based zpool for my Jails and FreeNAS VMs.
6. Use the SATA-based zpool for my data.
7. Test if the FreeNAS VM functionality works well for my needs (I want to be able to run Windows Server VMs and Debian/Ubuntu VMs).
8. If this doesn't work properly I'm going to give the ESXi setup you suggested a try.

Sounds reasonable right?

Also I want to note if my on-board NIC doesn't work properly I will definitely choose to sell these parts and save up a bit more and buy parts from the recommended hardware list. But I want to test everything first.

 

[Chris Moore]

Buy a high quality USB 3.1 (Gen 1) drive for FreeNAS to run on.

It would probably be better to use a USB 2.0 (still quality) but no real advantage to getting the newer/faster tech since it is just the boot drive and it might need newer drivers that FreeNAS doesn't have, so it could create problems.

Sounds reasonable right?

You might find that you don't have enough memory to run many VMs for Windows, you need a couple GB of RAM for each VM, Linux could probably run on one. Just depends on how many you want running at once and how responsive you want them to be.

 

[Mannekino]

It would probably be better to use a USB 2.0 (still quality) but no real advantage to getting the newer/faster tech since it is just the boot drive and it might need newer drivers that FreeNAS doesn't have, so it could create problems.

Thanks I'll keep that in mind.

As for the VMs, I will be using it mostly for educational purposes, for example:

• Testing open source software / utilities like network and hardware monitoring tools. I want to catch up on my outdated knowledge. Basically if I see something that peaks my interest I want to fire up a VM and test it.
• Using VMs for learning to program again. I might want to use Docker for this. Learning Docker is something I want to do also. I have an idea for an website with an phone app I want to develop but I need to learn a lot and I want to use VMs for the development.
• I want to brush up on my Windows Server knowledge also. I recently got a new job managing certain applications that mostly run on Windows Server. Even though I'm not actually doing any system administration work I would like to expand my knowledge so I can become better at my job and be able to bridge the gap between my role and my colleagues at IT.
• At the most I will run 2 or 3 VMs at the same time whenever I need them and not run them 24/7. As I mentioned it will mostly be used for educational purposes I'm not planning on running any services on them for myself. The only thing I'm considering running in a VM is some kind of software for two security camera's I want to install in the future (think of something like Synology has with their Surveillance Station). But maybe a Jail would be better for this, not sure.

 

[RodyMcAmp]

the Jails on freenas work a treat and i realy like them. the VM implimantation on the othe hand seems to be pretty poor at present. Good luck and have fun.... and back up anything important

 

[kdragon75]

Point blank if you using VMs in freenas of freenas on esxi, you absolutely need more RAM. For virtualized FreeNAS 24GB is an absolute minimum and you need to reserve at least 8 for FN. ESXi will consume just over one and each VM has overhead. You also cant configure freenas with more than 2 maybe 3 cores without tanking performance no mater how busy your other VMs unless there all single core.

 

[Mannekino]

I understand, well what I can give a shot is build a PC out of the parts I have now and try to sell it. It should be a really nice PC for home/office usage. If I can get the amount for it I want I will sell it and get new parts. If not, I will go ahead and just use these parts for a bare metal FreeNAS server and maybe do an upgrade in the future.

Edit:

I've been looking a bit at possible motherboard candidates for my usage. I came up with the following specification list:

1. Mini-ITX
2. Up to 6 SATA ports
3. Up to 32 GB of memory (this leaves out LGA 1150 boards with 2 memory slots)
4. Virtualization support

This leaves me with only one motherboard (I checked both SuperMicro and ASRock Rack). Supermicro doesn't seem to have any boards with these specification but ASRock does have one and it's based on the C236 chipset.

This one: http://www.asrockrack.com/general/productdetail.asp?Model=E3C236D2I#Specifications

That should be a very nice motherboard right?

 

[kdragon75]

Supermicro has several board that support tons of ram in small form factors with 6+sata ports.

 

[Chris Moore]

I've been looking a bit at possible motherboard candidates for my usage. I came up with the following specification list:

1. Mini-ITX

What is the deal with so many people wanting those tiny system boards. I wish nobody ever came up the the idea for those. They are crippled out of the box with no room for adding the hardware you need to make a full system.

 

[Mannekino]

Well Chris, you managed to change my mind on the hardware. I was looking at a second hand Fractal Design Node 804 case and someone was selling one but he also had a fairly complete set for sale with the parts listed below. I remembered the motherboard from the recommended hardware topic. This should be a great build for FreeNAS.

• Motherboard: Supermicro X9SCM-F
• CPU: Intel Xeon E3-1260L
• CPU cooler: Noctua NH-L9i
• Memory: 32GB ECC (4 x 8GB Kingstong ValueRAM KVR1333D3E9SK2/16G)
• Controller: Dell PERC H200 (flashed in IT mode)

I saw that this motherboard is on the FreeNAS favorite list. With this CPU and the additional memory I should be able to do some virtualization also. I’m starting off with 4x 4TB WD Red drives which I did buy new and I have a Seasonic Focus Plus Gold 550W PSU which should do fine.

Now I need to make a nice PC of the parts I initially bought and try to sell it, should make a great home/office PC.