Will this FreeNAS?

[tabzoid]

I'm on the verge of setting up my first FreeNAS, and for a number of reasons I've decided to run it as a VM on a linux server that I am about to build (using KVM).
I'd like to experiment and play around on this FreeNAS VM setup, and if it looks good over the next months, I might then move it into a standalone physical system.

I had a question about the hardware for the linux server on which I will run FreeNAS in a VM:
- As I need the CPUs to have PCI passthrough (VT-d), ECC RAM support, and multiple cores (for this and other VMs), I can see no option than going for a Xeon setup.
- Ideally I'd like to set up an E5-2670/2660 (single or dual CPU setup). But while I can find the CPU chips rather cheap, the motherboards cost a bomb. And just these 2 end up costing $500+.
- My cheaper alternative is a Dual Xeon X5670 system with a Supermicro X8DTI / X8DTI-F / X8DT6-F / X8DTE / X8DT3-LN4F motherboard. This would totally come to about $250.

Do you see the dual X5670 as a feasible option to run FreeNAS on as a VM?

[The rest of the system will likely include 32 GB ECC RAM, a SSD for the OS, 2x2TB HDD for the FreeNAS data]

 

[Chris Moore]

I don't know if anyone has done it in KVM before, but I know it has been done in VMWare (ESXi) so there may be some tips on the forum. If you search for virtualized, I know you will find a few listings.
Then there is this one form @Stux : https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

 

[danb35]

I don't know if anyone has done it in KVM before,

I'm pretty sure @jgreco has posted that KVM isn't a very good hypervisor for FreeNAS, but I don't recall seeing him post details. I'll use it for testing (I run two Proxmox VE machines, which use KVM for the underlying hypervisor), but my production FreeNAS is bare metal.

 

[tabzoid]

I don't know if anyone has done it in KVM before, but I know it has been done in VMWare (ESXi) so there may be some tips on the forum. If you search for virtualized, I know you will find a few listings.
Then there is this one form @Stux : https://forums.freenas.org/index.ph...node-304-x10sdv-tln4f-esxi-freenas-aio.57116/

Thanks for the link. I've read a few places where people have done it on KVM, and that gave me the confidence to go that route.
Once I have it running, I'll run diagnostics and stuff for a while to see if the performance is good enough. Nonetheless, this is more of a dev setup, and if I see that we are avidly using the FreeNAS setup at home, then I will likely port it to a bare metal production setup down the road.

 

[tabzoid]

So I'm still trying to figure out the best CPU / motherboard combination for my server.

Most signs point towards a dual X5670 setup (any opinions if that is worthwhile in this day and age)?
But I'm still toying with a dual E5-2670/2660 setup if I can find the motherboards for cheap.

 

[Chris Moore]

(any opinions if that is worthwhile in this day and age)?
But I'm still toying with a dual E5-2670/2660 setup if I can find the motherboards for cheap.

If I were in your place, looking to buy hardware, I would just buy the right hardware for FreeNAS out the gate, build a bare metal system and be done with it.

If you are going to buy something, this system would give you a great chassis and system board, you just add memory, a SAS HBA and hard drives. If you want faster CPUs, you can upgrade but these might even bee good enough for your virtualized testing. This would certainly be a great bare metal system for FreeNAS. How much storage are you looking to have in the end?

The more modern CPUs will not only use less power for more performance, but also have better support for hardware passthrough.
If you really want to have any sort of realistic test of FreeNAS, the way to test it is on bare metal. If you give us some idea what you want to do with it, there is sure to be someone on the board that has the first-hand experience to tell you what the best configuration is. Many of the regular users here use FreeNAS at work, in production, and at home and on a wide variety of hardware.

Do you see the dual X5670 as a feasible option to run FreeNAS on as a VM?

There are people using those older CPUs with FreeNAS in bare metal installs, but most of us are looking at getting away from that older generation hardware because of the heat and power consumption compared to newer hardware.

 

[tabzoid]

If I were in your place, looking to buy hardware, I would just buy the right hardware for FreeNAS out the gate, build a bare metal system and be done with it.

If you are going to buy something, this system would give you a great chassis and system board, you just add memory, a SAS HBA and hard drives. If you want faster CPUs, you can upgrade but these might even bee good enough for your virtualized testing. This would certainly be a great bare metal system for FreeNAS. How much storage are you looking to have in the end?

The more modern CPUs will not only use less power for more performance, but also have better support for hardware passthrough.
If you really want to have any sort of realistic test of FreeNAS, the way to test it is on bare metal. If you give us some idea what you want to do with it, there is sure to be someone on the board that has the first-hand experience to tell you what the best configuration is. Many of the regular users here use FreeNAS at work, in production, and at home and on a wide variety of hardware.

There are people using those older CPUs with FreeNAS in bare metal installs, but most of us are looking at getting away from that older generation hardware because of the heat and power consumption compared to newer hardware.

Thanks @Chris Moore !
I'll indeed consider in more detail to do the FreeNAS as a bare metal system, though I am still leaning towards the VM based setup. But I've needed to get the hardware done asap (I'm living in India where it's all much more expensive; so I'm having a friend in the US bring a lot of things over to me. He leaves this week, and so I had to choose things which would reach him on time. Which also reduces many options, as I can't then buy bulky stuff or stuff which will take longer to ship).

So I've been focussing on the hardware - I've taken your advice to move away from the much older stuff, and move to slightly less older stuff :).
I also decided that a Dual CPU system would be overkill for my needs. So, I'm going with a system as below (these parts are mostly in the pipeline):

• CPU - Intel Xeon E5-2670 / 2660 (this still needs to be ordered)
• Motherboard - Supermicro X9SRL-F-O
• RAM - Samsung 16GB (x2) DDR3 1600 ECC Registered
• SSD Drives - Samsung 850 EVO - 120GB (x2)
• CPU Cooling - Noctua NH-U12DXi4

This mainly leaves my storage drives, power supply, case, and possibly a HBA for PCI passthrough.
Hopefully I can still order the hard drives tomorrow - these have a much shorter shipping time. It's not been easy figuring out what I need here - I dont have so much data and so am thinking of going initially with 2x2TB (for freeNAS) + 1x4TB (for everything else).

 

[Inxsible]

initially with 2x2TB (for freeNAS) + 1x4TB (for everything else).

Will all 3 drives be in the same pool? What configuration are you planning for your pool?

 

[tabzoid]

Will all 3 drives be in the same pool? What configuration are you planning for your pool?

I've been mulling over storage needs and setup a lot over the last day; I think it better first to define the storage setup I am thinking about, and from there gauge what is the best kind of storage I need, and how many instances. I'll try to explain this below, and appreciate your insights (I'm a newbie to this, and might get terms and concepts wrong. Bear with me.)

I am thinking of 3 separate pools at the moment:

1. A Server OS Pool: This will hold the server image (and possibly also the different VM images and other core files). I am considering to do this using a mirrored pair of 120 GB Samsung 850 EVO solid-state drives (probably partitioned for main server vs. VM images).
2. A FreeNAS Pool: This will be the dedicated storage given to FreeNAS VM (via PCI passthrough) - I intend to get a HBA, and connect the drives to FreeNAS via this. I've been reading up bits and pieces on how to configure this (RAIDZ, RAIDZ2, mirrored vdevs,...), and have little idea at the moment as to how this this setup will look. At the moment I do not much much data to store on FreeNAS, but I see that growing over time. So, I'd like a setup where I can add more storage easily down the road. Towards this, I wonder if I should get a HBA with external SAS ports rather than internal ones, and setup this HDD pool on some external hot-swappable/pluggable storage (but I think this might be overkill at the moment). [Update1: Eventually, if I move the FreeNAS into a standalone bare metal setup, then it would be easy to have the external storage move with it.]
3. A Server Pool: This would be the storage for everything else running on my Linux server. Initially this would be a single storage setup, but as I might have specific apps/VMs moving away from development/playground stages to production stages, then I would likely need to create separate storage setups for these. I'm thinking of initially storing the HDDs for this inside my main case, and set this up as a RAID array (RAID5, RAID6, or RAID10 perhaps, but I need to do much more analysis here too). [Update1: Or can I perhaps store all this within the FreeNAS storage, with the caveat that I must ensure that FreeNAS is up and running before these are? If it's even possible for these apps to access the FreeNAS data]

Does this make some sense?
If the setup does, then I can venture into what HDDs I would need to buy for each.

 

[Inxsible]

I would just create 1 OS pool and 1 FreeNAS pool. Your Linux VMs can use the storage from FreeNAS via NFS. Going with SSDs for the OS pool makes sense.

I intend to get a HBA, and connect the drives to FreeNAS via this.

Why? The X9 series board that you are choosing has enough SATA ports to connect both your FreeNAS drives, both your SSDs and the 1x4TB that you intend to add (still not clear for what purpose)

 

[tabzoid]

I would just create 1 OS pool and 1 FreeNAS pool. Your Linux VMs can use the storage from FreeNAS via NFS. Going with SSDs for the OS pool makes sense.

Why? The X9 series board that you are choosing has enough SATA ports to connect both your FreeNAS drives, both your SSDs and the 1x4TB that you intend to add (still not clear for what purpose)

From what I have read, many people advise against running FreeNAS in a VM (unless one takes many precautions and such). One of these best practices is to give this VM direct access to storage via PCI Passthrough (it seems that ZFS works much better with this). e.g. via this link

Also, about 2 vs. 3 pools - I think initially I will just setup the Linux Server and so there will be 2 pools (OS vs. Server pool) - no freeNAS at this time. A little while later, once the server is up and running fine and I can do some basic stuff on it, I will likely start to setup FreeNAS (in a VM). I will then likely need to create a new FreeNAS pool (using HBA + new HDDs connected to this, either internal or external).

 

[Inxsible]

From what I have read, many people advise against running FreeNAS in a VM

I think you read wrong. Many users run FreeNAS in a VM.

(unless one takes many precautions and such).

That is true for anything.

I will then likely need to create a new FreeNAS pool (using HBA + new HDDs connected to this, either internal or external).

Do you mean external HDDs? That is not recommended for FreeNAS because it involves separate power sources introducing multiple points of failure etc. I would suggest you go with internal HDDs.

 

[danb35]

I think you read wrong. Many users run FreeNAS in a VM.

No, he read perfectly. Running FreeNAS in a VM is discouraged. See, e.g., https://forums.freenas.org/index.ph...nas-in-production-as-a-virtual-machine.12484/

 

[tabzoid]

I think you read wrong. Many users run FreeNAS in a VM.

That is true for anything.

Do you mean external HDDs? That is not recommended for FreeNAS because it involves separate power sources introducing multiple points of failure etc. I would suggest you go with internal HDDs.

Indeed, I think it safer and easier to run everything on internal HDDs.
If I ever need to scale up any part of my system later, I can then see what is the best possibility.

So, the config I am thinking of now is:

1. OS Pool: 2x 120 GB Samsung EV0 850 SSDs
2. Server Pool: 2x4TB NAS grade HDD
3. FreeNAS Pool (which will come some weeks/months after the upper stuff is setup): 2x4TB NAS grade HDD via HBA (same brand/type as from the server pool). I can then also add the 2 server pool HDDs to the NAS pool, and move the server data within there into the FreeNAS pool.

I'll just need to make sure I get a case big enough to hold at least 2 SSDs and 4 HDDs (preferably 6-8 HDDs in case I need to add more later).

 

[Inxsible]

No, he read perfectly. Running FreeNAS in a VM is discouraged. See, e.g., https://forums.freenas.org/index.ph...nas-in-production-as-a-virtual-machine.12484/

Interesting. Yet so many users use ESXi as their hypervisor with FreeNAS installed as a VM.

I use FreeNAS on bare metal, but that's only because I don't have much exposure with VMs, so I try to keep it as simple as I can.