Hardware recommendations for my virtualized mini-ITX FREE-NAS System

[SebbaG]

Dear folks,

I'm new to this place but I've read a lot and a lot before posting. As you can read from the title, I'm planing to run FreeNas in a virtual environment and before you ask, YES I've read both topics which address this in detail, YES I've read it very carefully and YES I'm fully aware of the risks it comes along with. Nevertheless I've decided myself to take that approach for several reasons!

I want to build a box that is able to host...
- FreeNas VM (HDD via PCI Passthrough and fs=ZFS in RAIDZ2)
- Ubuntu or CentOS VM (with DNS, DHCP, OpenLDAP, and eventually serve as a VPN-Gateway)
- Ubuntu VM with (Apache + XBMC)

At the hypervisor level I wanted to test a new approach and use KVM and OpenBSD with pf as a basic Firewall (one or two VM might be later exposed in a DMZ). If it won't run properly I would go back to ESXi after I tried XEN.

Right now I'm at the point of selecting hardware components for that system and therefore I'm asking you guys for help, opinions, recommendations & hints!!!
I was planning to use the following hardware:

Case:

• Fractal Design Gehäuse Node 304

Motherboard:

• ASRock E3C226D2I 2xGL/SATA600/USB3.0/VGA/Seriell Mini-ITX Sockel 1150

(Some people stated that this board supports PCI Passthrough, but I didn't any further information at the manufacturer)

CPU:

• Intel® Xeon® Processor E3-1220L v2 or Intel® Xeon® Processor E3-1240L v3

(it should support NI-AES, PCI-Passthrough, ECC Memory, VT-x, VT-d, have enough Power + at the same time low TDP)

Memory:

• 16GB Kingston ValueRAM DDR3-1600 regECC DIMM CL11 Single (reg or without reg?)

Fan:

• EKL Alpenföhn Sella Tower Kühler (looked nice and has 4 pins)

PSU:

• 350 Watt be quiet! System Power 7 Bulk Non-Modular 80+ Bronze (here I'm not sure what to take)

Harddrives:

• 6x 3TB Western Digital Red (5 for RAIDZ2)

(at this point I also dont know if I should use 1 SSD instead for the guest VM's and the host system or a 1 HDD instead)

By the way, yes I want to build a Mini-ITX system and eventually encrypt the drives. Budget is around 1000-€!

Thank you very much in advance for your opinions and help...
Greets Sebastian

 

[Ericloewe]

Those specs are decent for a dedicated FreeNAS server (apart from the crummy PSU - Seasonic is the way to go, Corsair is the cheaper alternative in this space). They are totally inadequate for what you propose (which is a bad idea anyway). 16GB of RAM for all that? Yeah, that's not nearly enough.

In addition to that, you're wasting money on a low-power processor. That CPU will never use less power at idle than its non-L counterpart, but it will be severely performance restricted.

 

[SebbaG]

First of all thx a lot for the reply.

About the problem with RAM, I know that's critical, but I see no alternative, since it is the only mini-ITX board with 6Sata, ECC support and PCI passthrough that I'm aware of.

Also I have to tell that it will be no enterprise system. I want to use it at home with my flatmates and friends with approx. < 20 users.
As for the RAM for the VM's I think 2GB per VM or less should be enough for a home environment. (can't imagine why I might need more) so there are about 12Gb left for FreeNas.
Yes this will mean performance issues but that's a tradeoff.

But as for the CPU, do you have any good suggestions? As said I do not have many experience so far, so I appreciate everything comment and critic.

The backup solution if the performance will be too low is to install freeNas on the bare medal.

By the way, what about the SSD vs. HDD?
thx

 

[sremick]

Everything I've read clearly states that FreeNAS should only ever be run in a VM to do basic testing/experimenting with, and is not designed to be run in a VM for real use, such use is not supported, performance will be poor, and there will be issues.

Use of Kingston RAM is also advised against.

In general, that's an underpowered system for what you intend. If you really want to push a server that hard you need to move beyond Mini-ITX to hardware that will support more RAM. And to a better CPU.

I am using the same motherboard, case and HDDs in a dedicated FreeNAS system I'm building, but with a faster Xeon and Crucial RAM.

 

[SebbaG]

thx again for the opinions... here is my new suggested setup:

Case: Fractal Design Gehäuse Node 304 schwarz USB3.0 mITX/DTX
MoBo: ASRock E3C226D2I 2xGL/SATA600/USB3.0/VGA/Seriell Mini-ITX Sockel 1150
CPU: Intel Xeon E3-1240V3 4x3.4GHz 8MB Turbo/HT (Haswell) Sockel 1150 BOX
RAM: 2 x Crucial DIMM-Speicher DIMM 8 GB ECC DDR3-1600 (unbuffered)
PSU: 400 Watt Seasonic Platinum Series Fanless Modula 80+ Platin
SSD: 256GB Crucial MX100 2.5" (6.4cm) SATA 6Gb/s MLC (CT256MX100SSD1)
HDD: 5 * 4000GB WD Red WD40EFRX 64MB 3.5" (8.9cm) SATA 6Gb/s
FAN: EKL Alpenföhn Sella Tower Cooler

What do you think about this setup?
(pls don't argue about the nonsense of the virtualized FreeNAS, since I'm aware of it)

 

[cyberjock]

/me is gonna laugh his ass off when we find out ASRock's board doesn't do proper VT-d and he can't do what he wants after he's paid for the hardware

 

[SebbaG]

:( well, this does not really help me with my project!
But if it won't work, I can still do a native installation no? So I don't see a big risk in my approach except that my CPU might be slightly overpowered then!

 

[joeschmuck]

I think you need to figure out if you want functionality or size of the product to have the priority. I'm not sure why you would need a mini-ITX MB, why not something in the ATX size? Is that going to make that big of a difference?

Once you decide that performance of the system is more important, and because you're spending all this money on something that you want to do at least what you expect, then you will be purchasing enough RAM and a proper CPU and MB.

And my comments have nothing to do with if I think you should or shouldn't run FreeNAS on a VM, that risk is understood as you have said. But if you purchase a system like above, well once you are done messing around with the VM stuff, you could just set it up as a standard FreeNAS system and be happy.

 

[cyberjock]

:( well, this does not really help me with my project!
But if it won't work, I can still do a native installation no? So I don't see a big risk in my approach except that my CPU might be slightly overpowered then!

Yes, the board can still work on bare metal and should be just fine.

Now I hope you understand why we don't recommend virtualizing. People that don't do it for a living often don't have a clue what they are getting themselves into until the money is spent. At that point it's too late to go back.

 

[mjws00]

Heh. I won't be quite as pessimistic. Been meaning to stick it's bigger brother ASRock E3C224D4I-14S in a Silverstone ds380 for a while, and also utilize its vt-d. Only thing stopping me has been the 32GB RAM limit that pisses me off to no end. You'll be even worse off with only 16GB max. Then I pinch myself and ask "WHY SO SMALL?" No small M-ATX cases that compare yet... hopefully soon.

With a little hunting you can find examples of the Asrock m-itx boards doing vt-d successfully. It was intended as a small form factor whitebox. But I doubt you can do anything with the onboard controllers. A M1015 would likely work perfectly. Or the onboard 2308 on it's bigger brother... as shown here: http://lime-technology.com/forum/index.php?topic=34624.0

If I had it in hand, I'd vouch but I don't. If you exclude virtualizing FN. You have a nice esxi whitebox that can't scale due to form factor. I'd look at the Supermicro x10sl7-f and pass through the 2308. Give you a bit more room to breathe. Very proven platform.

Your money. I do think you'll find the lack of RAM and flexibility limiting. Bad enough for Baremetal FN. Much worse for esxi. I won't cry for your data. But you have to know a thread with virtualizing in it is gonna get pounded. ;)

 

[Mlovelace]

You might look at an ASRock C2750D4I; 8 cores and you can install up to 64gb ecc ram. It supports VT-x so you'd have to pass through a HBA instead of using MB SATA ports (better way to go anyway). The LSI 9211-8i is good or the IBM serveRAID M1015 in IT mode is good too.

http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Specifications

 

[cyberjock]

You might look at an ASRock C2750D4I; 8 cores and you can install up to 64gb ecc ram. It supports VT-x so you'd have to pass through a HBA instead of using MB SATA ports (better way to go anyway). The LSI 9211-8i is good or the IBM serveRAID M1015 in IT mode is good too.

http://www.asrockrack.com/general/productdetail.asp?Model=C2750D4I#Specifications

VT-x and VT-d are NOT the same technology. ;)

You can't passthrough hardware with VT-x. :p

AFAIK Avoton doesn't do VT-d, which is the technology you need.

That's 2 people that have proven that they shouldn't run FreeNAS in a VM in this thread so far. :P

 

[leenux_tux]

SebbaG,

I'm not going to try to disuade you from virtualizing FN as you seem pretty much set on it looking at the previous messages, however, I'm trying to understand what the driver is for you wanting to Virtualize FN ? Hardware costs ? Running costs ? Space for the hardware ? And why FreeNAS ? You have described what you want to do with your Ubuntu/Centos Systems but not the FreeNAS system ?

On the Virtualization side, take a look at PROXMOX. I'm not saying it's better or worse than other virtualization options, however, it's one you have not mentioned. PROXMOX uses KVM (which you have mentioned in your original message) and "containers". It's "Open Source" so is free to download and use.

 

[SebbaG]

@Mlovelace thx for your ideas, but as cyberjock already pointed out, I need support for VT-d

@leenux_tux
the driving motor was flexibility and at the same time lack of space.
-> That means I do not want to have 2 separate boxes for running those VM's. I know that exposes me to additional risks, but since it's no productive environment I like playing around with new challenging ideas.
But at one point you got me: Why FreeNas? Well actually I have no good answer to that. I thought it will give me easy and nice handle of ZFS pools + full feature set and flexibility to try out different functions in the future + CIFS and NFS. But you are right, I could also manage this on my own or use different NAS-OS'es.
As for PROXMOX, I never heard about it, but I'll have a look into it thx.

Last but not least... as from the flexibility perspective I have to agree with you guys. Putting in a motherboard that restricts me at 16 GB of RAM in such a setup is not really a good idea. But because of space, I really would like to stick with mini-ITX and that case. But to raise two new questions:
- Do you have a good uATX + slightly bigger cases in mind?
- Has anyone heard that there is a mini-ITX board with 32GB ECC support and full feature set of the E3C226D2I out there or in roadmaps?'

thx for all your help and criticism!

 

[Ericloewe]

@Mlovelace thx for your ideas, but as cyberjock already pointed out, I need support for VT-d

@leenux_tux
the driving motor was flexibility and at the same time lack of space.
-> That means I do not want to have 2 separate boxes for running those VM's. I know that exposes me to additional risks, but since it's no productive environment I like playing around with new challenging ideas.
But at one point you got me: Why FreeNas? Well actually I have no good answer to that. I thought it will give me easy and nice handle of ZFS pools + full feature set and flexibility to try out different functions in the future + CIFS and NFS. But you are right, I could also manage this on my own or use different NAS-OS'es.
As for PROXMOX, I never heard about it, but I'll have a look into it thx.

Last but not least... as from the flexibility perspective I have to agree with you guys. Putting in a motherboard that restricts me at 16 GB of RAM in such a setup is not really a good idea. But because of space, I really would like to stick with mini-ITX and that case. But to raise two new questions:
- Do you have a good uATX + slightly bigger cases in mind?
- Has anyone heard that there is a mini-ITX board with 32GB ECC support and full feature set of the E3C226D2I out there or in roadmaps?'

thx for all your help and criticism!

I believe there's an extended miniITX (which requires a chassis that has extra room below the motherboard, beyond the bare minimum for miniITX) board similar to the E3C226D2I that does include four DIMM slots.

 

[SebbaG]

@Ericloewe I think you refer to this one: E3C224D4I-14S

 

[Ericloewe]

@Ericloewe I think you refer to this one: E3C224D4I-14S

Ah, it's even better than I remembered. Onboard LSI SAS 2308, very nice.

 

[Mlovelace]

VT-x and VT-d are NOT the same technology. ;)

You can't passthrough hardware with VT-x. :p

AFAIK Avoton doesn't do VT-d, which is the technology you need.

That's 2 people that have proven that they shouldn't run FreeNAS in a VM in this thread so far. :p

Derp, I thought it had directed I/O too. Sorry for the bad suggestion SebbaG.

 

[sremick]

Ah, it's even better than I remembered. Onboard LSI SAS 2308, very nice.

Yeah I was all excited for this board as I wanted the option to go to 32GB later in the future if necessary. But I couldn't find any decent case that would fit it. Since I've got a Node 304, I'm settling for a E3C226D2I and hoping 16GB will be enough for me far into the future (3 users).

 

[leenux_tux]

On the "space" question. Have you seen this web page at all ? "https://wiki.eth0.nl/index.php/LackRack". This page has had a mention on here previously. These folks are dong some crazy stuff with some Ikea furniture. Perhaps you could double up ? Replace a table somewhere in the house with a "Lack" table, but have some computer equipment hanging underneath it !!

Or, if you REALLY want to push the boundries take a look at this "http://www.formfactors.org/developer/specs/matxchk1_1.pdf". This PDF doc has information needed to assist building your own custom microATX case. It has information on (for example) IO sheild dimensions, Motherboard Mounting placements, information/checklists for height restrictions, power supply mounting features etc etc.

Now that would be a project.