BUILD compact Xeon based FreeNAS build (with virtualization)

[the_tux]

Hi together,
I was using a home server (ESXi) and FreeNAS for quite a long time but since I moved and didn't take everything with me it's time for something new.

I always had 2 separate boxes for the ESXi and the NAS and I was always thinking of a way to put those together. I know you don't like the idea of virtualizing FreeNAS but let's talk about the options. I also want to have the setup as future proof and flexible as possible.

This is what I've selected to the moment:
4 x Western Digital WD Red 4TB, 3.5", SATA 6Gb/s (WD40EFRX) zRaid1
1 x Samsung SSD PM951 256GB, M.2 (MZVLV256HCHP-00000) (as VM datastore)
1 x Samsung DIMM 32GB, DDR4-2133, CL15, reg ECC (M393A4K40BB0-CPB) (for the moment, expand it later)
1 x Supermicro X10SDV-4C-TLN4F retail (MBD-X10SDV-4C-TLN4F-O)
1 x SilverStone DS380, mini-ITX (SST-DS380B)
1 x SanDisk Cruzer Fit 16GB, USB 2.0 (SDCZ33-016G-B35) (boot device)
1 x Corsair SF450 450W SFX12V (CP-9020104-EU)

+ a used LSI 9211-8i (for passing to FreeNAS)

Only 4 drives for the beginning, I want to add a 2nd storage pool in the future.

I'm planning to run a few (lightweight) VMs and sometimes 2 or 3 little nested ESXi servers for testing purpose. Or should I just forget the idea and use the VirtualBox plugin for now (and bhyve in the future, does bhyve support nested VT?)

Questions so far:
- I'm unsure about the M.2 interface on the Supermicro board. As far as I can see I can only use PCIe SSD as there's no SATA on it. On the other hand according the Crucial their MX200 M.2 SATA SSD as compatible to that mainboard.
-> http://www.crucial.com/usa/en/compatible-upgrade-for/Supermicro/x10sdv-4c-tln4f

- Would you go for a Xeon E3 v5 rather than the Xeon-D setup (the price is about the same). I like the idea of having 10G (for future use maybe direct link to PC without a switch) and the possibility of adding tons of RAM (for the VMs).

- Does anyone know if it's possible to passthru onboard components of Xeon 1500 mainboards (I'm thinking of the SATA controller and 10G NICs)

The ideal board would be something with an onboard LSI controller but those are all at least micro ATX and I couldn't find any compact micro ATX NAS case with hotswap bays. Or is there any NAS chassis that fits the proprietary supermicro formfactor?

What do you say of the setup / my questions? Any recommendation? I really want to avoid spending tons of money for the wrong components.

Thanks a lot.

 

[jgreco]

Hi together,
I was using a home server (ESXi) and FreeNAS for quite a long time but since I moved and didn't take everything with me it's time for something new.

I always had 2 separate boxes for the ESXi and the NAS and I was always thinking of a way to put those together. I know you don't like the idea of virtualizing FreeNAS but let's talk about the options. I also want to have the setup as future proof and flexible as possible.

This is what I've selected to the moment:
4 x Western Digital WD Red 4TB, 3.5", SATA 6Gb/s (WD40EFRX) zRaid1
1 x Samsung SSD PM951 256GB, M.2 (MZVLV256HCHP-00000) (as VM datastore)
1 x Samsung DIMM 32GB, DDR4-2133, CL15, reg ECC (M393A4K40BB0-CPB) (for the moment, expand it later)
1 x Supermicro X10SDV-4C-TLN4F retail (MBD-X10SDV-4C-TLN4F-O)
1 x SilverStone DS380, mini-ITX (SST-DS380B)
1 x SanDisk Cruzer Fit 16GB, USB 2.0 (SDCZ33-016G-B35) (boot device)
1 x Corsair SF450 450W SFX12V (CP-9020104-EU)

+ a used LSI 9211-8i (for passing to FreeNAS)

Only 4 drives for the beginning, I want to add a 2nd storage pool in the future.

I'm planning to run a few (lightweight) VMs and sometimes 2 or 3 little nested ESXi servers for testing purpose. Or should I just forget the idea and use the VirtualBox plugin for now (and bhyve in the future, does bhyve support nested VT?)

ESXi runs better under ESXi.

Questions so far:
- I'm unsure about the M.2 interface on the Supermicro board. As far as I can see I can only use PCIe SSD as there's no SATA on it. On the other hand according the Crucial their MX200 M.2 SATA SSD as compatible to that mainboard.
-> http://www.crucial.com/usa/en/compatible-upgrade-for/Supermicro/x10sdv-4c-tln4f

You'll want to ascertain what the status of ESXi support for that would be. Suspect it probably works, but you might or might not be able to boot from it.

- Would you go for a Xeon E3 v5 rather than the Xeon-D setup (the price is about the same). I like the idea of having 10G (for future use maybe direct link to PC without a switch) and the possibility of adding tons of RAM (for the VMs).

The E3 v5 will be limited to 64GB. The D with its 128GB is a better choice from a memory perspective, where ESXi is always running out of memory. An E5 v3 (note: that's E*5*) is more expandable, but a midrange Xeon D like the 1540 has overall similar performance.

- Does anyone know if it's possible to passthru onboard components of Xeon 1500 mainboards (I'm thinking of the SATA controller and 10G NICs)

More likely the network.

The ideal board would be something with an onboard LSI controller but those are all at least micro ATX and I couldn't find any compact micro ATX NAS case with hotswap bays. Or is there any NAS chassis that fits the proprietary supermicro formfactor?

What do you say of the setup / my questions? Any recommendation? I really want to avoid spending tons of money for the wrong components.

Thanks a lot.

 

[joeschmuck]

Visualizing FreeNAS on ESXi works fine provided you setup the automatic shutdown and are disciplined enough to leave the FreeNAS system running like it should be and are not powering up/down your hardware all the time. If you plan to bring the system up and down, I'd run a separate FreeNAS system.

As for pass-through ... it's going to depend on the SATA controller. Some boards have a built-in separate LSI controller. For my system I was not able to so I had to add on some inexpensive SATA 4 port cards which are working like a champ. I could have gone with a single LSI controller but I already had one of these 4 port cards so a little more money and I'm cooking with gas as the saying goes. The Ethernet ports I have, even the motherboard ones I was able to setup as pass-through even though I reverted to using add-on cards because it was just easier for me to handle the virtual NIC setup. I will eventually grab the server and bring it back out of the basement and try to rework the Ethernet ports to remove all my add-on Ethernet cards. Right now my system has been running for a few months and if it works, don't fix it.

 

[the_tux]

Thanks so far, I'm not going shut the server down all the time, the base ESXi isn't going to be my lab. I would go for a larger mainboard but I don't know any compact case that has hot swap bays.

I'm not going to boot from the M.2 SSD (that's what the USB thumb is for) but since PCIe SSDs are relatively expensive and it doesn't make any performance difference for me (500MB/s is enough) I thought I can buy an M.2 SATA SSD (does anyone know if they are compatible like Crucial writes on their website?)

As you mentioned the 1540. That one is a little over my budget. For the same money I can go for a Xeon E5 2620v4 with a board that has a LSI 3008 on it (EPC612D4U-2T8R). Do you think to 1518 isn't enough for what I'm planning to do?

 

[jgreco]

It depends on what you're planning to do, which you haven't really outlined at all. There are people I know who've run two nested ESXi servers, a FreeNAS box, and another VM or two on a two core 16GB machine. You're not going to get a whole lot of heavy work done on that, but for ESXi lab work, it'd be fine. There are others who can blow through 128GB and a hexacore box and run out of space real quick.

 

[the_tux]

yes, you're right: The primary job is being a NAS, there are also a couple of other (lightweight) VMs such as Nagios, Webserver, pfSense and maybe some other stuff. For test and lab purpose I want to start some (up to 3) nested ESXi hosts with some RAM but that's all for testing and can be shut down afterwards.

What I also want to do is to encrypt the volume. I know I really have to take care of the key but having an unencrypted NAS isn't an option.

Data access will be mostly NFS

 

[the_tux]

Hi,
since things are changing all the time, Lab is getting more complex and actually the Xeon E5 platform isn't too expensive and I realize I run into troubles with the case and the HBA I'm thinking of changing my plans.

Get a larger case: Lian Li PC-9B (already has 4 hotswap, fill the rest with 3.5 / 2.5" bays later)
LGA 2011-3 Board: ASRock Rack EPC612D4U-2T8R (has an onboard LSI SAS 3008)
Get a Xeon E5v4 CPU
That makes it a bit more expensive, so the price will be about the same as the Xeon D-1540 plattform but I would say it gives me more flexibility so I'm futureproof (I know how fast time goes) for the moment.

Questions:
- anyone using that board? I can't really find much to it.
- Is the onboard SAS 3008 OK? Since It's connected via PCIe I should be able to passthrough. Is the 12G SAS already stable enough with FreeNAS?
- Does anyone know how the Xeon E5 performs with performance per MHZ? I'm thinking of getting a Xeon E5-2603 v4 or if it's worth the money go for a Xeon E5-2620 v4.

Thanks a lot guys!

 

[jgreco]

No, you really don't want to get an E5-26anything. The smart parts for single E5 are in the E5-16xx workstation lineup. The 1650 is just about the most bang for the buck you can get in E5, especially for NAS use where singlecore performance is helpful.

 

[the_tux]

OK, thank you! As I can see they are all 140W TDP. Does anyone know how the power consumption is in real life (idle)? And for the new v4 Xeons (14nm) there are no 16xx. Should I go with the old generation?
I know CIFS is single threaded and since I'm planning to go with 10Gb later it makes sense but are they really that slow?

 

[jgreco]

Getting all uptight about generational differences between the current CPU's is counterproductive. The most significant thing Intel's done in the last five years is take the E3's from 32GB to 64GB.

See for example the numbers I posted in

https://forums.freenas.org/index.ph...kylake-freenas-build.40711/page-2#post-259082

All the other excitement is happening outside the E3/E5 product lines (Avoton, Xeon D, etc). So do feel free to go compare the price and performance of something like the E5-1620v3 compared to the E5-2637v3, where you can see the insane price premium for the E5-26xx parts clearly demonstrated.

Intel does not publish idle power numbers.

 

[Ericloewe]

All the other excitement is happening outside the E3/E5 product lines (Avoton

Even Avoton has been stagnant for years. Back when it launched, Atom was supposed to receive a die shrink every year, and yet we're still "stuck" with the C2xxxs. Hell, they've been around so long that I think 16GB DDR3 UDIMMs have appeared from reputable manufacturers.

 

[jgreco]

Even Avoton has been stagnant for years. Back when it launched, Atom was supposed to receive a die shrink every year, and yet we're still "stuck" with the C2xxxs. Hell, they've been around so long that I think 16GB DDR3 UDIMMs have appeared from reputable manufacturers.

Yes, well, I'm talking *long* term trends. Shorter term, yes, I'd agree that for awhile it seemed like we were going to see some exciting stuff in the Avoton realm, but I think Intel developed a severe case of the Microsoft and is trying to fit everything into a smaller number of lineups. Despite the fact that the Avoton offered parts from 6...20w, I think Intel learned that people wanted the Avoton parts for server type usage, and that ate into their "Atom for storage" etc strategies. Then I think they figured out that it might be smarter to push them towards a slightly pricier/more capable SoC... the Xeon D...