New FreeNAS Setup & Design Advice (Dual Xeons)

[Jiryama]

Hello, I have decided to go ahead and jump into FreeNAS and all that it entails! Below is some information about what I would like to use it for, the hardware, and then the questions that I have - Thank you for any and all input! ( I own all of items mentioned except for the Case/HDDs at the moment)

The Plan: I am wanting to create an all purpose server that will let me access files while away, remote into the computer so I can use a throw away laptop while traveling, edit 4k video footage & RAW files, use make MKV on my current blu-rays. At first my plan was to get a Xeon E3-1246v3(Own it now) to run a small file server/cloud access off of and purchase an i7-6700k(own it) to build for my main PC. After browsing through FreeNAS and seeing everything that it was capable of, my desires expanded! I also learned about the facebook decommissioning their servers and got the processors below at $65 each. I finally have the ability to start editing 4k footage that I get from my gopro now! I am now just in a predicament on how I should set up this entire system given I have now 4 processors and only a plan for 2/4 that could potentially do everything I want - but I don't want to waste money spent on the other cpus. Any advice and ideas is great appreciated!

Use: I will be using the FreeNAS as a FileServer, PlexMedia Server, Mumble Server, ownCloud Server and potentially a Remote Connection/VM for windows(See questions).

Hardware:
Router: Netgear Nighthawk X8 R8500 (dual link agg)
Switch: HP ProCurve 24 port Managed Switch - JG924a (Quad link agg to mobo)

PCPartPicker part list / Price breakdown by merchant
CPU: Intel Xeon E5-2670 2.6GHz 8-Core Processor
CPU: Intel Xeon E5-2670 2.6GHz 8-Core Processor
Motherboard: ASRock EP2C602-4L/D16 SSI EEB Dual-CPU LGA2011 Motherboard
Memory: Crucial 16GB (2 x 8GB) DDR3-1600 Memory
Memory: Crucial 16GB (2 x 8GB) DDR3-1600 Memory
Storage: 4x Western Digital Red 4TB 3.5" 5400RPM Internal Hard Drive
Power Supply: EVGA 850W 80+ Gold Certified Fully-Modular ATX Power Supply

Questions:
1. What are your thoughts on a good SSI-EEB Case? I was thinking of doing the Fractal Design XL2 with HDD cages removed and an ICYDock hotswap 3x5.25" bay and a blu ray read/writer for x4 front bays

2. Can I create a VM using FreeNAS? I have an Samsung 950 pro M.2 ssd that I could purchase a adapter for an use the PCIE x4 to access and send it to a VM and run my main windows machine off of it. Send 16 of the 32 threads to a VM and a gpu for a main windows pc.

3. Will it be worth it to upgrade to more than 32gb of RAM and possibly max out to 128gb over time (16gb a month)?

4. With wanting to have the services I mentioned above, would it be wise to get an SSD to install those jails onto for better performance?

5. How much usable storage will I have out of 4 x 4TB drives? I was going to use the default Raid setup that it prompted but I would like the best performance with 1 maybe 2 HDD failures supported. I would replace immediately once one goes out. I edit raw photos and take about 100 hours of 4k footage a year. Lots of smaller files with fewer larger ones overall

6. I have an E3-1246v3 I intended to use at first, should I use that for the server and turn this dual xeon into a pure work station with windows for 4k editing or stick with the FreeNAS system on the dual xeon given the load I want to run on it?

7. What would you do with the leftover 6700k(Assuming I can use a VM to run windows off of for a main PC otherwise using this) and e3-1246 cpu if you were me? Does freenas work with PFsense where I could get another board and some ram for a router/firewall for the 1246?


Sorry for the multiple questions - just a bit unclear on the best path and design I should take from here.

Cheers!
Jiry

 

[Linkman]

(1) I've heard good things about the "Phanteks Enthoo Pro Series" http://www.newegg.com/Product/Product.aspx?Item=N82E16811854003 but I don't have any personal experience with it. When I was toying with the idea of building one of those eBay E5-2670 monsters, that was my pick for case.

 

[Jiryama]

(1) I've heard good things about the "Phanteks Enthoo Pro Series" http://www.newegg.com/Product/Product.aspx?Item=N82E16811854003 but I don't have any personal experience with it. When I was toying with the idea of building one of those eBay E5-2670 monsters, that was my pick for case.

I was definitely considering that one since you can get it for $70 right now, but I would have to remove the front USB if I wanted to add on the ODD. At the same time I don't really need inputs into the server though. I suppose that it would only matter if I can run a windows VM from freenas otherwise I would have a separate PC I would be plugging usb drives into.

 

[Nick2253]

Questions:
1. What are your thoughts on a good SSI-EEB Case? I was thinking of doing the Fractal Design XL2 with HDD cages removed and an ICYDock hotswap 3x5.25" bay and a blu ray read/writer for x4 front bays

Are you looking for a free-standing case, or do you want a rack-mount case? In the rack department, there are quite a few excellent cases (SuperMicro jumps to mind) that would give you lots of hot-swap bays.

2. Can I create a VM using FreeNAS? I have an Samsung 950 pro M.2 ssd that I could purchase a adapter for an use the PCIE x4 to access and send it to a VM and run my main windows machine off of it. Send 16 of the 32 threads to a VM and a gpu for a main windows pc.

FreeNAS supports a VirtualBox jail, so you could run a VM that way. Honestly, though, you might be a good candidate for virtualizing FreeNAS. I won't say too much more about that here, since there's a lot of information on the forums, but with the hardware you're bringing to the table, I think you could be very successful here.

3. Will it be worth it to upgrade to more than 32gb of RAM and possibly max out to 128gb over time (16gb a month)?

It largely depends on your workload. More memory will never hurt, but you'll rapidly reach your point of diminishing returns. The largest impetus for more RAM is more HDD space. However, with your service load, more might be needed. In your shoes, I'd probably start at 32GB, but be ready to jump to 64GB if needed.

4. With wanting to have the services I mentioned above, would it be wise to get an SSD to install those jails onto for better performance?

There's definitely something to be said for having SSDs behind your jails. However, you'd want at least two SSDs for redundancy, and for that price, you could add a few more HDDs to your array, which would give you good throughput and decent I/O. I'm thinking something like 6-8x drives in RAIDZ2. Or you could go the striped mirrors route, which would use more space, but give you much better I/O.

5. How much usable storage will I have out of 4 x 4TB drives? I was going to use the default Raid setup that it prompted but I would like the best performance with 1 maybe 2 HDD failures supported. I would replace immediately once one goes out. I edit raw photos and take about 100 hours of 4k footage a year. Lots of smaller files with fewer larger ones overall

The recommended setup is RAIDZ2 which is like RAID6: two redundant drives. That would only give you 8TBish (you need to adjust for the checksums and binary/decimal conversion). This is a good calculator: http://biduleohm.free.fr/zfsraidsarc/

6. I have an E3-1246v3 I intended to use at first, should I use that for the server and turn this dual xeon into a pure work station with windows for 4k editing or stick with the FreeNAS system on the dual xeon given the load I want to run on it?

I would say that dual E5s is way overkill for a home NAS load. In your shoes, I would use the E3 for a NAS, the dual E5s for a hypervisor, and the 6700k for your main PC. You said you already have a hypervisor, but I don't know what its specs are.

7. What would you do with the leftover 6700k(Assuming I can use a VM to run windows off of for a main PC otherwise using this) and e3-1246 cpu if you were me? Does freenas work with PFsense where I could get another board and some ram for a router/firewall for the 1246?

I'm not sure what you mean by FreeNAS working with pfSense? One's a NAS appliance, one's a firewall/router appliance. A 6700k is way overkill for pfSense, unless you want to do deep traffic inspection. I run pfSense with Snort off of an Atom D525, and I'm easily able to saturate my ISP (60Mbps).

 

[Jiryama]

Thanks for the response!

Are you looking for a free-standing case, or do you want a rack-mount case? In the rack department, there are quite a few excellent cases (SuperMicro jumps to mind) that would give you lots of hot-swap bays.

I have a 15U component rack from Sanus with 4U available for use however, it isn't a standard server rack so it is only 19" Deep and when I am using a SSI-EEB motherboard I have found it very difficult to find a case to accommodate that and everything else. So most likely looking for a free standing case unless you know of rack case that would fit it. Another concern I had with it being in my smaller component rack is the air flow as it is in a closet that granite has a vent.

FreeNAS supports a VirtualBox jail, so you could run a VM that way. Honestly, though, you might be a good candidate for virtualizing FreeNAS. I won't say too much more about that here, since there's a lot of information on the forums, but with the hardware you're bringing to the table, I think you could be very successful here.

Hmm.. are you referring to using something like VMWare to host 2 separate OSs, FreeNas with 8c16T and a Windows Unit with another 8c16T?

There's definitely something to be said for having SSDs behind your jails. However, you'd want at least two SSDs for redundancy, and for that price, you could add a few more HDDs to your array, which would give you good throughput and decent I/O. I'm thinking something like 6-8x drives in RAIDZ2. Or you could go the striped mirrors route, which would use more space, but give you much better I/O.

I could get 2x 240gb sandisk ultra II for $150 where expanding the array 2 more drives would run me $300. The other problem that I will run into soon is that I only have 6 SATAIII ports on this motherboard, 4 of them being marvel and 2 of them being something different. Limits my ability for expansion which is why I am going with 4TB hdds instead of 8x2tb which would have had better I/O. It is using Striped mirror similar to a raid 10 then? That would give me same available space since two hdds not being used for storage but faster I/O as you mentioned.

I'm not sure what you mean by FreeNAS working with pfSense? One's a NAS appliance, one's a firewall/router appliance. A 6700k is way overkill for pfSense, unless you want to do deep traffic inspection. I run pfSense with Snort off of an Atom D525, and I'm easily able to saturate my ISP (60Mbps).

I was just wondering if FreeNAS has a pfSense plug in or vice versa. It could become literally a one stop shop for all networking needs if it did.

I would say that dual E5s is way overkill for a home NAS load. In your shoes, I would use the E3 for a NAS, the dual E5s for a hypervisor, and the 6700k for your main PC. You said you already have a hypervisor, but I don't know what its specs are.

I wasn't sure since I was wanting to host the voice chat server mumble as well as Plex media Server for up to 4 people and was contemplating ownCloud. I was planning under the assumption 4 people would be wanting to watch from Plex and talk in a voice chat at the same time - as well as have resources for whatever else I expand to since I am looking at this to be a 5+ year system.

As far as set up sounds like my options are as follows:
Option 1: Dual E5s as host machine with FreeNAS at 12 threads/64gb ram and a Windows at 20 Threads/32gb ram. 6700k and E3 wouldn't be used (or a E3 pfSense project?)
Option 2: E3 for the NAS with 64gb ram and the Dual E5s with 32gb ram running windows. 6700k wouldn't be used
From my viewpoint I don't have a use for the 6700k honestly.. I don't really game and even when I do a dual e5 will still perform more than satisfactory. Only incorporation with all 3 is Dual E5 NAS/Plex/Mumble, E3 pfSense Project, 6700k main pc which seems completely over the top across the board.

 

[Ericloewe]

I was just wondering if FreeNAS has a pfSense plug in or vice versa. It could become literally a one stop shop for all networking needs if it did.

No. That's simply not going to happen. There was even a sticky thread on that subject on the pfSense forums, detailing why it's a genuine Very Bad Idea™.

Firewalls really should be separate devices. Virtualization makes sense in some very specific scenarios, but even then it's not alongside a file server.

 

[Nick2253]

Hmm.. are you referring to using something like VMWare to host 2 separate OSs, FreeNas with 8c16T and a Windows Unit with another 8c16T? ... I wasn't sure since I was wanting to host the voice chat server mumble as well as Plex media Server for up to 4 people and was contemplating ownCloud. I was planning under the assumption 4 people would be wanting to watch from Plex and talk in a voice chat at the same time - as well as have resources for whatever else I expand to since I am looking at this to be a 5+ year system.

There's really no reason to run all those services on FreeNAS, other than the convenience of being able to leverage one piece of hardware to do it. With your system, you'd be more than capable of running separate VMs for each service. Or you could use Docker Containers to do it. FreeNAS Plugins aren't the most reliable things, and they will be going away when FreeNAS 10 comes out (to be replaced by Docker Containers), so there's not a ton of longevity there.

2I could get 2x 240gb sandisk ultra II for $150 where expanding the array 2 more drives would run me $300. The other problem that I will run into soon is that I only have 6 SATAIII ports on this motherboard, 4 of them being marvel and 2 of them being something different. Limits my ability for expansion which is why I am going with 4TB hdds instead of 8x2tb which would have had better I/O. It is using Striped mirror similar to a raid 10 then? That would give me same available space since two hdds not being used for storage but faster I/O as you mentioned.

For your hard drive situation, especially if you decide to virtualize, I would get an IBM M1015 (or equivalent), which can be had extremely inexpensively off eBay. That would give you more than enough ports for HDDs.

Striped mirror is similar to RAID10.

As far as set up sounds like my options are as follows:
Option 1: Dual E5s as host machine with FreeNAS at 12 threads/64gb ram and a Windows at 20 Threads/32gb ram. 6700k and E3 wouldn't be used (or a E3 pfSense project?)
Option 2: E3 for the NAS with 64gb ram and the Dual E5s with 32gb ram running windows. 6700k wouldn't be used
From my viewpoint I don't have a use for the 6700k honestly.. I don't really game and even when I do a dual e5 will still perform more than satisfactory. Only incorporation with all 3 is Dual E5 NAS/Plex/Mumble, E3 pfSense Project, 6700k main pc which seems completely over the top across the board.

I want to make sure you understand the limits of virtualization. Typically, you wouldn't equip a hypervisor with a mouse and keyboard, and sit down and use it like a client PC. Instead, the hypervisor hosts a whole bunch of systems, and you access those using some kind of remote access (like SSH, VNC, or RDP) from a different computer. When you get into virtualization, you are really getting into enterprise-level things. That's not to deter you; just to remind you that there's something of a barrier to entry.

An E3 pfSense system would probably be overkill, unless you have 1Gbps internet. However, you could sell it and buy something smaller. Though depending on what you get for your E3, that may or may not be worth it.

A third option is that you could sell both the E3 and the 6700k, and get a couple smaller-duty systems: an Atom for pfSense, and a slightly older i3 for your main PC. Then, if you need the heavy duty power of your hypervisor, you could remote in to your Windows VM and run whatever you need to run.