BUILD Building a server for FreeNas primary for Plex

[bestboy]

Why would you recommend a E3 over a E5?

I recommended the E3 for wave 1, because it allows for a powerful yet economic build. I assumed that an E3 with maxed out memory was capable of handling all your storage needs while still saving some money to allow a realization of wave 2 within the budget.

I building the machine with 10-6TB (~60TB)

Then I overestimated your storage needs. I assumed you'd want to salvage the disks from you current NAS boxes and add another 10 disks of additional space.

Also you mention a HBA controller, wouldn't the built in controller on the MB be sufficient?

The onboard controllers are fine but usually do not provide enough channels for a storage system.

 

[AltecBX]

Could someone point me out to a inexpensive 10Gb switch?
I'm connecting (2 maybe 3) 10Gb Cat5e <30 ft (Server, PC, and future server) and (9) 100/1000Mb Cat5e (PC, Roku, 2 Wireless AP, Camera, 4 Rooms)

So far my my built is looking like this:
Motherboard - SUPERMICRO MBD-X10DRI-T-O Extended ATX Xeon Server Motherboard Dual LGA 2011 Intel C612 1 $529.99
CPU - Intel Xeon E5-2630 v3 Haswell-EP 2.4GHz LGA 2011-3 85W Server Processor BX80644E52630V3 1 $671.99
Memory - Crucial 64GB Kit (16GBx4) DDR4 2133 MT/s (PC4-2133) CL15 DR x4 ECC Registered DIMM 288-Pin CT4K16G4RFD4213 1 $831.45
Flash Drive - ADATA DashDrive UV128 64GB Flash Drive Model AUV128-64G-RBE 1 $28.99
Hard Drive - WD Red WD60EFRX 6TB 64MB Cache SATA 6.0Gb/s 3.5" NAS Hard Drive 10 $2,899.90
Case - 3U Supermicro 16 bays Storage Server Chassis SC836E26-R1200 SAS2-836EL2 6Gbs 1 $375.00
Rack - 12U IT Wall Mount Network Server Data Cabinet Rack Glass Door Locking Lock & Key 1 $189.95
OS - FreeNas - 9.3 1 $-

 

[Ericloewe]

Could someone point me out to a inexpensive 10Gb switch?
I'm connecting (2 maybe 3) 10Gb Cat5e <30 ft (Server, PC, and future server) and (9) 100/1000Mb Cat5e (PC, Roku, 2 Wireless AP, Camera, 4 Rooms)

No such thing exists, as of two months ago. Even 24GbE + 2 10Gbe uplinks cost 250 bucks, for a nice smart switch.

 

[diedrichg]

I'd suggest buying your drives from multiple sources with the idea that that would split up the manufacturing dates on the drives and could save you from multiple failures in case they had bad batches.

 

[AltecBX]

I think I lucked out; all my drives have June & July manf dates on them

When I say inexpensive, I know I won't find anything under $200 but need pointing where to get something not to expensive with at least 12-16 ports. I just don't want to spend >$500 on a switch.

 

[mjws00]

If you find one let us know. I think the cheapest I've seen has been the netgear prosafe. Best price I've seen was ebay at $729. They haven't hit a low price point yet. My intent is to direct connect until I can land a deal.

 

[AltecBX]

If you find one let us know. I think the cheapest I've seen has been the netgear prosafe. Best price I've seen was ebay at $729. They haven't hit a low price point yet. My intent is to direct connect until I can land a deal.

So are you crossing over Cat 5e or 6 cables from an additional ethernet ports on both PC? If that's doable, thats not a bad idea.. Might have to do the same.

 

[mjws00]

Yep. Direct connect between my greedy workstation's 10Gb nic and one of the server 10Gb nics. Everyone else gets 1Gb. ;)

 

[AltecBX]

Anyone know where can I get a wall mount rack to handle this chassis?
Dimensions
Height 5.2" (132mm)
Width 17.2" (437mm)
Depth 25.5" (648mm)
Gross Weight 52 lbs (23.6 kg)

http://www.supermicro.com/products/chassis/3U/836/SC836E26-R1200.cfm

I'm looking for a wall mount since I will only need this 3U and maybe a future 2U plus modem and switches. I've been looking for a 9U rack on NewEgg & eBay, but they don't seem to be deep enough to handle this chassis. Any suggestions?

 

[AltecBX]

So I'm finalizing my build guys. So far this is what I have:
HD - 10x WD Red 6TB
Rack - Tripp Lite SRW12US33
Chassis - 3U Supermicro 16 bays Storage Server Chassis SC836E26

I have a question about the MB. I'm on the fence between the SUPERMICRO MBD-X10DRI-T-O and the SUPERMICRO MBD-X10SRH-CLN4F-O.
One has dual chip and 10Gb at $529 & the other one is single chip, 1Gb, with SAS3 (12Gbps) via LSI 3008 at $419.
So with the first one I have 10x SATA3 and the second one I have 18x SATA connection (10x SATA3 through the Intel C612 & 8x SAS3 through the LSI 3008.

At the moment I bought the Rack to mount in my garage. That's where all the cat 5e cable terminate so that's the central hub of my network. I'm installing my new 3U server, switch, modem, and router on that rack. The problem is that my workstation will be in my office and I only have one cat 5e cable and I won't be able to do the crossover cable trick @mjws00 mentioned. And I don't want to invest $1,600 on (2) 10Gb switch. I'm also not installing a 2nd cpu at the moment. So the X10DRI-T-O is looking like less of an option.

On the other hand I'm using 10x WD Red 6TB out the bat. The X10SRH-CLN4F-O also lets me add an additional 8x drives up to 12Gb speed. With a 16 Bay chassis, I can easily throw in 6x more as needed and scale vertical. I believe the E5-2630 v3 will also serve me fine for what I'm going to do. Going this route saves me $110 and gives me 8 more ports for hard drives. The only downside I see to this, is that I'm not getting the 10Gb ports.

What to do?

Also I've been familiarizing myself with ESXI 5.5 vSphere Hypervisor and I really like the idea of me installing it on this rig. This machine will be powerful enough to share its resource with FreeNas, Kali Linux, and Windows Server 2012. I've read on this forum that you should stay away from installing FreeNAS on a VM. But I've also read others with FreeNAS running on VM. You guys see my setup and hardware, will you guide me to stay clear from a VM or go for it? I like @bestboy advice on scaling; this would give me the option to do so if needed on one box. I will admit, if one of you guys have free time and are up to the challenge with this new hardware I build, I would like help getting this sucker up. I won't decline the offer.

ps - If I do end up with the X10SRH-CLN4F-O, do I populate the 8 ports on the LSI 3008 first, and then hook up the other 2 to the C612 or it doesn't matter?

 

[mjws00]

You can always get the 10gb later. The extra ports match the case nicely. You pretty much have ideal hardware for an esxi build only snag I see is a 10 drive pool can't run on the 8 port 3008 if you pass it through. I don't know if you can pass through the pch 10 ports without hassle.

Changing from esxi to baremetal is literally the swap of a USB if you follow jgreco's guide. So if crap hits the fan you can troubleshoot a baremetal rig here. I lost my reservations about virtualizing when jgreco was kind enough to point out his long term install. Mine has taken a 6 month unmerciful beating like a champ... but that isn't that long. There are more moving parts so you have to evaluate appropriate risk for yourself. This isn't an enterprise install so flexibility is a consideration, as is better leveraging the investment. I don't cry over data, and I back my crap up.

Experiment and see where you land is the best advice I can give. That is less paternalistic than some advice. But it is bits... no one dies if you lose one at home.

 

[AltecBX]

I see your point. I just took my old quad core computer to play with ESXI. I just got it running and I'm tinkering with it. One thing I notice is that it says I have 60 days. I thought ESXI was free?

Also If I go with a 10 drive setup for FreeNAS, I won't be able to do it with the built in LSI 3008 plus 2 ports on the C612?
Or do recommend using the 10 SATA ports on the C612(Wellsburg PCH) for ZFS on FreeNAS, and then using the LSI 3008 8 ports for other OS and data on the ESXI?

 

[Ericloewe]

I see your point. I just took my old quad core computer to play with ESXI. I just got it running and I'm tinkering with it. One thing I notice is that it says I have 60 days. I thought ESXI was free?

Also If I go with a 10 drive setup for FreeNAS, I won't be able to do it with the built in LSI 3008 plus 2 ports on the C612?
Or do recommend using the 10 SATA ports on the C612 for ZFS on FreeNAS, and then using the LSI 3008 8 ports for other OS and data on the ESXI?

Pass the 3008 through to FreeNAS.

 

[AltecBX]

Pass the 3008 through to FreeNAS.

So If I pass through 8 SATA 12Gbs through the LSI 3008 to FreeNAS, will I still be able to add another 2 SATA 6Gbs through the C612(Wellsburg PCH) to built the ZFS pool?
Is there a reason you recommend the LSI 3008 rather than the C612(Wellsburg PCH)?

ps. v1 & v2 processors focus on SAS connectivity. At least the v3 processors plus MB alleviate that.

 

[Ericloewe]

So If I pass through 8 SATA 12Gbs through the LSI 3008 to FreeNAS, whill I still be able to add another 2 SATA 6Gbs through the C612(Wellsburg PCH) to built the ZFS pool?
Is there a reason you recommend the LSI 3008 rather than the C612(Wellsburg PCH)?

Not unless you pass the PCH's SATA controller through - which leaves you with no storage.

To be honest, this is the kind of thing you must absolutely know before even thinking of VMs. This kind of thing is essential and you will have trouble if you do not understand exactly what you're doing.

Generally, PCI-e controllers are easily passed through, unlike the PCH. You can use expanders with the SAS controller to add more drives (with 12Gb/s per channel, there's bandwidth to spare, but the prices for expanders are probably rather high).

 

[AltecBX]

Good to know, many thanks in advance. This is why I'm asking the questions now.
I then will lower my 10 drive pool to 8 for FreeNAS and go through the LSI 3008. ((8*6TB=48TB total pool size) - (2*6TB=12TB partiy ZFS2)) = 36TB of space for FreeNAS right?

I have a 16 bay chassis. So if I want to grow my FreeNAS pool in the future, you recommend me buying an SAS expander($300) to add more drives(space) rather than using the PCH ports? So basically leave the 10 ports(Wellsburg PCH) for the VM to give out to the other VM's?

 

[bestboy]

Good to know. I then will lower my 10 drive pool to 8 for FreeNAS and go through the LSI 3008. ((8*6TB=48TB total pool size) - (2*6TB=12TB partiy ZFS2)) = 36TB of space for FreeNAS right?

Nope, you also have to consider the ZFS CoW tax. Your pool should never exceed 80% capacity. That's especially true for VM storage that will likely see massive fragmentation.

36 TB * 0.80 = 28,8 TB

BTW: RaidZ is not really good for VM storage.

 

[gpsguy]

One thing I notice is that it says I have 60 days. I thought ESXI was free?

If you don't install a license you are granted a 60 day evaluation version with access to a bunch of product features. For the "free" version you need to request a license from VMware. Once you install it, most of the product features will disappear.

The free version is quite usable - I use it at home and at work (for testing stuff). I also have a $$$ license at work for VM's used in production.


Sent from my phone

 

[AltecBX]

Nope, you also have to consider the ZFS CoW tax. Your pool should never exceed 80% capacity. That's especially true for VM storage that will likely see massive fragmentation.

36 TB * 0.80 = 28,8 TB

BTW: RaidZ is not really good for VM storage.

29TB is to small for what I invision in the next coming years. I need 18TB to start when I transfer all my media now.
I've been reading mix signals about this on this forum and on the internet. Some ppl has run it without a glitch and some had problems. Some developers are pointing that FreeNAS 10 will fully support VM. What's the deal with this? I'm building my machine in about 2 weeks. By that time I believe FreeNAS 9.3 will be out of beta. I believe even @jkh even pointed out that VM is the future which I believe as well. This is why I'm asking the questions now so I can iron out any of problems and glitches.

If you don't install a license you are granted a 60 day evaluation version with access to a bunch of product features. For the "free" version you need to request a license from VMware. Once you install it, most of the product features will disappear.

The free version is quite usable - I use it at home and at work (for testing stuff). I also have a $$$ license at work for VM's used in production.

Sent from my phone

Ahh thanks. I will contact them once my trial expires.
ps - Any Mac vSphere or access to the Hypervisor?

 

[mjws00]

You don't need another sas expander, a $100 m1015 would give you 8 more ports. It may be possible to pass through the 10 ports on the pch, but depending on how things are done it may force you to pass through the usb controllers along with it, who knows. You need something to boot from, so you can't do that. If the usb controllers are broken out for esxi, you can pass through all your sata ports, but short of finding someone with that board, and the esxi device list in front of you... you won't know. What you do know for certain is that you can pass through that 3008.

The big deal with virtualization is that it adds risk. Often people try and half-ass it, and end up burning themselves with a random pool loss. Frankly you have about the best possible hardware for a vt-d set up. So while you can't dismiss a problem entirely, you've at least optimized your scenario. There really isn't much you can do beyond that, besides not do it.

The bottom line on doing this is learning your recovery scenario's. Boot esxi, boot to baremetal ensure you know how to backup and restore the config. Your fallback on any crazy panic is... boot to baremetal and worry about the pool from there. Since you've tested and played with your set up you'll know exactly what is going on, and it is a non-event.

A healthy respect is a good thing. Utter fear is useless. But bear in mind... when you color outside the lines. You bear full responsibility. :)