BUILD Proposed 10x6TB Home Server Build

[n3IVI0]

Greetings! This is my first post here, by the way. Been spending some time perusing the boards, in particular @cyberjock 's helpful Hardware Recommendations guide (https://forums.freenas.org/index.php?threads/hardware-recommendations-read-this-first.23069/).

The purpose of this home server will be general file storage, media streaming via Plex, disc ripping via Windows 7 and Virtualbox, and running Transmission, SabNZBD, Sonarr, Madsonic, Owncloud, and Mylar.

It will be a 10x6TB setup, utilizing RAIDZ2, which based on my reading seems to be the best balance between parity and storage, and still gives enough speed to saturate a gigabit ethernet pipe. Coupled with offsite backup via Crashplan and an APC UPS, it should be reasonably bulletproof.

I had initially intended to only have 8 drives, but discovered that RAIDZ2 is better with 10. I also ran a calculator to discover that I'd be losing enough storage to parity with 8 as to not be a sufficient increase over what I already have (I am planning to re-convert my media as uncompressed mkv rips). By my rough calculations, I should end up with approximately 49 TB of useable storage, of which 80% is 39.2 TB. This should be more than adequate for my needs.

I have modified the design proposed by @Pheran (https://forums.freenas.org/index.php?threads/pherans-32tb-freenas-build-with-photos.35694/), only increasing the storage a bit, and changing a few components to accommodate the extra hard drives. Thanks for that by the way. The pictures were EXTREMELY useful in helping me visualize how it will all go together.

Here is the shopping list:

• Case: Fractal Design Define R5 (I want the silence, and the cable management)
  
• Motherboard: Supermicro X10SL7-F (obviously)
  
• CPU: Intel Xeon E3-1241 v3 (this seems to be merely a newer, faster version of the CPU recommended in both guides)
• PSU: Seasonic SS-660XP2 ATX (I went with this one because it's Platinum rated, and has 10 SATA power connections)
• RAM: Crucial 16GB (2 x 8GB) 240-Pin DDR3 SDRAM ECC Unbuffered DDR3 1600 (PC3 12800) Server Memory Model CT2KIT102472BD160B (I intend to get 32GB, so two kits; also I've always been a huge fan of Crucial memory)
• HDD: 10 - WD Red WD60EFRX 6TB (ouch, I know; I have been using 3TB WD Reds in my current build, and I won't use anything else. Never had any problems. Run cool, easy on the watts)
• Boot Media: Supermicro SSD-DM064-PHI SATA DOM (I realize 64GB is probably overkill, but I prefer to have lots of wiggle room for as yet unrealized future needs)

As @Pheran has done, I will be adding an additional 140mm fan to the front (Fractal Design Dynamic GP-14 140mm Fan). I will be using these Akust 3.5" to 5.25" HDD Sink Dual Adapters to shoehorn two more drives into the bottom 5.25" bay. I will be adding this bay cooler, whereupon I will replace the stock fans with three Fractal Design Silent Series R2 FD-FAN-SSR2-40 40mm Fans to keep the drives cool and filtered. My trusty Pioneer Bluray burner will then go in the top 5.25" bay. I will be using all 8 SAS ports on the motherboard, having flashed it to IT mode, and two of the SATA III ports, leaving one SATA II port for the SATA DOM.

I think that's everything in a (large) nutshell. If anybody has any addendums, alterations, or suggestions, please chip in. This will be a significant investment. I have tried very hard to consider all possibilities, but it never hurts to have extra eyes on a project in case I missed something.

Thank you for your help! Greatly looking forward to the big step up to server grade components and RAID.

 

[Jailer]

Why are you putting the blu ray drive in your server? With IPMI there is no need for an optical drive.

It's entirely up to you but a 32GB SATA DOM is more than sufficient for a boot device. I use a 16GB and have plenty of available space even with multiple boot environments saved. Just suggesting one area you could save a couple bucks.

 

[n3IVI0]

Right. I was just going to use it for ripping discs. It is currently connected to a Mac mini via a USB3 enclosure. If I could get it internalized and run directly from the system, it would speed up the ripping. Is there a reason ripping discs via VirtualBox would not work? Has anybody tried running Handbrake and MakeMKV via Virtualbox?

 

[Jailer]

I run handbrake in a windows vm but I've never tried makemkv. I run my rips from my windows box and just rip them to a share. Give the windows VM share access and and encode using handbrake in the windows VM.

If I had a newer/faster desktop I'd just do it all on my desktop.

 

[n3IVI0]

I keep backup copies of all the original BluRays and DVDs on a stack of hard drives external to my array. When I need to access a disc, I have a dual hot-swappable external drive bay, so I just drop the disc into the slot, mount it, work my magic on it, and copy the resulting file to the server for viewing. What I was thinking was put the Bluray drive into the new server, mount the backup HDD via USB, rip the discs to the backup HDD for storage, then run MakeMKV on the copies to generate the mkvs for the media server. I was hoping to eliminate a possible bottleneck by removing the USB connection for the Bluray, and just do the ripping and encoding all on the server in Virtualbox.

 

[danb35]

I think I've seen some threads around here asking about the idea of using the server's optical drive with VirtualBox. My memory is that it isn't well, if at all, supported. I don't have any examples, but I'd suggest you do a little digging to see if this is even possible.

 

[n3IVI0]

Well, I'm glad I asked the question. It's not a problem. I'll just install a hard drive cage in the two 5.25" bays, one that has a front fan and filter. This way will ensure proper ventilation for the extra drives. I can still keep doing what I was doing, which is using the optical drive with a Mac mini. No problem. I was a little concerned about keeping those two drives cool enough, so this way will probably be better in the long run.

 

[NAStard]

The open box sale from newegg is reliable based on my experience.
Check these two out:

SUPERMICRO MBD-X10SRI-F ATX Server
SUPERMICRO MBD-X10SL7-F-O uATX Server

Cheers!
:D

 

[n3IVI0]

Thank you!

 

[NAStard]

I also saw another model with 4 Intel Gigabit network ports plus a dedicated IPMI (~$350). I wish I had bought that one. Now that I got my FreeNAS running, I realize that I will want aggregated network interfaces because these storage boxes can easily max out the capacity of Gigabit network, it appears.

 

[n3IVI0]

I think then I will place the two extra hard drives with this 3-in-2 cooling box, and replace the fan in it with a Fractal Design Silent Series R2 FD-FAN-SSR2-80 80mm Fan. That will satisfy my hard drive needs, and keep them cool and filtered.

 

[Robert Trevellyan]

I had initially intended to only have 8 drives, but discovered that RAIDZ2 is better with 10.

Any advice claiming that 2^n + p disks is optimal for ZFS is invalid except under very specific conditions, which do not apply to you, so don't base any decisions on that.

 

[NAStard]

Thanks for sharing info about that cooling box. The ones I had seen before, I would not buy because they seemed so cheaply made and over-priced as well. Buying this one will not make me feel foolish. haha.

Are there downsides to using 6TB (as compared to 4TB, for example) when it comes to resilver/rebuild?
What is your pool org going to look like with 10 spindles?
BTW, I use a relatively slow and old 4GB USB stick to boot this box. Works fine, so far.

Like Robert said, the powers of two thing seemed to have been debunked; keep those RAIDz numbers low. Perhaps make more than one raidz vdev and mirror them?
It is at this juncture I want to try introducing an additional LSI controller for each vdev. :)

Cheers!
:D

 

[danb35]

I realize that I will want aggregated network interfaces

Aggregated network interfaces are only useful in a pretty limited set of circumstances. What may be better would be to install a 10G NIC in your server and connect it to a switch with a suitable 10G port.

 

[NAStard]

I am so glad you share that, dan.
Thank you for the direction.

 

[NAStard]

Why are you putting the blu ray drive in your server? With IPMI there is no need for an optical drive.

Hi Jailer... I'm curious if you could elaborate on what IPMI can do with regard to optical drive? I'm not aware. The only thing I heard could be done (but have not implemented myself) is to route serial console data via IPMI.

Thank you!
:)

 

[n3IVI0]

Should I have multiple vdevs? I was just going to have the one 10 spindle vdev in RAIDZ2. If, say, I had a pool with two 5-spindle vdevs, how would this affect total usable storage? Wouldn't that cut it in half?

 

[Jailer]

Hi Jailer... I'm curious if you could elaborate on what IPMI can do with regard to optical drive? I'm not aware. The only thing I heard could be done (but have not implemented myself) is to route serial console data via IPMI.

Thank you!
:)

You can mount an image file to do an OS install over the network interface instead of using physical media.

 

[n3IVI0]

Is there any reason I couldn't have one vdev with 10 drives in it in RAIDZ2? Is this dangerous? Impacts performance?

 

[Bidule0hm]

One vdev is fine. The limit is usually 11-12 drives per vdev.