48TB NAS Build / Compile Hardware List

[thepixelgeek]

So close to pulling the trigger...just need a little advice first. Especially, since this will be my first build.

I think I narrowed it down, but open to advice to reduce cost while keeping same effectiveness.

To be honest, my head is spinning from all the options. I've been reading for days, and still need to read the user guide all the way through. Hardware recommendation guide was very helpful to get started.

I recently ordered the ASRock E3C224D41-14S, but I may be returning it because of the 32gb limit. I realized if I want a 48TB NAS, I will need more RAM due to the 1G per TB requirement. 64gb by my calculation.

I'd like to start with (6) 4TB drives and add 6 more in the near future. VDevs for each set of 6 drives. RaidZ2 for safety/redundancy.

NAS GOALS:

1. Plex server Transcode & Stream (3 x 1080p - 20 Mbps Max)
2. "Futureproof" - transcoding of 4k movies...or the very least, have a fighting chance...only 1 stream at a time.
3. Run multiple plugins/Jails with Couchpotato, Sonarr, sabnzb, etc
4. Basic storage music, movies, pictures, and backups/images of home computers.
5. Use as a shared drive for family photos and music
6. Video surveillance storage 1-2 cameras max
7. Potentially remote access..cloud storage
8. 24/7 operation
9. Cost to performance ratios. Obviously would like to get the best bang for buck.
10. Have fun and learn about FreeNAS without too many install/compatibility troubles.

48TB NAS BUILD:

1. MOBO/CPU: SuperMicro MBD-X10SRL-F-O ($270)
   [Edit:] Intel Xeon E5-1620 v3 @ 3.50GHz ($305 / Passmark 9704)
   MEM-DR432L-SL01-LR21 Samsung 32GB DDR4-2133 4Rx4 LP ECC LRDIMM Memory ($269)
   
   or
2. MOBO/CPU: Supermicro MBD-X10SRH-CLN4F-O ($389)
   Intel Xeon E5-1620 v3 @ 3.50GHz ($305 / Passmark 9704)
   MEM-DR432L-SL01-LR21 Samsung 32GB DDR4-2133 4Rx4 LP ECC LRDIMM Memory ($269)
   
   or
3. MOBO/CPU: Supermicro MBD-X11SSH-LN4F-O ($239)
   [Edit]: Intel Xeon E3-1270 v5 @ 3.60GHz ($360 / Passmark 10107)
   MEM-DR416L-SL01-ER21 Samsung Memory - 16GB DDR4-2133 ($100)

Common:

1. HDD: (12) WD Red 4TB NAS Hard Disk Drive - 5400 RPM
2. BOOT: (2) Mirrored SanDisk Ultra Fit 32GB USB 3.0 or (1) SSD
3. PSU: [Edit:] SeaSonic SS-1050XP3 1050W ($169)
4. CASE: Rosewill RSV-L4412 - 4U - 12 SATA / SAS Hot-swap Drives

Thank you in advance.

 

[Spearfoot]

With 12 HDDs you might consider using a larger PSU, probably 750W or more... see this thread:

https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

Or go to the Supermicro website and research their pre-configured systems using the motherboard you select and see if they offer a 12-drive system -- then see how many Watts they specify for the PSU.

 

[tvsjr]

A few random thoughts...
I have no experience with the Rosewill cases, but some of their stuff is regarded as pretty cheap quality. You might do a little searching to see whom else might be using one, and their experiences with it.
IMO, SSD for the boot. Mirror it if you're extra-paranoid (like me).
On the processor, there's no real point in an E5-2xxx processor unless you intend to add a second chip down the road. Also, you have an E3-1241v3 matched to an X11 motherboard - which requires E5-12xx v5. This feels like a good build for an X11 with an E3v5.

 

[Arwen]

Please note that the 1GB of RAM per 1TB of storage is a general guideline. After you get to 16-32GB RAM
it's less important. (Unless you use de-duplication, which is NOT recommended for first time ZFS users.)

That said, you plan on several plugins and jails, so going with at least 16GB to start would be recommended.
FreeNAS has tools to determine if you are running low on ZFS ARC, (Adaptive Read / Replacement Cache).
Use them over a few weeks or months. Then you can increase to 32GB or higher if it seems warrented.

I do recommend that you use the largest density DIMM size available for your board. Makes it less painful to
upgrade. Meaning you don't have to pull out a lower density DIMM to replace it with something larger.

 

[Dice]

I'd grab mobo from (1.) and CPU from (2.). That is my preferred setup.
As mentioned - PSU needs a buff.
Agree with memory module size mentioned by @Arwen

 

[thepixelgeek]

Please note that the 1GB of RAM per 1TB of storage is a general guideline. After you get to 16-32GB RAM
it's less important. (Unless you use de-duplication, which is NOT recommended for first time ZFS users.)

That said, you plan on several plugins and jails, so going with at least 16GB to start would be recommended.
FreeNAS has tools to determine if you are running low on ZFS ARC, (Adaptive Read / Replacement Cache).
Use them over a few weeks or months. Then you can increase to 32GB or higher if it seems warrented.

IF that's the case, would it be wise to keep the ASRock board and max out the RAM?

 

[thepixelgeek]

Narrowing it down, and considering the ASRock again (only if RAM is not considered an issue?):

1. MOBO/CPU: ASRock Rack E3C224D4I-14S ($366) !! Not sure if the negative reviews and RAM constraints are show stoppers? !!
   Intel Xeon E3-1241 v3 @ 3.50GHz ($288 / Passmark 10028)
   (2) Crucial 16GB DDR3 SDRAM ECC Unbuffered CT2KIT102472BD160B ($119)
   
   or
2. MOBO/CPU: Supermicro MBD-X11SSH-LN4F-O ($239)
   Intel Xeon E3-1270 v5 @ 3.60GHz ($360 / Passmark 10107)
   (4) MEM-DR416L-SL01-ER21 Samsung Memory - 16GB DDR4-2133 ($100) - (QVL)

 

[Ericloewe]

Not sure if the negative reviews

The negative reviews display pure ignorance, as is usual. Don't worry about them.

 

[thepixelgeek]

The negative reviews display pure ignorance, as is usual. Don't worry about them.

Would you do the ASRock or the X11, based on my goals? Or, perhaps a better question, which will satisfy my goals best?

 

[Jailer]

Couple of quick notes to add what the others have contributed.

I'd like to start with (6) 4TB drives and add 6 more in the near future.

If you are starting this build with the expectation of adding more drives in the future why not start with larger drives? It's your build to do what you want but if your storage needs warrant it I'd start with larger drives. Do keep in mind the actual usable capacity of your pool after formatting, ZFS overhead and the 80% guideline.

• Plex server Transcode & Stream (3 x 1080p - 20 Mbps Max)

• "Futureproof" - transcoding of 4k movies...or the very least, have a fighting chance...only 1 stream at a time.

An X-11 skylake based xeon build will fit this at the present time. It's also my opinion that it's a fools errand to try and "future proof" a computer build when you don't know what the future will bring. When the future comes there will likely be more capable hardware that is better up to the task of completing the goal warranting an upgrade. Kaby Lake and native x265 decoding is a perfect example of this.

Run multiple plugins/Jails with Couchpotato, Sonarr, sabnzb, etc

Keep in mind that FreeNAS 10 is doing away with jails in favor of virtualization and Docker containers. This shouldn't hold up your build but plugins will be going away and your CPU memory requirements might change because of this.

Cost to performance ratios. Obviously would like to get the best bang for buck.

This is one that gets lost on a lot of new users as it did on me when I first started. You have to look at the cost of the entire system including drives for your cost/performance/storage requirements. That's what drives my recommendation for starting with larger drives on your build.

Good luck and if you get stuck on anything post up for help. There are plenty of knowledgeable members here willing to help out.

 

[tvsjr]

Narrowing it down, and considering the ASRock again (only if RAM is not considered an issue?):

1. MOBO/CPU: ASRock Rack E3C224D4I-14S ($366) !! Not sure if the negative reviews and RAM constraints are show stoppers? !!
   Intel Xeon E3-1241 v3 @ 3.50GHz ($288 / Passmark 10028)
   (2) Crucial 16GB DDR3 SDRAM ECC Unbuffered CT2KIT102472BD160B ($119)
   
   or
2. MOBO/CPU: Supermicro MBD-X11SSH-LN4F-O ($239)
   Intel Xeon E3-1270 v5 @ 3.60GHz ($360 / Passmark 10107)
   (4) MEM-DR416L-SL01-ER21 Samsung Memory - 16GB DDR4-2133 ($100) - (QVL)

Of these, #2 would absolutely be my pick. I have experience with only one ASRock board (a mini-ITX vehicle PC) and it worked well... but Supermicro is very well supported around here and Just Works. There's a lot to be said for that.

 

[Ericloewe]

Would you do the ASRock or the X11, based on my goals? Or, perhaps a better question, which will satisfy my goals best?

I'd probably go straight to 6x 8TB with the X11, assuming a return is feasible.

When you do want to expand later, you just add an HBA.

 

[thepixelgeek]

I'd probably go straight to 6x 8TB with the X11, assuming a return is feasible.

When you do want to expand later, you just add an HBA.

Ha! I wish I could afford to do 6x 8TB...I'll stick with 4TB versions.

Thoughts on 12x WD Red together in same chassis? Manufacturer rates it at 8 max. Not sure why...vibration perhaps? Heat?

Thinking I may need to move up to Pro version?

 

[tvsjr]

Ha! I wish I could afford to do 6x 8TB...I'll stick with 4TB versions.

Thoughts on 12x WD Red together in same chassis? Manufacturer rates it at 8 max. Not sure why...vibration perhaps? Heat?

Thinking I may need to move up to Pro version?

I'm sure they'd tell you vibration and heat. Of course, if you manage those properly, no issue. The real reason is probably product tiering... they want "advanced" users to pay the Pro premium.

Keep in mind, the Pros are 7200RPM, meaning more heat. That might be fine (I run 7200 HGST SATA NAS drives, the few extra IOPS can't hurt and the extra heat isn't an issue) but it might be overkill.

 

[thepixelgeek]

So much great info here, and on other forum posts...Based on my current understanding, here's my build list and cost:

MOBO: Supermicro MBD-X11SSH-LN4F-O ($239)
CPU: Intel Xeon E3-1270 v5 @ 3.60GHz ($360 / Passmark 10107)
RAM: (4) MEM-DR416L-SL01-ER21 Samsung Memory - 16GB DDR4-2133 ($100) - (QVL)
HDD: (12) WD Red 4TB NAS Hard Disk Drive - 5400 RPM ($140)
BOOT: ADATA Premier SP600 32GB SATA III (ASP600S3-32GM-C) ($30)
PSU: SeaSonic SS-1050XP3 1050W ($169)
CASE: Rosewill RSV-L4412 - 4U - 12 SATA / SAS Hot-swap Drives ($230)
PSU: APC Back-UPS Pro 1000VA (BR1000G) ($122)

TOTAL COST: $3230 / $80.75 per TB

Something to consider...with 6TB drives, the cost per TB comes way down:

HDD: (12) WD Red 6TB NAS Hard Disk Drive - 5400 RPM ($230)
TOTAL COST: $4310 / $71.80 per TB

 

[Ericloewe]

Not sure why...

Marketing.

 

[thepixelgeek]

Please correct me if I'm wrong...

I realized the X11SSH ($239) only has 8xSATA3, which I believe that means 8 drives directly attached to the board. That said, I assume the IBM M1015 controller (highly recommended) is needed to achieve my goal of 13 total drives? If so, is there anything lost/gained when using the M1015? ...still researching on the forums... Is it worth the extra $130?

Or

Would the Supermicro MBD-X11SSL-CF ($270) be a better choice since it has 6xSATA3 AND 8xSAS3 via LSI3008 controller? Is the LSI3008 better/worse than M1015? The way I see it, I'm saving $$, but is at the cost of frustration down the road?

Lastly, what's the difference between the chipsets (C232/36)?

 

[diedrichg]

The E3-1270 v5 will get you about 5 simultaneous 1080p transcodes. (~2000 PassMark score per 1080p transcode)

 

[Jailer]

Or

Would the Supermicro MBD-X11SSL-CF ($270) be a better choice

If you know for certain you are going to add more drives this is the route I would go. It can be flashed to IT mode just like the IBM 1015 so no worries there and it would save some money on your build.

Lastly, what's the difference between the chipsets (C232/36)?

More PCIe lanes, more USB 3.0 ports and more virtualization specific features.

C232: http://ark.intel.com/products/90578/Intel-GL82C232-PCH#@product/specifications

C236: http://ark.intel.com/products/90594/Intel-GL82C236-PCH#@product/specifications

 

[Stux]

http://ark.intel.com/m/compare/90578,90594