BUILD Required RAM for my First FreeNAS Build incl. Plex (MB: A1SA7-2750F)

[moxxii]

Hi guys,

I´ve read cyberjock´s PowerPoint presentation and his HW recommendation guide.

Purpose:
- running Plex Media Server on the FreeNAS build
- about 22 TB of usable storage (Raid-z2), 1 vdev
- 12 hours per day operation in my living room
- Backups on Qnap TS-569 Pro with 5x WD30EFRX (RAID5), very important stuff is backuped on an additional external Hard Drive. The Qnap is located at my parents apartment, so if anything happens to my own apartment, I still have most of my data.

What I have ordered first (and why I opened this thread):
- Mainboard: Supermicro A1SA7-2750F (order cancelled)
- ECC-RAM: 2x Intelligent Memory DIMM 16GB, DDR3L-1333, CL9, ECC (IMM2G72D3(L)DUD8AG-B15E) for a total of 32GB (order cancelled)

What I´ve ordered finally:
- CPU: Intel Xeon E5-1620v3
- CPU cooler: Noctua NH-U12DX-i4
- Mainboard: Supermicro X10SRH-CLN4F
- RAM: 2x 32GB DDR4 LRDIMM-2133 Samsung M386A4G40DM0-CPB
- HDDs: 6x WD60EFRX 6TB Drives (Raid-z2)
- Fans: 3x Be Quiet! Silent Wings 2 120mm, 2x Be Quiet! Silent Wings 2 140mm
- SATA cables: 5x SilverStone 4 Way SATA Power Splitter Cable
- PSU: Seasonic Snow Silent 1050
- Case: Lian Li D8000
- UPS: Eaton UPS Pro 850 DIN


After I read cyberjock´s PowerPoint, I realized, that for my purpose, I would need 12 GB RAM for FreeNAS + Plex Media Server. Leaving 20 GB RAM for Storage.
I have 4x 6 TB = 24 TB of raw storage now, that means my build is lacking 4 GB RAM.
Another thing is that cyberjock´s PowerPoint recommends 3, 5, or 9 disks for a Raid-z1 pool, for best performance. That´s less important to me, but if I would go for 5 Drives, I would definetely have to buy 2 additional 16 GB DIMMS.

And thats the problem:
One 16GB DIMM for the Avoton Mainboard costs me 337 Euros (just one seller in Germany sells this RAM)! 670 Euros for 32 GB. I am asking myself, was the board the wrong choice? Paying 1340 Euros for 64 GB of Registered ECC-RAM? This build will become very expensive once it is finished and I am limited in RAM and so am I in Storage. I contacted Supermicro before ordering and they told me, that this is the only tested 16GB DIMM for this board.
I went with this MB because it has a lot of SATA Ports, LSI 2116 Controller, IPMI, Support for ECC RAM, silent (no fan) and low power consumption compared to many of the Supermicro XEON Builds I considered too. The main reasons I didn´t go with another XEON Supermicro build was, the number of identical SATA ports on the board and the power consumption.

Maybe I was too shortsighted, but it´s still not too late to correct my orders.
I am still thinking about the Supermicro X9SRH-7F. I have one last question about this board and its SATA Ports. Can I combine all the 14 SATA Ports (2 SATA 3.0 ports, 4 SATA 2.0 ports, 8 SAS2 ports) to lets say, 2 Raid-z2 pools with 6 disks each? I mean the 6 disks of pool #1 connected to all different types of SATA ports? For example: 2 to SATA 3.0, 2 to SATA 2.0 and the remaining 2 hdd´s to the SAS2 ports?

 

[marbus90]

Cyberjock runs a 64TB payload array with 32GB as his plex homeserver. And yes, in case you want 64GB RAM, it's around 400EUR cheaper to order a Socket 2011-3 system since those support Registered DIMMs - the Avoton needs UDIMMs.
You can run that asrock on 4x8GB UDIMMs easily.

Also note that the Supermicro board you selected with the LSI controller requires special power connectors and is generally designed just to work in the http://www.supermicro.com/products/system/1U/5018/SSG-5018A-AR12L.cfm barebone system.

It's no problem to mix Intel SATA with LSI SAS HBA for FreeNAS use. Also mixing speeds is no issue since HDDs will barely saturate SATA-I aka 1.5Gb/s.
Utilizing a Supermicro 24/36 bay chassis might be the best option if you plan to upgrade anyway. With the expander backplane you won't even need one HBA per 8/16 drives but just 4 SAS/1x SFF-8087 ports to the backplane.

 

[moxxii]

@marbus90
Thanks for pointing me to the proprietary power connectors of the A1SA7-2750F, I´ve absolutely missed that. I could not find a PSU with compatible connectors for this Mainboard. Seems like I will have to look for another mainboard.
Any idea which of the new 2011-3 Socket Mainboards incl. which Xeon CPU is power saving? I found the E5-2608L to have the lowest TDP of the new Xeon Generation, but I am not sure if it will handle 2 simultaneous Plex Full HD Transcode´s.

The big Supermicro chassis with the expander backplanes look very good, but are no option for me as I am going to run it in my living room (I assume they are very loud). Their build-in PSU´s have very small fans and I think they run on high RPMs. There are no other rooms where I could place the NAS.

 

[marbus90]

You don't want low power CPUs. They only save power if the system is under full load. Since that's not the case most of the time, it'll regulate itself down. E5-1620 v3 is the best choice for a FreeNAS. It'll handle 4 transcoding sessions.

Also for 20 HDDs the 520w PSU doesn't cut it. I'd recommend something in the 800-1000w range. And have fun with cabling up 20 individual SATA HDDs - I've read in IRC about such projects, it's usually accompanied with much gotchas and damnits.

 

[Ericloewe]

For 20 HDDs I wouldn't go lower than 750W. 850W sounds like a good place to start.

 

[moxxii]

I absolutely believe you guys, I ordererd the PSU because of what "garycase" wrote in the second last post:
http://lime-technology.com/forum/index.php?topic=28064.10;wap2.

It wouldn´t be a problem to send the PSU back. Thought it would be a good choice because of the fanless design and the 12v single rail with 43 Ampere, what should accomplish the spinup of the drives. One drive running takes 5,3 Watts x 20 drives = 106 Watts. Leaving enough power for the rest of the system, I thought. But all my calculations are based on datasheets and what other people say, but maybe I forgot something?

Would you recommend a single or multi rail PSU? It seems like the Multirail PSU´s have all the hdd´s connected to a single 12V Rail, which has in most cases about 30 A. That will be a problem without staggered spinup of 20 drives (one WD Red needs about 1,7 A at spinup). The Seasonic P-Series XP2 860 has a single rail 12V with 71 A which is more than enough for 20 drives spinup. I hope its fan isn´t too loud, but it has a hybrid mode, which means the fan stands still when the temps are low.

 

[Ericloewe]

Anything by Seasonic will work well in the desired capacity.

One rail vs several is debatable, but you'll have trouble finding PSUs with more than one rail, these days.

 

[marbus90]

Good bigger PSUs usually don't utilize their fan until they're at ~50% load. If you buy a 800W PSU, that means that it eventually spins up the fan during the initial HDD spinup phase, but never after that.

 

[Kayman]

Also for 20 HDDs the 520w PSU doesn't cut it. I'd recommend something in the 800-1000w range. And have fun with cabling up 20 individual SATA HDDs - I've read in IRC about such projects, it's usually accompanied with much gotchas and damnits.

800-1000W PSU is overkill. I run a 20 HDD freenas box off a 650 watt psu and it's fine. I've measured how much it pulls from the wall. It doesn't even break 500 watts on startup with no staggered drive spin up. It pulls just over 300 watts during a scrub and about 280 idling.

As for the cabling. Use these http://www.pccasegear.com/index.php?main_page=product_info&cPath=19_104&products_id=27074. It has a big cap in it that gets pre charged before the drives spin up. So much less strain on the psu during start up and stable power once up and running.

 

[Kayman]

Good bigger PSUs usually don't utilize their fan until they're at ~50% load. If you buy a 800W PSU, that means that it eventually spins up the fan during the initial HDD spinup phase, but never after that.

Also I wouldn't worry about this. Your gonna need a lot of decent fans running constantly to keep a cluster of 20 HDDs cool and they'll be much louder then any quiet psu that switches it's fan off.

 

[moxxii]

The cable with the cap´s seems rewarding to me. I initially thought to buy this http://www.lian-li.com/en/dt_portfolio/bz-h06/
For a "manual" staggered spinup of 6 or 12 drives (if I buy two of them). I´m planning to run 3x 6-disk Raid-z2 pools in the future, but I don´t need them running all the time. Does this scenario work with FreeNAS? For example, use one Raid-z2 pool all the time and power on the other 6-drives when I want to manually start a backup? Or could there be issues if not all vdevs are present at startup?

Did you just do a test w/o staggered spinup or aren´t you generally using this option?

If you say that your 20 Drive Box uses 300 W when scrubbing, do you think my PSU from the first post could handle this? It should be at 60-70% load all the time, maybe a bit too much for the fanless psu. marbus99 and ericloewe advised against my Seasonic 520 psu. I am planning to buy the E5-1620 v3 and the Supermicro X10SRH-CLN4F with 64 GB of ECC RAM, not sure how much energy these components gonna need. But as my drives draw as much as yours, the average energy usage should be comparable.

I am currently considering one of these:
1. Seasonic Platinum-660 (SS-660XP2)
2. Seasonic Platinum-760 (SS-760XP2)
3. Seasonic Platinum-860 (SS-860XP2)
4. Seasonic Platinum Snow White 1050

 

[Kayman]

I run a E3 Xeon. A E5 would take a bit more power I would assume. I'd say a 750 watt psu would be plenty. Think of it this way a HDD will take a absolute max of 2A off the 12v line to spin up. Multiply that by 20 and your still under 500 watts and during drive spin up the computer hasn't even started booting witch means the load it's under is virtually zero.

I run all 20 Drives in 1 pool no staggered spin up.

 

[moxxii]

@Kayman
When I read post #9 in this forum: http://hardforum.com/showthread.php?t=1752253 I am wondering if it can be true? 105 watts when idle with this system:

Xeon E5-1620 3.6ghz Quad Core/HT idle's at 300mhz (socket 2011) using FreeBSD/ZFS v28 which controls the throttle.
8x 2TB WD Red Drives
Supermicro motherboard
4 80mm cooling fans (Supermicro chassis stuff)
32GB Registered DDR3-1600 RAM
Intel XF-SR 10gb 850nm 10gb/e pci-e 2.0 8x card. Fiber optic.

total draw from APC-SmartUps 1500 is between 98-105 watts at idle.

I assume that one of your drives consumes ~3.3 watts when idle. So if we subtract 12x 3.3 watts to match the posters 8 drives your system should idle around 240 watts. This seems high compared to his system. Too bad that the guy didn´t give more information about his MB.

 

[marbus90]

For safe measures, calculate 30w per HDD: roughly 2A@12V and 2A@5V.

 

[Mguilicutty]

Just for reference, I fired up a bunch of devices on Plex. One xbox 360, two android tablets, two PCs and an ipad all watching different movies, no stuttering on any of them. My system is in my sig. According to my UPS, the system is pulling between 190 and 205 watts under this load, while also acting as an ISCSI target for a few HyperV machines which aren't doing much at the moment. I attribute a good portion of this load to the fans in the chassis, of which there are six, very high powered and almost definitely overkill. It is a rack mounted beast intended to cool (15) 15k drives and a dual cpu mobo. Were I to build a new NAS, I would use an E5-16xx versus the E5-26xx, I just had this one handy. The Plex jail resides on the Samsung SSDs, the media on the raidz2.

The post from above where you mentioned the hardforum build, that 10Gb card itself probably draws in the neighborhood of 30w. My case is utilizing all three power supplies, probably also increasing the draw from the wall socket.

 

[moxxii]

I have another question regarding future SAS3 compatibility.
Do you guys think that this backplane http://www.lian-li.com/en/dt_portfolio/bp2sata/
does passthrough the SAS3 signal (12Gbps speed)? The specs say "The SATA back panel doesn’t have a RAID or SATA chip on the PCB. It requires a motherboard which has RAID onboard and SATA capabilities to enable the hot swap and RAID features." So maybe it doesn´t alter the signal that comes from the Motherboard? Or does SAS3 always require a special sort of backplane for HotSwap capability?

I wrote an Email to the Lian Li support, but got no answer until now.

 

[Ericloewe]

I have another question regarding future SAS3 compatibility.
Do you guys think that this backplane http://www.lian-li.com/en/dt_portfolio/bp2sata/
does passthrough the SAS3 signal (12Gbps speed)? The specs say "The SATA back panel doesn’t have a RAID or SATA chip on the PCB. It requires a motherboard which has RAID onboard and SATA capabilities to enable the hot swap and RAID features." So maybe it doesn´t alter the signal that comes from the Motherboard? Or does SAS3 always require a special sort of backplane for HotSwap capability?

I wrote an Email to the Lian Li support, but got no answer until now.

Don't expect SAS support. SATA should be fine. That said, it'll probably work with single-path SAS.

Do note that the backplane is likely not to work for hot-swapping. It doesn't have any capacitors to help keep inrush current in check, nor are the pre-charge traces longer than the rest of them. That's a recipe for hot plugging not to work.

 

[marbus90]

it's possible that SAS drives don't even plug into the SATA connectors. it needs to be a backplane with SAS connectors on the disk side for that.

 

[moxxii]

Maybe you guys can help me out. I finished putting together my system from the first post. When booting I hear 3x fast short beeps and 1x long beep. I am not 100% sure because the first beeps come so fast. I managed to boot into bios, my memory and CPU are shown with the correct specs there.
I switched the DIMMs from Slot DIMMA1 to DIMMA3 and the other way round. And also with just one DIMM inserted in DIMMA1. But still the same beeps.

Do you think that the E5-1600v3 Series is not compatible with LRDIMMs? I read something in the Supermicro Forum: http://www.supermicro.com/support/faqs/faq.cfm?faq=20536

I wrote an Email to the Supermicro support, but did not get an answer so far.

 

[Ericloewe]

Is the system working? If so, don't worry.
The IPMI log should have any important issues.
Supermicro boards can be very beepy.