BUILD Request expert opinion on build plan

[mailman757]

Hello girls and boys,
thank you for for your attention. In the following, I'd like to sum up my build plan. This plan is the result of quite some research on the subject. I hope I have covered most bases correctly and would very much like expert opinions and feedback. Thank you for your time and effort.

Reason for build
The new build is intended to replace a NG ReadyNAS NV+ (sparc)/4x2TB. It is running fine, has served reliably for 5 years. I am approaching maximum capacity though, and cannot expand further; the sparc driven NAS does not allow for HDD of more than 2TB in capacity.

I am exploring the FreeNas build, as it has distinct advantages over a off-the-shelf NAS (QNAP TVS-871, e.g.).
- standard and replaceable components
- ZFS file system
- not restricted by product end-of.life
- higher value for money

The disadvantage is, that I am a server newbie, and would like a stable system and as little maintenance as possible

Spec of intended utilization
- Minimum maintenance requirement - I am not an IT professional
- multimedia storage and local network transcode streaming of HD content
- business file storage and serving
- vpn limited access
- High availability not immediately required
- central storage for all group users (<10)
- storage of critical data
- data integrity has high priority

Intended software
FreeNAS latest stable release

Intended Hardware; stage 1 (stage2):

1. Motherboard: Supermicro x11SAE
2. CPU: Intel Xeon 1225v5; air-cooled by Dynatron R17
3. RAM: Crucial 16GBx2 DDR4-2400 EUDIMM (16GBx4), unbuffered ECC
4. HDD: ST4000DM000 4x (8x) in 1 RaidZ3 array; Inter-tec VT-106 TL Hot Swap
5. PSU: 850 Watt Corsair HXi Series Modular 80+ Platinum
6. Add-On cards: Addonics M2 PCIe SSD adapter w/64GB Plextor m6G-2280 M.2 2280 SATAIII MLC
7. UPS: APC Smart-UPS 1500VA LCD 230V
8. Enclosure: Inter-tec 19" IPC-9008 5U
9. Router bintec RS353jw / 1000 Mbps
10. Crash Plan: critical data local back-up to RadyNAS NV+ and off-site to Backblaze

Choices explained:
1. MB offers 8 SATA III ports, thus eliminating the need for SATA expansion cards
2. MB can run 64GB RAM, which would be sufficient for the intended 8x6TB HDD
3. MB and CPU are server grade
4. CPU has ample performance to serve 2 simultaneous trans-coding HD streams
5. HDD can be purchased w/ 4 years of warranty at reasonable cost, are low rpm, and have adequate reliability, provided they survive a thorough burning-in (https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/)
6. Addonics card and m.2 are for swift system boot
7. UPS sufficient autonomy (5-15 min) for safe shutdown
8. Enclosure is reasonably priced, has room for all the intended HDD in hot-swap config, and can be tower or rack-mounted
9. router has been in service for 12 months, and is running well
10. crashplan seems economically sound, balanced and covers complete loss of local equipment

 

[Ericloewe]

So, thoughts on the hardware:

Motherboard: Supermicro x11SAE

Consider a Supermicro X11SSM-F/X11SSH-F instead, as those are real server boards.

Also, there might be some slight growing pains with that new stuff, so consider the proven X10 stuff instead. Even X9 is fine.

air-cooled by Dynatron R17

Stock cooler is fine.

Add-On cards: Addonics M2 PCIe SSD adapter w/64GB Plextor m6G-2280 M.2 2280 SATAIII MLC

What for? PCI-e SSD as the boot device? That is overkill.
Also, you can't use a SATA M.2 device in a PCI-e only M.2 slot (which that one might be).

• RAM: Crucial 16GBx2 DDR4-2400 EUDIMM (16GBx4), unbuffered ECC

Start with 32GB, add more later if/when needed.
Also, try to get something from the QVL.

SU: 850 Watt Corsair HXi Series Modular 80+ Platinum

I'd prefer a Seasonic Platinum 850.

 

[mailman757]

On MB: you are right. I missed, that my choice was a WS board. Concerning X11, I looked for 8 SATA ports and the possibility to install minimum 64GB of Ram. Also I found the v5 Xeon performs very nicely...on paper anyway.
It welcome , suggestions with those specs and X10/9 chipset.

Cooler: the offer i looked at was attractive, but as WOC, so I had to chose something. So went with this one, as it is installed on our servers

PCI-e: i dug a little deeper, sicne my write up. I read about the possibility of having two ssd in raid 1 (superDOM). I have since changed the specs to a PCIe card with two mSATA sockets, iso one m.2

Ram config: the board does not accept registered ecc. At the moment, it seems to be difficult to find 2x16 GB sticks. I would have gone for 2133 otherwise.As you rightly suggested, starting off with 32G is the plan. The number in brackets is for stage two.

Can you elaborate on the choice of PSU?

Thank you for your help and opinion

 

[solarisguy]

@mailman757, I would like to talk to you about your disks.

Why did you choose Seagate, as opposed to Western Digital Red, Toshiba or HGST ?

Could you please elaborate on HDD: ST4000DM000 4x (8x) in 1 RaidZ3 array ?

P.S.
What is your estimated power consumption?

 

[mailman757]

check out this report: https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/

The ST4000 is the most frequently used drive by them, and it still has a decent reliability - especially considering the unit cost. After thorough burn-in, I would hope to have weeded out any bad drives and install replacement on standard warranty. Also, the low unit cost allows me to have 2 cold standby drives ready to go (which of course, should also be burnt in). Even if a drive fails after warranty end, you get 2 ST4000 for the price of 1 WD pro 6TB. The WD red pro 4TB is almost 100€ more expensive.

I chose 4TB iso 6TB, because they will be give me the best economy for my first stage of my build (4 drives). And by the time the final build (8 drives) gets filled to capacity (I estimate approx. 5 years), I hope, that 6TB drives will have become more affordable. After 5 years, I would consider exchanging drives anyway.

I would have chosen WD Red Pro, as they have a 5 year warranty in my country. I opted against them, as I think I cannot justify the cost for that added benefit. I will go for the cold stby solution.

Why did you choose Seagate, as opposed to Western Digital Red, Toshiba or HGST ?

Could you please elaborate on HDD: ST4000DM000 4x (8x) in 1 RaidZ3 array ?
Which raid config I will eventually use, remains to be seen. This build has reliability and integrity as priority. So, it will be a config, which has properities similar to Raid10.


What is your estimated power consumption?
Can't be certain, really. According to my calculations, I should not exceed 670W even at peak (all drives spinning up simultaneously). As long as I don't have to change my plans for the 80W TDP CPU, I expect to be fine with the 850W PSU. There will even be enough overhead for 8 hot swap frames, with 1 fan each.

 

[mailman757]

Sorry, but my reply got scrambled.....
better below:

check out this report: https://www.backblaze.com/blog/hard-drive-reliability-q3-2015/

The ST4000 is the most frequently used drive by them, and it still has a decent reliability - especially considering the unit cost. After thorough burn-in, I would hope to have weeded out any bad drives and install replacement on standard warranty. Also, the low unit cost allows me to have 2 cold standby drives ready to go (which of course, should also be burnt in). Even if a drive fails after warranty end, you get 2 ST4000 for the price of 1 WD pro 6TB. The WD red pro 4TB is almost 100€ more expensive.

I chose 4TB iso 6TB, because they will be give me the best economy for my first stage of my build (4 drives). And by the time the final build (8 drives) gets filled to capacity (I estimate approx. 5 years), I hope, that 6TB drives will have become more affordable. After 5 years, I would consider exchanging drives anyway.

I would have chosen WD Red Pro, as they have a 5 year warranty in my country. I opted against them, as I think I cannot justify the cost for that added benefit. I will go for the cold stby solution.

Why did you choose Seagate, as opposed to Western Digital Red, Toshiba or HGST ?

Could you please elaborate on HDD: ST4000DM000 4x (8x) in 1 RaidZ3 array ?
Which raid config I will eventually use, remains to be seen. This build has reliability and integrity as priority. So, it will be a config, which has properities similar to Raid10.

 

[solarisguy]

[...] What is your estimated power consumption?
Can't be certain, really. According to my calculations, I should not exceed 670W even at peak (all drives spinning up simultaneously). As long as I don't have to change my plans for the 80W TDP CPU, I expect to be fine with the 850W PSU. There will even be enough overhead for 8 hot swap frames, with 1 fan each.

My similar CPU (Xeon E3-1220 v3 @ 3.10GHz), 16GB of RAM, 2 disks 2.5", 8 disks 3.5", and 5 case fans consume 100W. At startup, less than 200W.

 

[mailman757]

Interesting. Did you measure this on a live system?

I had to go by values provided by the manufacturers and conservative guesstimates.
According to my calculations, the drives alone have a potential peak draw of 280W.

here's my sum of estimates for peak loading:
1225 Xeon 80W
SM MB 30W
8 x HDD @35W ea. 280W
8 x 4" HDD Fan @5W ea. 40W
4 x RAM @ 6W ea. 24W
M2 PCIe SSD adapter card 10W
2 x m.2 SATA sticks 12W
3 x 14" fans 60W
TOTAL estimated 536W
25% margin 134W
PSU Taget Rating 670W

I have estimated idle loading to be around 160-200W.
If I find that I need increased airflow I could mount an additional 3 fans, adding another 60W to the sum - 730W, i.e. So the next available rating would be 750W. Another 25€ and a Platinum 850 PSU, should keep decent efficiency to 20% of rated output - 170W, i.e.

 

[Bidule0hm]

650 W is more than enough for this build ;)

 

[solarisguy]

Interesting. Did you measure this on a live system?

I had to go by values provided by the manufacturers and conservative guesstimates.
According to my calculations, the drives alone have a potential peak draw of 280W.

here's my sum of estimates for peak loading:
1225 Xeon 80W
SM MB 30W
8 x HDD @35W ea. 280W
8 x 4" HDD Fan @5W ea. 40W
4 x RAM @ 6W ea. 24W
M2 PCIe SSD adapter card 10W
2 x m.2 SATA sticks 12W
3 x 14" fans 60W
TOTAL estimated 536W
25% margin 134W
PSU Taget Rating 670W

I have estimated idle loading to be around 160-200W.
If I find that I need increased airflow I could mount an additional 3 fans, adding another 60W to the sum - 730W, i.e. So the next available rating would be 750W. Another 25€ and a Platinum 850 PSU, should keep decent efficiency to 20% of rated output - 170W, i.e.

Actual measurement, using this device https://www.conrad.de/de/belkin-f7c005-conserve-insight-ekm-104854.html

I have moved to disks that use a little bit less power. No, not WD Green, I am using WD Red. That had two consequences: eliminated HDD fans and I was able to use fans that use less power themselves (less than 2W per fan, with 5 fans that is only 10W in total). That saved me 90W => 536W - 90W = 446W

Ultimately, I settled on 430W power supply. And I was able to use it with 14 (fourteen) 3.5" disks, when I was copying data from older disks to the new ones. Read on to learn how :D

Above that was about me, but I think you miscalculated your ST4000DM000 power needs. You should redo your calculations. I have looked at the detailed ST4000DM000 specifications listed at
http://www.seagate.com/www-content/product-content/desktop-hdd-fam/en-us/docs/100782401b.pdf
The startup power for each disk is only 2A * 12V = 24W. That means 192W (8 * 24W), instead of 280W, if all of them were starting at the same time. In my HBA (9211-8i), I was able to configure BIOS to have one disk starting every 2 seconds. Can you do something similar in your BIOS?

So 536W - 88W = 448W.

With 25% margin 448W * 1.25 = 560W.

Going back to your disks, at the time they are spinning up, the remainder of your system is not consuming their maximum power. Most notably the CPU and memory do not. Thus, in my opinion, the 25% safety margin is too much. I would use any quiet, quality power supply in the 500-575W range.

Please notice that when ST4000DM000 are operating they consume only 5.6W. Thus, since 8 * 5.6W = 44.8W, havining everything normally operational requires 300.8W = 536W - 280W + 44.8W

A 500W power supply would be more than enough, and a 550W power supply would give you a piece of mind for a possible expansion.