New Build September 2016

[rovan]

Hi All,

I've been planning this build for a little while now, If I could get a once over by some more experienced users that'd be great.

CPU: Intel Xeon E5 1620v3 3.5GHz Socket 2011-3
HS: Noctua NH-U12DX i4 (120mm) CPU heatsink
RAM: 32GB (2 x 16GB) Crucial DDR4 2133MHz ECC Buffered
MB: Supermicro X10SRi-F Socket 2011-3
SSD: 120GB SSD Mirror or USB3.0 Flash memory stick
HDD: 6x WD Red Pro 8TB 7200RPM 128MB cache RAIDZ2
CASE: Antec P182 ATX
PSU: Antec 650w

Drive bays have rubber/silicon screw mounts, there's 3 120mm fans in the case.
1 in the bottom HDD bay which houses 4 drives.
1 in the middle HDD bay which houses 2 drives. I'll add an additional 120mm fan to the middle HDD bay.
1 back exhaust fan 120mm and 1 top exhaust fan also 120mm. These come standard with the case.

I have also looked at the X11 boards there is basically not a lot of cost differential between the 2011-3 and 1151 builds ... I could save maybe $200 on the CPU going with an i3 but I've already decided to go quad core. The extra PCI Express lanes on the Socket 2011-3 and expandable memory (Past 64GB) are attractive to me ... If i want to add some NVMe drives for more performance I can do so utilizing the PCI Express slots.

Thoughts and/or comments welcome. Thanks in advance.

 

[Ericloewe]

RAM: 32GB (2 x 16GB) Crucial DDR4 2133MHz ECC Buffered

Do you mean unbuffered or registered?

PSU: Antec 650w

Try to get something better. Seasonic X-650 is an excellent PSU.

 

[Stux]

1620v3, check out the v4 if there is not much price differences.

1650v4 is the 6 core version, but twice the price, just FYI.

If you go for a v4 ( broadwell) the. You should get ddr 2400 ram instead of 2133. I found this made no cost difference.

Ps: your build is very similar to mine.

Wd red pros run significantly hotter than the non pros.

 

[rovan]

Thanks for the replies!

@Ericloewe I mean registered (Isn't buffered the same?) I believe UDIMMs aren't supported on this board.

@Stux Yeah I saw your specs earlier :) I wouldn't mind getting a hex core, but I'm not convinced that I'll need it. I can pick up the 1650v4 for about 1k AUD. it's still pretty new. around 820 for the 1650v3. I'm investigating the the 1620v4 and some compatible DDR4-2400 ECC memory (Probably Crucial).

I'm pretty happy with the WD Red's i've had one 3TB fail on me (not catastrophically so I was able to copy some new data off it) The others are still humming along (A couple 4TB drives also WD Red). This will be the first build with WD Red Pro's. They also have a 5 year warranty as apposed to 3 with the WD Red.

 

[Ericloewe]

(Isn't buffered the same?) I

They're not called buffered. Registered is also less prone to unfortunate misunderstandings.

 

[Stux]

Thanks for the replies!

@Ericloewe I mean registered (Isn't buffered the same?) I believe UDIMMs aren't supported on this board.

@Stux Yeah I saw your specs earlier :) I wouldn't mind getting a hex core, but I'm not convinced that I'll need it. I can pick up the 1650v4 for about 1k AUD. it's still pretty new. around 820 for the 1650v3. I'm investigating the the 1620v4 and some compatible DDR4-2400 ECC memory (Probably Crucial).

I'm pretty happy with the WD Red's i've had one 3TB fail on me (not catastrophically so I was able to copy some new data off it) The others are still humming along (A couple 4TB drives also WD Red). This will be the first build with WD Red Pro's. They also have a 5 year warranty as apposed to 3 with the WD Red.

Not suggesting you purchase from this seller, but:
http://www.ebay.com.au/itm/Intel-Xe...155265?hash=item41b0dd1d81:g:3A4AAOSwdzVXtMOE

That's about how much mine cost, and it actually came from superbiiz.com,

http://www.superbiiz.com/detail.php?name=E51650V4BX

I have a set of 5 3TB WD Red Pros... they're fine...

 

[rovan]

Had a look on Kogan and found one 1650v4 for around $804 AUD. Interesting.

Also I've already got the case and PSU I will replace it if it does not do the job.

Any definite advantages going hex core as apposed to quad?

 

[Stux]

Btw, I wasn't suggesting you should, just that you can ;)

 

[rovan]

Well Stux, I recieved my 1650v4 around christmas :D I've got all the drives just need to decide whether to get a new case.

 

[rovan]

Finished setting it up last night thanks everyone :)

 

[rovan]

Is 8GB enough for the boot drive? there's not a huge difference between the 8 and 16GB USB3.0 drives .. I was thinking of setting up a mirror for the o/s.

 

[Stux]

8GB is fine. 16GB is plenty.

The important thing is to mirror the boot drive if you're using USB flash disks.

Boot Mirroring can be done from the installer and from the system GUI too. (See the manual)

 

[rovan]

sweet :) thanks. Also is there a stress test for memory allocation? I will monitor it over the next couple weeks and see how it goes. Anything to look for in particular to decide if more memory is required?

Update: Found this post here. I'll check it out tonight ^_^, open to other suggestions :)

 

[rovan]

Completed a number of sets of Memtest. RAM tests passed, Disk tests are still ongoing. My Noctua CPU fan cycles between 300RPM and Max. I haven't found BMC thresholds settings that work yet. CPU temp sits around 20-22c idle (I think thats not bad).

The upper FAN NR NC values in the BMC are like 22500 RPM+ lol!

I tested some more sane values (lower) 0 100 200 (upper) 1300 1400 1500 for FAN1. 100 200 300 didn't seem to work. Currently they are 000 000 200.
My case fans sit between 1200 and 1300RPM (Without LNA), I'd like to get these down to a more relaxed level also. I might end up installing the LNA.

I'll focus on getting the CPU FAN1 speed working for now ...

 

[Stux]

Are you running a fan control script?

https://forums.freenas.org/index.php?threads/script-hybrid-cpu-hd-fan-zone-controller.46159/

 

[rovan]

I'll try it tonight :) thanks Stux. I've been reading your post already ^^ any suggestions for the fan thresholds ?
NF-F12 (1500RPM) (included with the NH-U12DX i4 cooler)
FAN1 0 0 200 1400 1500 1600
NF-S12 (1200RPM) cooling the top bay of hard drives
FANA 0 0 200 1100 1200 1300
NF-P12 (1300RPM) cools the bottom bay of hard drives.
FAN4 0 0 200 1200 1300 1400

 

[Stux]

Iirc you want the upper and lower thresholds to be impossible to reach in normal situations.

Your 1500RPM fan (if that's what they're specified at) may reach 1600 and even 1700 rpm

 

[rovan]

I don't get how it calculates the value for upper threshold .. it'd have to be in a wind tunnel with air forcing the fan faster or something? so it's technically never going to happen... so the upper values that where in there originally (22k/25k something like that) was fine?

 

[Stux]

I don't get how it calculates the value for upper threshold .. it'd have to be in a wind tunnel with air forcing the fan faster or something? so it's technically never going to happen... so the upper values that where in there originally (22k/25k something like that) was fine?

Yes. Basically if it gets there it indicates a fan fault

 

[rovan]

OK so after reading the ipmitool info properly I realized i set the NC NR thresholds around the wrong way for the upper values.

Also I had the CPU fan plugged into one of the system fan headers, apparently the FANA header is for the CPU, it's also the furthest away? (Page 6 on the manual ) Fan RPMs are now stable on the Optimal setting :D NAS is also saturating gigabit, and I'm out of ports on my switch (ffs) need to get a new switch ... or maybe some 10GigE cards :D.