First NAS build: Avoton 8 Core and 24TB WD Red

[blahhumbug]

Hi,

I'm brand new to the FreeNAS community and NASes in general. I've been planning to build one for years, but was just never happy with the part combinations I could find, the price points, and the total noise levels and power consumption levels of the parts I would try to combine. This year, I finally came up with a part list that makes me really excited to build my first NAS, so I just pulled the trigger and ordered everything.

Asrock C2750D4I (Avoton 8-core)
4 x Kingston 8GB DDR3-1600 ECC Unbuffered (KVR16E11K4/32)
8 x Western Digital Red 3TB (WD30EFRX) - raid-z2
Silverstone DS380B Mini ITX Tower Case
Silverstone Strider Gold 450W SFX Power Supply (ST45SF-G)
Sandisk Cruzer Fit 32GB USB 2.0 Flash Drive (for FreeNAS)
Cyberpower CP850PFCLCD UPS (850VA/510W)

I recognize that some of the parts are a little overkill for the components, particularly the PSU (450 watt) and the UPS (510 watt). I estimate that the system idle power will be around 75w and 120w at full load. However, I have a secondary goal of having the quietest system I possibly can. It will be sitting only 4 feet away from my main workstation which is also very silent, so I'm trying to avoid adding more noise to the room as much as possible. The case requires an SFX power supply, so that also removed a lot of lower power options. I don't mind the UPS being oversized though as that just increases my uptime during a powerloss to somewhere around 20-30 minutes and also gives me room to plug the wifi/switch and cable-modem into it as well.

I chose the 8-core Avoton instead of the 4-core because I plan to run a plexserver and a few other services on the machine. I also wanted to leave a little bit of headroom for future growth and additional services. I'd like this NAS build to last me at least 5 years.

I really wanted to pick up 2 x 16GB of ECC unbuffered memory so that I had room to add 2 more dimms for 64B total in the future. However, I couldn't find anyone selling that size with unbuffered ecc. I saw a lot of forum posts here with people using this combination of Kingston memory, so I decided to just stick with that. I think 32GB of mem should be fine in this machine for the next 5 years as I do not have plans to do deduplication.

I made the decision to go with WD Red drives over the Red Pro drives due to both cost and and noise. The Pros were significantly louder during seek in WDs published specs (34dba vs 24dba). I chose the Red drives over other similar 3TB options mostly because of their idle power consumption (~5w) and noise levels.

I chose to fill out all 8 hotplug slots of the case with 3TB drives rather than going with a smaller number of 6TB drives. I decided I liked the idea of being able to slowly upgrade 3TB drives to 6TB drives if/as they fail. And if at anypoint I start getting too full, I can pull the trigger and upgrade them all to 6TB and grow the pool. Additionally, the pricing point of 8x3TB vs 4x6TB was better.

I hope to get the parts this week, and will update this post with some photos of the system and benchmark numbers once I've got it up and running.

Cheers!

 

[blahhumbug]

Oh, and thanks to Cyberjock! His noob guide was extremely useful and heavily influenced my component decision making process.

 

[BigDave]

Welcome to the forums. Your efforts to read first, has saved you alot of time already;)
Test and stress your components before loading drives with data.

 

[DrKK]

I'd like to ask a dumb question.

The original poster mentioned 8xHDD. This board only has 6 ports on it that we'd recommend using. The other 6 Marvell ports are not reliable, last I knew. Is the OP aware of this limitation? Or am I misinformed?

 

[Ericloewe]

I'd like to ask a dumb question.

The original poster mentioned 8xHDD. This board only has 6 ports on it that we'd recommend using. The other 6 Marvell ports are not reliable, last I knew. Is the OP aware of this limitation? Or am I misinformed?

You're not misinformed. According to Cyberjock, ASRock's been hacking away (or more likely poking Marvell with a stick made of future products that can still switch to LSI SAS solutions) to try and make those controllers more reliable. They have released a RAID-free firmware that apparently provides a decent speed boost and marginally improves stability. More than that is just a promise.

You'd think that after 5 or 6 years, there would be some kind of reliable SATA3 controller around that doesn't need to be attached to an Intel CPU (I guess AMD's controllers are decent too, but I haven't used them).

 

[blahhumbug]

I've been following the issues with the Marvell ports. I plan to use the March firmware update for the Marvel SE9230 and do some stress testing under some various configurations listed below. If I end up dropping down to only 6 drives and all Intel sata ports I'll just throw the extra 2 drives in my workstation. Here are the configurations I want to test out in order:

2xIntel 6gbps, 6xMarvell
6xIntel, 2xMarvelSE9172
2xIntel 6gbps, 2xIntel 3gbs, 4xMarvel SE9230
6xIntel

 

[DrKK]

I've been following the issues with the Marvell ports. I plan to use the March firmware update for the Marvel SE9230 and do some stress testing under some various configurations listed below. If I end up dropping down to only 6 drives and all Intel sata ports I'll just throw the extra 2 drives in my workstation. Here are the configurations I want to test out in order:

2xIntel 6gbps, 6xMarvell
6xIntel, 2xMarvelSE9172
2xIntel 6gbps, 2xIntel 3gbs, 4xMarvel SE9230
6xIntel

Out of curiosity, can you tell us why you would put the 6xMarvell 2xIntel configuration in your pole position, as opposed to vice versa?

 

[blahhumbug]

Only because all the Marvell ports are 6bps, but only 2 of the Intel ports are 6gbps. I've seen a few reports of good stability with the latest Marvell firmware update, so thought I'd give it a shot, since at least they finally responded with an attempted fix.

 

[blairjj]

I have this board and love it. It is so quiet that I hardly even realize it is running. Use it for Plex, Transmission, RageTV, Minecraft (for kids) and Calibre. I would use the same setup again as it was painfree and very dependable:

Bitfenix Prodigy
Cosair 600 W PS
ASROCK C2750D4I
Crucial 16GB Kit (8GBx2) DDR3L 1600MT/s (PC3-12800) DR x8 ECC
(5) Toshiba 2TB 7200 RPM SATA 6Gb/s 3.5" Internal Hard Drive DT01ACA200
Dual PNY 16GB USB 2.0 Flash Drive P-FDU16G/APPMT-EF

 

[cyberjock]

Only because all the Marvell ports are 6bps, but only 2 of the Intel ports are 6gbps. I've seen a few reports of good stability with the latest Marvell firmware update, so thought I'd give it a shot, since at least they finally responded with an attempted fix.

I've heard of no such reports.

And even if the ports were 1.5Gbps, they still wouldn't be a bottleneck for your hard drives (your 1Gb LAN connection will most certainly be). Hence the question DrKK was asking. Put things in perspective and go with the stuff that is more reliable. ;)

 

[blahhumbug]

Thanks for the feedback, it's always appreciated.

I've been having trouble finding more detailed info on the problems and whether they affect both the Marvel SE9230 and SE9172 controllers, or only the SE9230, which is why I planned to do some in-house stress testing.

While I'll eventually use this as a production system, putting it together is also part hobby and for-fun, so I don't mind spending a few weeks playing around with some different combinations and stress testing. If I can get 8 drives stable by only avoiding the SE9230, then great! Otherwise, I'll just go with 6 drives on only the Intel ports.

 

[cyberjock]

I don't have extensive information on the SE9172, but AFAIK the problems are on the SE9230 only.

 

[blahhumbug]

Just wanted to thank everyone who replied to the thread. After running weeks of torture tests and RMAing one bad WD Red that was failing smart long tests, I believe I have a stable C2750D4I system using 6 Intel Sata ports and 2 Marvel SE9172 ports.

I'm not super happy with the HD cooling in the Silverstone DS380B case with the stock fans. Under normal usage all drives stay below 40c, but if I stress the system I can get three middle bays to go above 40c. Since silence is still a goal of mine, I'm going to attempt to create an air barrier between the drive cage and the left side of the case to force more of the intake fan air through the tiny drive cage air holes.

 

[DrKK]

I don't suggest you allow NAS drives to regularly go above 40C. I'd shoot for low 30's, under stress.
Of course, that generally means louder fans. There is a slight bit of evidence that you actually don't want to run them too cool either. Some of the studies, particularly the famous Google one, indicated that low to low-mid 30's was the ideal for long term reliability.


My hard drives have never been above 32C in the FreeNAS. Never once.

 

[blahhumbug]

My target goal is to get all drives at 35C or lower under full system load. Currently, I'm getting large variations between drive temperatures in the different bay slots. Bay0 is the bottom of the case, bay7 is the top.

bay7: 29 C
bay6: 33 C
bay5: 31 C
bay4: 30 C
bay3: 35 C
bay2: 38 C
bay1: 38 C
bay0: 36 C

Bay's 4-7 already meet my goals under full system load, but bays 0->3 either cross 40c under load or come close. At first I thought maybe there were issues with the bottom intake fan closest to these bays, but BIOS reports 1100RPM and I can visually see it is spinning. I think the design of the case and drive cage just doesn't lend itself well to good airflow through the drives. My first plan is to tape some clear pastic between the side of the case and the drive bay to force all intake air through the drive bays as much as I can and see what affect that has on drive temperatures befire I load it with production data.

Edit: Forgot to mention that removing the side dust filter on the case improves the intake airflow enough to also keep the drives all under 40C under full load. However, I'd rather keep the dust-filter on. But if faster/louder fans is what I have to do to keep the drives at good temperatures, it's what I'll do.

 

[DrKK]

It doesn't take much to cool drives. You could "rig" something up. Perhaps a very low RPM 200mm <20dB noise (you won't hear anything) could be rigged up (a couple of zip ties in the corner, cinch them up to the cages, boom done) against the affected part of the cage, pushing air over the drives.

Something like this which is like $16 after rebate today.

There are also directed cooler things that I've seen around.

 

[Ericloewe]

It doesn't take much to cool drives. You could "rig" something up. Perhaps a very low RPM 200mm <20dB noise (you won't hear anything) could be rigged up (a couple of zip ties in the corner, cinch them up to the cages, boom done) against the affected part of the cage, pushing air over the drives.

Something like this which is like $16 after rebate today.

There are also directed cooler things that I've seen around.

I have a couple of those, courtesy of my HAF X cases. Be warned that they do not take kindly to dusty environments. They're just your basic average sleeve bearing fan - unfortunately, they're also the best option around. The 200mm fans Noctua promised for late last year haven't materialized yet, which is a pity, because nobody seems to make 200mmx30mm ball bearing fans.

 

[Jailer]

I think the design of the case and drive cage just doesn't lend itself well to good airflow through the drives. My first plan is to tape some clear pastic between the side of the case and the drive bay to force all intake air through the drive bays as much as I can and see what affect that has on drive temperatures befire I load it with production data.

Edit: Forgot to mention that removing the side dust filter on the case improves the intake airflow enough to also keep the drives all under 40C under full load. However, I'd rather keep the dust-filter on. But if faster/louder fans is what I have to do to keep the drives at good temperatures, it's what I'll do.

Don't forget to address your exhaust as well. You want balanced air flow through the entire case and you need enough exhaust to move what the intakes are blowing in.

As far as isolating where the intake air is going, make sure you seal every escape point in front of those drives or your effort will be fruitless. Air flows the path of least resistance and if you give it any easy path that's where it's going to go.

I just looked at some pictures of that case on line. You're going to have your work cut out for you trying to get some cooling air across those drives. Nice case but what a terrible setup as far as cooling goes.