Planning this X11 build

[Rational]

Hi Guys,
Visited here 4-6 months ago and read like a madman for days and days. Planned out a build but am only now getting back to it.
Wanted to run it by you to validate all looks good and get any pointers on improvement or changes since my last intense study.

My goal is to keep idle wattage as low as possible (20-50watts max) but still have lots of power for transcoding when I need it. Plan to run PLEX and get back into running virtual machines and other newer technologies like docker potentially if that's an option. Mostly will be used to consolidate 10 years of hard drives onto a truly reliable and redundant system. Host mostly family picture and videos, music, movies, my music software and samples, and run PC backups via Acronis and time machine for 5 PCs and 3 laptops.

Was considering a Synology DS916+ (8GB) for $599, but it's underpowered for transcoding and for $600 I could get more bang for my buck going with FreeNAS I believe.


So here's my build:
Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSM-F-O https://www.amazon.com/gp/product/B018AX449G/?tag=ozlp-20
or
SM X11SAE-M-O https://www.amazon.com/gp/product/B018AX44Q4/?tag=ozlp-20
Anything better? No longer remember exactly why I chose these. Think I wanted a micro ATX build at the time. And the second one had 8 SATA connectors. I wanted a board I could throw a GTX1070 into someday should I want to watch movies in 4k@60 right from the NAS if that's an option. I know QNAP is offering onboard video now and I think its a great competitive advantage over Synology. Plus throw in my own pro audio soundcard perhaps in the PCI slot. Doesn't necessarily have to be a micro build anymore.

Intel i5-6400T to keep TDP as low as possible. https://www.amazon.com/gp/product/B01BH4XX74/?tag=ozlp-20
Love that it is Skylake. Handles ECC and has onboard video. Though I've since read that the "T" models are artifically capping performance and that other models run low power at idle as well. So is the non "T" model of this i5 a better choice? I think only the second mobo above handles the i5 CPUs? That's what the Amazon description says anyway.

16 or 32gb of ECC ram:
16gb Samsung $97 https://www.amazon.com/gp/product/B018CZWVVU/?tag=ozlp-20
32gb Crucial https://www.amazon.com/gp/product/B00KUSMULY/?tag=ozlp-20

4x8TB WD Red Drives $250 each (Taken out of WD external enclosures) https://www.amazon.com/gp/product/B01B6BN0Q2/?tag=ozlp-20
Or I may buy the WD dual drive NASs and take the drives out of them. That gives me two cheap alternate NAS units for $50 each that I can put old drives in. Not a bad deal?
https://www.amazon.com/gp/product/B01B6BN1CU/?tag=ozlp-20

Passive CPU cooler NoFan 95 https://www.amazon.com/gp/product/B009OXB1ZM/?tag=ozlp-20

PSU: Seasonic Fanless 520 Watt Platinum. https://www.amazon.com/dp/B009VV56TO/?tag=ozlp-20
I want this build as silent as possible. I know I can't get the drives to not emit noise, but at least I can prevent everything else in the case from doing so.

I don't like the idea of booting off of USB sticks. Costs these days are not the limiting reason. To me, reliability of the boot disk is. As well as speed. So I was thinking of using M2 or just a low price SSD. I have an old Crucial 64gb SSD I'll probably use, or this one even though its size will be wasted.
Crucial MX300 275GB SSD https://www.amazon.com/dp/B01IAGSD5O/?tag=ozlp-20

So my questions:
1. What do you think of my 4x8TB choice? A good one now considering cost? I thought 4 drives would keep power down over running additional smaller drives.
2. What am I forgetting/not aware of?

I was originally just going to mirror the drives to get 16tb out of 4 8tb drives. But might uze zfs if it gives me more space and a further increase in reliability and features/benefits. But I like less complexity and maintenance, so was thinking just mirroring them. That way I could get the data off of them easier in case of a major catastrophy buy just plugging them into another board? Interested in input on this. I also plan to add more drives later should I want even more storage. Up to a total of 8 drives max.

Will be connected to an UPS.

Thanks for your input!
R

 

[wblock]

Some random notes:

The i5-6400 (or T) does not support ECC, unfortunately. For that kind of performance with ECC support, a Xeon is needed.

Even if people have found 8TB Red drives in those USB drive cases before, probably not safe to count on that. And some of those drive cases make it impossible to remove the drive without damaging the case and losing the warranty. People here might recommend a larger number of smaller drives.

If you use FreeNAS, you get ZFS. However, where you talk about using ZFS, you probably mean RAIDZ. Depending on the use, four drives in RAIDZ2 might be a better way to go.

The X11SAE boards are okay, I have one. The X11SAE-F has a better selection of slots than the micro ATX version.

Direct video playback on the NAS would probably require X. That should be possible, but might be challenging.

The passive CPU cooler looks interesting but makes me skeptical. Not sure about the 20-50 watts at idle. And finally, noise: Supermicro's fan controls seem poor to me. It's fine for server machines which are generally not worried about noise.

 

[Stux]

How much space do you want ? 6x4TB in Raidz2 will be cheaper than 4x8TB and will give you higher performance too.

 

[Rational]

Thanks for the replies so far. wblock, I don't think they make any other 8tb drives other than the red drives from what I've read around the web. Probably won't be any other models for awhile either. But I appreciate the tip. I'll be careful. The point about the case is correct though. It is a tough nut to crack open. But there are some guides by others doing it regularly. Cost savings is substantial over the bare drive.

That CPU doesn't support ECC? Oh, man... you're right. Just looked it up on ark. Well.... there goes that cost savings. Thank you for that. Glad I asked. Now to find the next best value that still supports ECC...

Yes, I mean RAIDZ2. Been too long since I last studied up. Thank you.

Yeah, I think I may actually go with a full sized board. Now just need to determine which one has the most options. I like the boards with 8 sata ports and as many slots as I can get. I'll check into the F board. Thanks.

Yeah, onboard video playback was just a high hope. Not an essential. I have plenty of other TV set top boxes that can handle that duty already. Like the Nvidia 4k box. I'll just use that.

On the wattage thing, I'd read another poster here say he was averaging around 50 watts idle on his x11 board. So I know that's possible based on his experience and testing with a kill-a-watt meter.

Thanks for the tip on the supermicro noise potential relating to their fan control limitations. That's good to know. Might just reinforce my desire to go as passive as possible or handle my own airflow/fan speeds externally to ensure quiet and reliable ventilation.

Stux, I'm looking to get as much space as possible for my dollar with the fewest drives. Isn't 6x4tb only 24tb vs 32tb going with 4x8tb? I don't care as much about performance as I do having as much TB as I can get. Do you still recommend I reconsider for other possible benefits/reasons? Thanks for your input on this as I may be unaware of something essential to the decision.

 

[Dice]

X11SSM-F-O

cool.

I wanted a board I could throw a GTX1070 into someday should I want to watch movies in 4k@60 right from the NAS if that's an option

At this time, this is not something FreeNAS does. You don't need any gfx chip at all. Transcoding is done on the CPU (PLEX). Hence the interest in Xeon 1230v5.

Intel i5-6400T to keep TDP as low as possible.

Forget about -T versions. They are not more efficient, just <gimped> to not overheat in cooling critical situations (something freeNAS system should not be aimed towards)

Seasonic Fanless 520 Watt Platinum

Lacking fan? Avoid that. A good fan sounds less than drives. Get a fan. Platinum is killer tho.

I don't like the idea of booting off of USB sticks. Costs these days are not the limiting reason. To me, reliability of the boot disk is. As well as speed.

Good. Having a config file backup is essential. then installing and importing your old config is trivial. A boot drive failure is not the end of the world with freenas. Speeds may be somewhat relevant during updating. Other than that - never. Since the OS is loaded into the RAM upon boot.

What do you think of my 4x8TB choice? A good one now considering cost?

Stealing drives from usb drive cages can be done, but you'll need to make the math if it is worth it considering warranty void. It is tempting, but cannot see how a failed batch of drives for instance, 3-4 of the drives turn out to be monday morning production half assed units, breaking after 2 months. Then what? no warranty? would it be worth that gamble? I don't think so. I'd rather look into smaller drives.

I was originally just going to mirror the drives to get 16tb out of 4 8tb drives. But might uze zfs if it gives me more space and a further increase in reliability and features/benefits. But I like less complexity and maintenance, so was thinking just mirroring them. That way I could get the data off of them easier in case of a major catastrophy buy just plugging them into another board? Interested in input on this. I also plan to add more drives later should I want even more storage. Up to a total of 8 drives max.

You'll need to sharpen up your use of terminology. This is a guessing game. Cyberjocks newbie guide is helpful - rehearsal is key to learning. check my signature.

On the wattage thing, I'd read another poster here say he was averaging around 50 watts idle on his x11 board. So I know that's possible

Some 60-100w can probably be achieved.
With drives and a Xeon CPU. Marginally helped by getting a platinum PSU, not helped at all by getting a -T CPU. Definitely not worth the trade off if reducing fans and cooling - to save power.

 

[Ericloewe]

My goal is to keep idle wattage as low as possible (20-50watts max)

That should be easy.

Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSM-F-O

Excellent choice.

I wanted a board I could throw a GTX1070 into someday should I want to watch movies in 4k@60 right from the NAS if that's an option.

Plus throw in my own pro audio soundcard perhaps in the PCI slot.

That's not going to be particularly viable. Maybe with VMs and PCI-e passthrough, but it's far from a trivial thing to do.

Though I've since read that the "T" models are artifically capping performance and that other models run low power at idle as well. So is the non "T" model of this i5 a better choice?

Yes, the low-TDP versions do not have any tengible benefits unless the system is seriously thermally or power-delivery constrained - which it shouldn't be.

Passive CPU cooler NoFan 95

Looks dubious. You'll always need fans, so using a fanless cooler is a fool's errand (unless it's a rackmount chassis with immense airflow).

PSU: Seasonic Fanless 520 Watt Platinum.

Same as above, good PSUs are essentially inaudible unless you hold your ear next to them.

I want this build as silent as possible. I know I can't get the drives to not emit noise, but at least I can prevent everything else in the case from doing so.

You need fans to cool the drives, no way around that.

I have an old Crucial 64GB SSD I'll probably use

Fine.

I thought 4 drives would keep power down over running additional smaller drives.

Slightly, but both my servers idle below 50W, easily.

But I like less complexity and maintenance, so was thinking just mirroring them.

You'll always be using ZFS, that's the whole point (and only option) of FreeNAS. RAIDZ2 is inherently more reliable than a set of two mirrors, but less flexible.

2. What am I forgetting/not aware of?

What's the chassis?

Even if people have found 8TB Red drives in those USB drive cases before, probably not safe to count on that. And some of those drive cases make it impossible to remove the drive without damaging the case and losing the warranty.

Important points to keep in mind, though most people seem to be getting equivalent drives (Helium 8TB drives), if not always WD Reds.

And finally, noise: Supermicro's fan controls seem poor to me.

It works fine for me, my Noctua fans nearly always run close to their minimum speed.

 

[Rational]

Thanks Dice, Eric.
Actually Eric, I was reading your advice and noticing your signature build a few months ago. The more I read the more I see the simple elegence of it.

Thank you for the confirmation on the T CPUs. Throwing that idea out.

The i3-6100 seems an especially good value considering it offers ECC support. I think I just may go for this and 'someday' upgrade to a Xeon v5 if I really feel the need when prices eventually come down.
https://www.amazon.com/dp/B015VPX2EO/?tag=ozlp-20

Unless Dice, you are saying I may still need the Xeon for the transcoding still over the i3?
And I'll take your advice on skipping the fanless PSU. You are right, I have tried some over the last few years with fans that only turn on under load and they are not able to be heard. It's a wise thing to have in place for protection. I'll still go for the Platinum model. Just need to choose which model I want now. Thank you.

And Eric,
I'm curious why everyone seems to love the x11-SSM-F-O board. Even over the SAE with 8 white (I'm assuming native chipset) SATA ports? Are there some differences/concerns with the SAE instead of the SSM-F models?

Right now I'm leaning toward going with the x11-ssm-f-o board based on everyone praising it and it is also your stock build. If it works for you it'll work for me.

And yeah, you're right. I will need fans for the hard drives. I like the Icy Dock Fatcage recommendation. But I think I'll go for the 5 bay model in the 3x slot. This is another golden find from your signature. This should fit my 4 drives and my SSD I believe just fine. https://www.amazon.com/gp/product/B00DWHLFMA/?tag=ozlp-20

My case is just an old ATX case I have. Had it for a decade or more. Bought 4 of them identical. Has 3 51/4 drive bays at the top. So think that Icy Dock will slide right in there perfectly. No space for CD Roms though thereafter. Oh well.. I really wouldn't have one on it anyway.

So it looks like I'm either going with the i3 or the Xeon depending on further counsel. And an Icy Dock.

And yeah Eric, I had been reading (3 months ago or so) that some of those drives they are giving in the external enclosures are helium. But yes, the lack of warranty support is a point to consider Dice. Thank you. But the full retail price on the bare drive is +$70 per drive. So that drives up costs $280. No small percentage. And I lose the 4 free external enclosures for older drives to use. :) USB 3 at that.

Thanks for the confirmation on your idle being under 50 watts Eric. Good to have that confirmed. Nice! What I was hoping for.

And yeah, I was originally interested in FreeNas for it's strong ZFS capabilities. So perhaps I need to restudy everything I've forgotten and get back up to speed anyway and just utilize it for it's strong points. E.g. skip mirroring and let it do what it was designed for as one big array.

Lastly, I was reading up on the supericro DOM boards. Sounds perfect. Need to verify which models work on the x11 boards as their website page for them listed x10 compatibility even though the sales page for the x11 boards say it supports them.

And I'll give up on the idea of trying to play video direct from the NAS. Maybe FreeNAS will someday build this into a future version to keep current with the competition. Until then, I'll drop this pursuit.

Thanks again guys for the very sage advice. It's greatly appreciated. Really. Thank you.

 

[Stux]

Stux, I'm looking to get as much space as possible for my dollar with the fewest drives. Isn't 6x4tb only 24tb vs 32tb going with 4x8tb?

Assuming Raidz2, they're the same,but one is higher performance and much cheaper.

Subtract 2 drives for redundancy in Raidz2 and you're left with 4x4TB or 2x8TB. Either way is 16TB. But one has a 50% parity loss, where the other has 33%.

 

[Rational]

Ugh, I can see I'm going to have to refresh my knowledge all over again on this for sure. Thanks Stux. Right now I'm not seeing how 32tb of raw drive space is the same as 24tb of raw HD space, but I'm sure it's just that I am not up to speed on the logical partitioning limitations you're trying to make me aware of here. I'll study up again over the next week so I understand. Thank you for trying to save me here.

Wait a minute... so upon further rereading of that... you're saying I lose 2 drives for redundancy even under raidz2? Or is that only because I'm only planning on having 4 drives? If I'd planned to have 6 for example, then only 2 are designated for redundancy? Not 4. So really, with 4 drives, it sounds like it doesn't benefit me much over mirroring except I get one big array instead of two pairs of mirrored drives, and perhaps some additional error correction and checking.

Time to RTFM.

 

[danb35]

Since the OS is loaded into the RAM upon boot.

This has not been the case since 9.3 was released, nearly two years ago. ZFS caching means that most of the OS will be in RAM most of the time, but it isn't loaded into a RAMdisk as it once was (under 9.2.x and prior).

 

[Ericloewe]

Wait a minute... so upon further rereading of that... you're saying I lose 2 drives for redundancy even under raidz2? Or is that only because I'm only planning on having 4 drives? If I'd planned to have 6 for example, then only 2 are designated for redundancy? Not 4. So really, with 4 drives, it sounds like it doesn't benefit me much over mirroring except I get one big array instead of two pairs of mirrored drives, and perhaps some additional error correction and checking.

• Each RAIDZ2 vdev has two drives' worth of parity
• RAIDZ doesn't have designated parity drives, all drives get a bit of everything
• Striped mirrors are faster than RAIDZ2, but you'd never notice with your workload. RAIDZ2 is significantly safer.

 

[Chris Moore]

Ugh, I can see I'm going to have to refresh my knowledge all over again on this for sure. Thanks Stux. Right now I'm not seeing how 32tb of raw drive space is the same as 24tb of raw HD space, but I'm sure it's just that I am not up to speed on the logical partitioning limitations you're trying to make me aware of here. I'll study up again over the next week so I understand. Thank you for trying to save me here.

Wait a minute... so upon further rereading of that... you're saying I lose 2 drives for redundancy even under raidz2? Or is that only because I'm only planning on having 4 drives? If I'd planned to have 6 for example, then only 2 are designated for redundancy? Not 4. So really, with 4 drives, it sounds like it doesn't benefit me much over mirroring except I get one big array instead of two pairs of mirrored drives, and perhaps some additional error correction and checking.

Time to RTFM.

My NAS, with 14 drives, only consumes around 150 watts. You need to check the specs on the specific drive you want to use, but my drives draw around 8 watts each at idle.
The lower the wattage per drive and the lower the number of drives, naturally it will consume less power. The thing you loose is redundancy. I can have as many as four drives fail and not loose any data because of the way I have the system configured. I understand that you want to keep the power consumption low, but the six drive configuration proposed by Stux is better from the standpoint of redundancy and it won't draw much more power.

The other issue that you have stated is in regard to fans. In one of my early builds, I did not have enough air-flow and my drives were overheating. This caused many premature failures of the drives. I had to make some changes to fix my air-flow and it caused me to have to replace drives that might have lasted much longer under other circumstances. You need to monitor the heat on the drives and keep it below 40C (as a maximum) I keep mine below 35C. The highest temp right now is 34 and the lowest is 29. I am looking at the 34 degree drive and wondering if something is wrong with it because it stands out from all the rest. The point is, you have to use fans to keep the heat off the hard drives because they make heat. Western Digital drives (when I was using them) ran hottest of all and were putting out over 45C with the same fan solution I am currently running. The HGST drives ran a little cooler, but the lowest temp drives I have used are the Seagate NAS drives and I have used all the big name drives, even Toshiba. The thing that ultimately became standard for me are the Seagate Barracuda (Desktop) drives because I like the level of detail that their SMART function gives. You may have heard bad things about the Seagate Desktop drives, but of the 28 that I have used only three have developed bad sectors and none have caused data loss. Bad sectors is a cause for replacement. Any drive can fail, that is why more redundancy is better.

Monitoring of your system. Once you build it, you can't just ignore it, unless you want to loose all your data. Please look into the scripts on this page:
https://forums.freenas.org/index.ph...d-identification-and-backup-the-config.27365/
You can setup cron jobs to run SMART tests and e-mail the results to you so you can know what is going on with your system because the hard drives are the most likely thing to fail.

Speaking of drives, I suggest using actual spinning disks for boot drives because you can more easily monitor the health of spinning disks than SSD and USB drives are worse than useless. I would never use USB with FreeNAS primarily because there is no way to tell if it is going to go bad before it just stops working. I have had to recover from a USB boot drive failure. Never again. I get a daily SMART report on my boot drives just like any other drive in my NAS. You can have FreeNAS install to a single drive and add a mirror later or you can install to a mirror from the beginning. I use 40 GB laptop drives. They don't pull much power, they are quiet and they have given me no problems so far. These would be an excellent choice: http://www.ebay.com/itm/321882627076 . They don't need to be fast, and since you have redundancy, they don't even have to be that reliable, and they are only used a very little because the data that is stored on them will probably never be more than 8 to 16 GB.

I hope the solution you develop is all you could want it to be.

 

[Dice]

This has not been the case since 9.3 was released, nearly two years ago. ZFS caching means that most of the OS will be in RAM most of the time, but it isn't loaded into a RAMdisk as it once was (under 9.2.x and prior).

Thanks for correcting.
Though it is still correct to put forward the merciful performance requirements of the boot drive, which was my point.

I suggest using actual spinning disks for boot drives

Never seen this argued before. I like the concept. Unorthodox, but why not.

 

[Chris Moore]

Thanks for correcting.
Though it is still correct to put forward the merciful performance requirements of the boot drive, which was my point.

Never seen this argued before. I like the concept. Unorthodox, but why not.

Mostly, people appear to want to save the SATA ports for their storage drives. That and the fact that FreeNAS did not demand much from the boot drive was the whole reason that the USB drive was the "desirable" option. I did that on my first few builds, but now I have a SAS controller for my data drives and the SATA ports on the system board would not be doing anything if I didn't use them for this. It really is, as you say, "why not." There are advantages and no down side. There is no advantage to using a SSD for booting and they are more expensive but the SSD presents a downside in that older ones do not give useful SMART data to monitor the health. The laptop drives I am using present the same type of SMART data as my data drives and it allows me to have some warning if something is going wrong. I have been running this configuration for almost a year and have had zero problems with it and don't anticipate any. I actually got the idea from some old commercially produced NAS units that we were decommissioning where I work. They had a mirrored pair of laptop drives for their boot device (using hardware RAID) and we never had a lick of trouble out of those in the 7 years they were in service. Some of the storage drives had to be changed but the boot drives never failed. It makes a kind of sense too because laptop drives are designed to work in a mobile environment where they have to deal with vibration and temperature extremes. If you put them in a server chassis, they are bound to be more reliable.

 

[Rational]

Some wonderful input here guys. Thank you. I'll be digging into it in depth and taking into account all your input.
Interested in good case recommendations in the meantime if you'd like to offer up any. Something that can handle 8 5-1/4 drives. Mostly likely in the Icy Dock 5 by 3x enclosures. Or just a better case solution if you know of some.

Also interested in PSU brand/model recommendations. I usually go for Seasonic Platinum, but there are a lot of higher quality offerings nowadays that I may not be up to speed on. I prefer energy efficiency for cost savings over the long term as a priority. And of course the highest, rock solid reliability.

Thanks again for all the input so far. It's really helping. I just need to get into the manual and better understand the ZFS/Raidz2 aspects before deciding on number and size of drives. And might do a satadom on this board too to free up a sata. Or use Chris's idea of using two old laptop drives as I have like 5+ of those lying around unused. Curious what SAS controller you're using Chris. I'll take a look at your builds and determine and then check them out.

Thanks again everyone.

 

[Ericloewe]

I usually go for Seasonic Platinum,

You won't find anything that's better than the Seasonic Platinums in every aspect. Seasonic Prime Titanium comes close, but the fan is crap and transient performance is on the weird side.

 

[joeschmuck]

Seasonic

Seasonic is the best. Now I want to buy a low wattage unit, and I do need one so... Time to start looking for myself.

 

[Ericloewe]

Seasonic is the best. Now I want to buy a low wattage unit, and I do need one so... Time to start looking for myself.

How low? For 650W, the X-650 (or platinum 660, if you can find it for a trivial price difference) is very hard to beat in quality for money.
Pricing should be somewhat more favorable now that the Prime Titaniums are generally available.

 

[joeschmuck]

How low? For 650W, the X-650 (or platinum 660, if you can find it for a trivial price difference) is very hard to beat in quality for money.
Pricing should be somewhat more favorable now that the Prime Titaniums are generally available.

I'm looking for something in the 300 Watt range. I'll need to do a little research as I might want something small too. I think I'm building a firewall computer now, or should I say, moving it from a large case and into a small case and it doesn't need a 650 Watt PS.

 

[Chris Moore]

Some wonderful input here guys. Thank you. I'll be digging into it in depth and taking into account all your input.
Interested in good case recommendations in the meantime if you'd like to offer up any. Something that can handle 8 5-1/4 drives. Mostly likely in the Icy Dock 5 by 3x enclosures. Or just a better case solution if you know of some.

Also interested in PSU brand/model recommendations. I usually go for Seasonic Platinum, but there are a lot of higher quality offerings nowadays that I may not be up to speed on. I prefer energy efficiency for cost savings over the long term as a priority. And of course the highest, rock solid reliability.

Thanks again for all the input so far. It's really helping. I just need to get into the manual and better understand the ZFS/Raidz2 aspects before deciding on number and size of drives. And might do a satadom on this board too to free up a sata. Or use Chris's idea of using two old laptop drives as I have like 5+ of those lying around unused. Curious what SAS controller you're using Chris. I'll take a look at your builds and determine and then check them out.

Thanks again everyone.

This is a very good choice for a case, it includes redundant power supplies and you don't have to buy and of those over priced Icy Dock things.
http://www.ebay.com/itm/172315518085

I use a Dell H310 SAS controller. It can be flashed to IT mode using the LSI firmware. I have two of them and they work super. Here is a link:
http://www.ebay.com/itm/Dell-H310-P...-0HV52W-0T3F4V-RAID-0-1-5-10-50-/152297561810

After being flashed, it shows up as: "Avago Technologies (LSI) SAS2008"
If you are interested in getting one, but need help with flashing it, just shoot me a message.