BUILD Proposed FreeNAS build

[MindBender]

My ReadyNAS 6 Pro is getting a bit slow and seems to get in a live-lock regularly after scrubbing the array, so it is time to replace it. I have been aware of FreeNAS for a long time, but didn’t dare to take the plunge, because I just wanted a box to safely store my data, not something I need to tinker with myself before getting it to work. But both my ReadyNAS 6 Pro and the Thecus that preceded it have been a bit of a disappointment (the latter more than the first), so I decided to go for FreeNAS. Base upon FreeBSD, built like a brick and supporting ZFS, what more could one want?

I’ve been doing a lot of research, the past couple of days; I’m on a holiday resort so I’ve got plenty of time, nothing to do and a good WiFi connection ;-). Now before I start ordering parts, I thought it may be a good idea to run my ideas by you guys.

CPU
At first I wanted to go for a Xeon E3-1285L, but after reading many reviews and benchmarks I decided to go for a Xeon D-1540, which will immediately limit my choice in mainboards.
Mainboard
My choice of mainboard is pretty much settled on the SuperMicro X10SDV-TLN4F. [Ordered]
Edit: The SuperMicro X10SDV-8C-TLN4F is identical, but with a passive heatsink, for setups with an airflow plan with which a fan may interfere. Chances are you aren't Dell or HP, and if you aren't, trust me: You don't have an airflow plan, so stay away from it.

Memory
I’m thinking about getting 2 DDR4 RDIMMs of 32GiB each, leaving to slots free to max it out later. The user’s manual shows only one tested DIMM, made by Samsung. In my mind, Samsung is still cheap Korean imitation stuff, but if SuperMicro trusts them, so will I. [Ordered]

Drives
For now I will reuse the drives from my ReadyNAS, which are 4x Seagate Constellation ES.2 Enterprise Grade 3TB drives. Seagate isn’t my favourite brand, but since all brands - including Seagate - have let me down, I decided to purchase drives of which I was 100% sure there were Enterprise Grade. So far they have been holding up well. I may add a 5th (and 6th) drive to improve RAIDZ1 (or RAIDZ2) performance.
I’m also considering to add two Seagate Archive 8TB drives, just for backups.

Boot disk
For previous customers I have done extensive research on NAND flash reliability and especially power-fail robustness, all the way down to flash translations layers remapping bad erase blocks and levelling wear evenly across the NAND flash device, and doing that concurrently. They just cannot seem to get cross-page corruptions in MLC NANDs right, or at least not concurrently across a power failure. The conclusion was that there are no devices doing that 100% reliable, which was to be expected, but also that there are very, very few devices doing a pretty decent job. The investigation covered both SD-Cards and USB flash drives. I haven’t done any research on SSDs, but I have had a couple during the past few years and all failed undetected until it was too late. These drives include an OWC Mercury Extreme PCIe card, an OZC RevoDrive X2 PCIe card and multiple OCZ Vertex drives.
The Mainboard I have chosen does have an M.2 slot and again I’m tempted to put an SSD in. Selecting a PCIe V3.0 module is especially attractive, because that will make all 6 SATA connections on the board available for drives. Unfortunately the only PCIe M.2 module I could find is a €300 Samsung 950 Pro of 512GiB. That seems like a bit overkill. For that kind of money I’d rather have a smaller SLC based drive.

Case
The case is a bit of a problem. I found my Thecus NAS a big turn-off from the moment I opened the box and felt the flimsy plastic drive trays. The ReadyNAS drive trays a much better. In my experience I don’t think I will ever need more than 6 drives, because experience teaches me that I usually replace them before adding a 7th drive. But the bare minimum is 5, due to RAIDZ1 performance. The SilverStone D380 is a no-go because of the flimsy drive trays. The SilverStone CS01-HS is still a contender, but may be just a flimsy. The Norco ITX-S8 may have better drive trays, but is flimsy on the rest. And I don’t have any information on the build quality of the U-NAS NSC-800. This latter is my favourite, but I already know that the PCIe extender causes bus errors, so that doesn’t make a good impression. [Ordered]
If only there were a well-build case, like my ReadyNAS, with decent drive trays and a display on the front….

Power supply
The power supply depends on the case I will choose, and on the number of drives it can house. Dimensioning it is not an easy problem to solve. Most power supplies peak in efficiency around 50% of their maximum capacity. Efficiency for loads under 20% often isn't specified, while my system will spend most of it's time in idle. So I need to determine the load bandwidth and try to map that on the 20...80% range of a power supply. Unfortunately the higher-efficiency power supplies are also the higher capacity power supplies; 80 Plus gold supplies start at 400Watt, meaning that their efficiency rating starts at 20% of that, being 80Watt. My will only use that kind of power a small part of the time. 80 Plus bronze supplies start at 250Watt, making it hit 81% efficiency from a load of 50 Watt and up. I still don't think my setup will do that in idle with the drives spun down, but in reality I have never seen my ReadyNAS or Thecus spin down any drives, so I'll keep them running and the 50Watt limit won't be a problem.
I will probably end up with a SeaSonic; I have read many good things about them.

What do you guys think of my proposed build?

 

[Ericloewe]

In my mind, Samsung is still cheap Korean imitation stuff,

What? Samsung is one of the few manufacturers of DRAM that matter (along with NAND flash and processors), along with Micron and Hynix. They're tier 1.

I decided to purchase drives of which I was 100% sure there were Enterprise Grade

That strategy is of dubious value. We tend to advocate more reliable setups with consumer drives, since they're cheaper and any reliability difference is dwarfed by an extra drive's worth of redundancy.

For previous customers I have done extensive research on NAND flash reliability and especially power-fail robustness, all the way down to flash translations layers remapping bad erase blocks and levelling wear evenly across the NAND flash device, and doing that concurrently. They just cannot seem to get cross-page corruptions in MLC NANDs right, or at least not concurrently across a power failure. The conclusion was that there are no devices doing that 100% reliable, which was to be expected, but also that there are very, very few devices doing a pretty decent job. The investigation covered both SD-Cards and USB flash drives. I haven’t done any research on SSDs, but I have had a couple during the past few years and all failed undetected until it was too late. These drives include an OWC Mercury Extreme PCIe card, an OZC RevoDrive X2 PCIe card and multiple OCZ Vertex drives.
The Mainboard I have chosen does have an M.2 slot and again I’m tempted to put an SSD in. Selecting a PCIe V3.0 module is especially attractive, because that will make all 6 SATA connections on the board available for drives. Unfortunately the only PCIe M.2 module I could find is a €300 Samsung 950 Pro of 512GiB. That seems like a bit overkill. For that kind of money I’d rather have a smaller SLC based drive.

Tune it down. Waaay down. The boot device is decidedly non-critical. A cheap but reliable, smallish consumer SSD is more than fine.

display on the front….

You can't trivially drive it anyway, so there's not much of a point to it. If you want hot swap bays (for whatever reason), consider and ATX chassis with 5.25" hotswap bays that take multiple 3.5" drives.

I will probably end up with a SeaSonic; I have read many good things about them.

Seasonic is good. A G-450 will work well for 6 drives (and a bit more). A G-550 is slightly more future-proof. Beyond that, I'd step up to an X-Series/Platinum.

 

[joeschmuck]

For the MB you selected, I'd find out some more about the SATA0 and M.2 usage before committing. Also, it would be nice to know if anyone has had success using an M2 PCI-e card as a boot device for FreeNAS.

 

[MindBender]

What? Samsung is one of the few manufacturers of DRAM that matter (along with NAND flash and processors), along with Micron and Hynix. They're tier 1.

For bare NAND flash, I'm still sticking with Toshiba. SanDisk for NAND flash based products. But you're right and I'm a bit old-fashioned ;-).

That strategy is of dubious value. We tend to advocate more reliable setups with consumer drives, since they're cheaper and any reliability difference is dwarfed by an extra drive's worth of redundancy.

That's sound advice, hence the 'ID' in 'RAID'. The reason I'm using Enterprise Grade drives isn't a very sane one: At some moment I got so tired of swapping out a broken down drive every 3 to 6 months, having to rebuild the array and having it vulnerable why doing that for 24 hours (!), that I decide to throw some money at that problem and be done with it. The drives have been holding up very well since October 2011. Once was starting to produce SMART errors, but was immediately replaced with a repaired drive. I was surprised that they actually did that. I kind'a assumed that the Enterprise Grade drives were just consumer drives with different firmware and better warranty, but they really seem to be worth repairing. And I was sceptical, but the repaired drive is holding up perfectly.
But I'm going to take your advice, because chucking in another >€200 harddisk is kind'a pointless when using RAIDZ2 and a 3TB WD Red drive is less than half that price.

Tune it down. Waaay down. The boot device is decidedly non-critical. A cheap but reliable, smallish consumer SSD is more than fine.

Yeah, I know; It wasn't a very modest first post ;-). Nah, I got what you mean. I just got a little paranoid with NAND flash devices. Call it an occupational deformation: I you work in a radio repair shop you start thinking that all radios are rubbish and break down easily ;-).
So the Root File System isn't on the boot device? I'll have to dig deeper into the boot sequence. I know FreeNAS uses GRUB and GRUB probably reads a Kernel and InitRD from a ZFS partition on the boot device. That basically only makes GRUB being loaded and executed with being checksummed.

You can't trivially drive it anyway, so there's not much of a point to it. If you want hot swap bays (for whatever reason), consider and ATX chassis with 5.25" hotswap bays that take multiple 3.5" drives.

Thanks; I'm leaving it out. It's stationed under the stairs anyway, so nobody's watching it.

Seasonic is good. A G-450 will work well for 6 drives (and a bit more). A G-550 is slightly more future-proof. Beyond that, I'd step up to an X-Series/Platinum.

Really 450 Watt? I'm a bit surprised, because I was thinking 250 Watt... The board does 60Watt at most, and 8 drives do 12Watt each, still leaving 250 - 60 - 8 * 12 = 94 Watt for fans, boot devices, USB devices and headroom for drives spinning up. Can you tell me where I'm mistaking?
Speaking of which: I will see if I can measure the rush-in current of a drive spinning up and post the result here.

 

[MindBender]

For the MB you selected, I'd find out some more about the SATA0 and M.2 usage before committing. Also, it would be nice to know if anyone has had success using an M2 PCI-e card as a boot device for FreeNAS.

Well, somebody has to be the first one, and I don't mind being that scmuck ;-). I just wish there were smaller (read: cheaper) M.2 PCIe devices out there...

 

[Bidule0hm]

Really 450 Watt? I'm a bit surprised, because I was thinking 250 Watt... The board does 60Watt at most, and 8 drives do 12Watt each, still leaving 250 - 60 - 8 * 12 = 94 Watt for fans, boot devices, USB devices and headroom for drives spinning up. Can you tell me where I'm mistaking?
Speaking of which: I will see if I can measure the rush-in current of a drive spinning up and post the result here.

Already discussed many times but the short answer is you don't want to use a PSU near its 100 % rating and 8 drives spinning up eat 168 W on the +12 V rail (for WD Red, it can be more with other drives) and you need to add the power on the +5 V rail too.

Note that if you don't measure the current on the +12 V rail (e.g. on the mains side) the measure will be useless.

 

[Arwen]

Tune it down. Waaay down. The boot device is decidedly non-critical. A cheap but reliable, smallish consumer SSD is more than fine.

...
Yeah, I know; It wasn't a very modest first post ;-). Nah, I got what you mean. I just got a little paranoid with NAND flash devices. Call it an occupational deformation: I you work in a radio repair shop you start thinking that all radios are rubbish and break down easily ;-).
So the Root File System isn't on the boot device? I'll have to dig deeper into the boot sequence. I know FreeNAS uses GRUB and GRUB probably reads a Kernel and InitRD from a ZFS partition on the boot device. That basically only makes GRUB being loaded and executed with being checksummed.

As long as you backup your configuration file off the boot drive, FreeNAS can be restored. My configuration backups go to both my ZFS Pool and 2 remote locations, (netbook and media server). With the configuration in my ZFS Pool, that get's backed up to my 8TB Archive drive, (used for, well, backups).

Last, for those extra special paranoids, (me included , it's now EASY to mirror the FreeNAS boot drive. When using mis-matched media, (M.2 and USB for example), install on the smaller one first.

 

[Ericloewe]

8 drives do 12Watt each

30W is a better estimate for spinup power, with proper margins. Then many people here advocate keeping PSUs loaded below 80% (I think that's a bit exaggerated if we're talking about a very short transient and proper PSUs. I expect anything from Seasonic to not break a sweat with such a load, for instance.).

 

[MindBender]

Already discussed many times but the short answer is you don't want to use a PSU near its 100 % rating and 8 drives spinning up eat 168 W on the +12 V rail (for WD Red, it can be more with other drives) and you need to add the power on the +5 V rail too.

Note that if you don't measure the current on the +12 V rail (e.g. on the mains side) the measure will be useless.

Yeah, I know the topic is coming up frequently, yet I was really surprised by the 450 Watt suggestion, double what I was thinking about. And yes, I'll keep a sharp eye on the 12V rail. I've got a nice Agilent MSO-X4000 series oscilloscope and a set of current probes, so there's no need for guessing. I'll post the oscillographs here (as soon as I'm back from my holiday), so we will all have some real-world data.

As long as you backup your configuration file off the boot drive, FreeNAS can be restored. My configuration backups go to both my ZFS Pool and 2 remote locations, (netbook and media server). With the configuration in my ZFS Pool, that get's backed up to my 8TB Archive drive, (used for, well, backups).


Last, for those extra special paranoids, (me included :), it's now EASY to mirror the FreeNAS boot drive. When using mis-matched media, (M.2 and USB for example), install on the smaller one first.

Thanks for the advice. I am definitely among the paranoid when it comes to that. Especially for my configuration, because that's nothing I will do every day and re-figuring that out can be tedious. I already have a big problem if my Juniper router configuration and L2 managed switch configuration get damaged for exactly that reason.

30W is a better estimate for spinup power, with proper margins. Then many people here advocate keeping PSUs loaded below 80% (I think that's a bit exaggerated if we're talking about a very short transient and proper PSUs. I expect anything from Seasonic to not break a sweat with such a load, for instance.).

That was actually my plan too; I planned to dimension my PS in such a way that it is running at 20% load in idle, 50% at regular use and 80% at full load. Below 20% load efficiency of modern supplies is really poor, so that area has to be avoided. I don't think a high quality power supply will suffer from being under 100% load for a long time. As long as you keep it cool, it will last; Capacitors don't like heat. Most industrial power supplies can handle 150% load for 10 seconds, so I'm not to worried about hitting the 100% mark.

Thanks for all your help guys!

 

[Bidule0hm]

Oh, if you can post screenshots of currents of a drive on the +5 and +12 V rails at start up it would be great (if you can do the same for the MB it would be wonderful, but the high number of wires for the smae rail will probably complicate things a little bit) :)

Below 20% load efficiency of modern supplies is really poor, so that area has to be avoided.

One of the things who make me like SeaSonic PSUs is that I know they have very good efficiency at low load.

I don't think a high quality power supply will suffer from being under 100% load for a long time.

No, it'll not, it'll even improve his lifetime. However some bad quality PSUs can be unstable at very low load (< 10 %) but if you have such a PSU in your server then you've made a mistake during choice of components.

Most industrial power supplies can handle 150% load for 10 seconds, so I'm not to worried about hitting the 100% mark.

The thing is most industrial applications use the PSU far below the 100 % mark, I've seen as low as 50 %. As a side note I generally tend to follow what is done in the industrial realm because it's made for reliability, ease of maintenance, etc...

 

[MindBender]

For the MB you selected, I'd find out some more about the SATA0 and M.2 usage before committing. Also, it would be nice to know if anyone has had success using an M2 PCI-e card as a boot device for FreeNAS.

I have stepped away from the M.2 solution because it defeats the purpose: There are really very few PCIe M.2 SSDs out there, and the ones that are, are very new high-performance consumer grade SSDs, too large and too unreliable for my purpose and too expensive as well.
The really reliable SSDs are all still SATA devices, so I have to accept loosing one of the 6 SATA ports for my boot drive. Intel has a DC S3500 in M.2 form factor, but when using a SATA port for it, the M.2 form factor no longer has any benefits. Reviews indicate that the Intel DC S3700 series is still a lot better than the DC S3500 and hardly more expensive, so may go for that one.

 

[MindBender]

You guys have been recommending specific fans all over this forum, but I cannot remember exactly which one, so it's hard search for forum for that. Can anybody tell me what 120mm fans are recommended?
Is it perhaps the Noctua NF-F12 industrialPPC-2000 IP67 PWM 120mm? My board can handle 4-pin fans, so I'd like to use that feature. And I'm in doubt between the 2000 and the 3000 RPM version. My case will take 2 of these, so no shortage in airflow, but if the low-end is similar on both fans, I prefer the extra headroom of the 3000 RPM version.

 

[Fuganater]

Why are you worrying about using an SSD for your boot devive? They give no real benefit and you can pick up 2x sandisk fit USB drives for about $10 and you won't lose a SATA port.

Sent from my SM-T700 using Tapatalk

 

[joeschmuck]

Why are you worrying about using an SSD for your boot devive? They give no real benefit

Not true. They are more reliable than USB Flash Drives. It's also up to personal preference. If I were building another FreeNAS machine from scratch, I'd also plan to use a SSD as a boot device.

so I have to accept loosing one of the 6 SATA ports for my boot drive.

I thought you were going to start with 4 or 5 HDD's, not a full 6. If you really wanted to you could install an add-on card to add a few more SATA ports.

Can anybody tell me what 120mm fans are recommended?

It depends on what you are looking for and if you engineered the case properly for optimum air flow across the right areas. I prefer a silent system for my main computer so I've been using "be quiet!" Pure Wings 2 fans (mine run at 7VDC with my home made adapter). If sound isn't important and quantity of airflow is then you could use 120VAC fans which can push lots of air. When looking at fans you should be looking at several key things: CFM, Air Pressure, RPM, dB. CFM and Air Pressure are related values. RPM it nice to know these days unless you are concerned about a sensor requiring a specific minimum value. And of course dB is the sound level at full RPM.

I'd like to start this entire conversation over again... What is it your expect to use the FreeNAS system for (please be specific), I didn't get the feeling this was for an office environment based on the original posting. Also, do you need this to be a quiet machine or is a jet engine sound okay? The reason I'm asking is because you have selected likely the most expensive motherboard I've seen anyone build a FreeNAS machine out of. Don't get me wrong, for $800 it is probably a great motherboard/CPU combo but for me it had better come with a new TV too (some exaggeration).

 

[MindBender]

Why are you worrying about using an SSD for your boot devive? They give no real benefit and you can pick up 2x sandisk fit USB drives for about $10 and you won't lose a SATA port.

Not really a sane reason, but I prefer not to have vital part sticking out as a removable. I know you could say the same for the power cord, but still... I just need a small drive to boot the machine and small high-quality rotational drives are hard to find, and having it spin the whole day doesn't seem to make sense. An SSD is a bit in between. It's not a performance based choice.

I thought you were going to start with 4 or 5 HDD's, not a full 6. If you really wanted to you could install an add-on card to add a few more SATA ports.

Yes, I will go with the 4 drives I have right now, probably extended with a 5th drive (and 6th) for better RAIDZ1 (or RAIDZ2) performance. My first instinct said to immediately add a SAS/SATA controller, but I think I will start without one, do a couple of tests and order one when needed.

It depends on what you are looking for and if you engineered the case properly for optimum air flow across the right areas. I prefer a silent system for my main computer so I've been using "be quiet!" Pure Wings 2 fans (mine run at 7VDC with my home made adapter). If sound isn't important and quantity of airflow is then you could use 120VAC fans which can push lots of air. When looking at fans you should be looking at several key things: CFM, Air Pressure, RPM, dB. CFM and Air Pressure are related values. RPM it nice to know these days unless you are concerned about a sensor requiring a specific minimum value. And of course dB is the sound level at full RPM.

I will end up under my stairs, so unless it sounds like a jet engine, sound really isn't much of a problem. I've read a lot of good things about the Noctua products; They really seem to put a lot of engineering in their products, and that appeals to me.

I'd like to start this entire conversation over again... What is it your expect to use the FreeNAS system for (please be specific), I didn't get the feeling this was for an office environment based on the original posting. Also, do you need this to be a quiet machine or is a jet engine sound okay? The reason I'm asking is because you have selected likely the most expensive motherboard I've seen anyone build a FreeNAS machine out of. Don't get me wrong, for $800 it is probably a great motherboard/CPU combo but for me it had better come with a new TV too (some exaggeration).

I'm sorry; I neglected to tell you guys what I was planning with the machine. First of all I'm going to put all of my files on it, basically everything I have, so it must be reliable. Second I'm going to run a SubVersion server on it for my work. It will also run a Plex server, my network's DHCP server, a DNSsec server and a gateway to a VPN so my TV can get US Netflix. But not of that justifies the amount of processing power. However, I'm also planning it to have it run a Windows VM to replace all the small breadbox Windows machines I have around the house for the HomeSeer domotica server, the Unify controller and the HP network scanner daemon.
So yes, this is going to be my do-everything-box. I've read most of you guys aren't in favour of that, but I really like to reduce the always-on computer park to just this box.

 

[Fuganater]

If you change from RAID Z1 to Z2 don't you destroy the pool?

Sent from my SM-T700 using Tapatalk

 

[Pheran]

Not really a sane reason, but I prefer not to have vital part sticking out as a removable. I know you could say the same for the power cord, but still... I just need a small drive to boot the machine and small high-quality rotational drives are hard to find, and having it spin the whole day doesn't seem to make sense. An SSD is a bit in between. It's not a performance based choice.

I would consider that to be a pretty sane reason if we were talking about the big, chunky "thumb" drives of yore, but have you looked at the recommended drives such as SanDisk Fit/Ultra Fit? The non-USB plug portion is tiny - they don't stick out.

 

[Pheran]

If you change from RAID Z1 to Z2 don't you destroy the pool?

Sent from my SM-T700 using Tapatalk

Correct, switching from RAIDZ1 to Z2 requires destroying and recreating the pool. Also, adding drives that are meant to be in the same vdev would also require destroying the whole thing.

 

[MindBender]

I would consider that to be a pretty sane reason if we were talking about the big, chunky "thumb" drives of yore, but have you looked at the recommended drives such as SanDisk Fit/Ultra Fit? The non-USB plug portion is tiny - they don't stick out.

I haven't thought about that. I like the suggestion, but the drive has been ordered and paid for. I'm still considering to change the order from an Intel DC S3700 into an Intel DC S3500. The first has an Intel controller on it and the second a SandForce controller, a brand that has given me trouble before. But the latter has so much better reviews, even in a side-by-side comparison, and a lot more write cycles, that it makes me hesitate. I know it doesn't matter for this application, but still... Feedback is welcome, as always!

Correct, switching from RAIDZ1 to Z2 requires destroying and recreating the pool. Also, adding drives that are meant to be in the same vdev would also require destroying the whole thing.

I'm gonna go RAIDZ1; I've got all Enterprise Grade drives and I'll put in 1 or 2 of those 8GB archiving drives for backups, so I should be fine.

 

[Ericloewe]

The first has an Intel controller on it and the second a SandForce controlle

SandForce? No, S3500 and S3700 use the same controller hardware (not necessarily firmware, though).