DIY NAS - 1st build

[cpotter638]

I'm planning on building my 1st NAS. NAS will be for home media storage (files, photos, music, video). Most critical data include files, music, photos, & family home movies. These will be backed up external to the NAS. Other movies will not be backed up. Have a separate A/V distribution system (Control4 with Dune media server). NAS will interface with this and home PC's (both Windows7 & Mac). Over the past month, I've extensively researched both software and hardware. Current setup in mind:

FreeNas 0.8 with ZFS. Raidz2
Initially 6 HD (2 redundant). Want the ability to expand the system to a toal of 12 HD down the road. I have purchased the inital set of drives (6x 2.0TB Seagate Barracuda Green)

Other items to be selected:

Motherboard:
Intel LAN is a must. At least 1, possibly 2.
Planning on LGA 1155 socket.
I initially wanted to obtain 6x Sata3 (6.0). However, to my knowledge, there are no MOBO with both Intel LAN & 6x Sata 3 (6.0). Addtionally, Sata3 may or may not provide any real benefit to the system I'm building (info I've found and have been told online is confilicting).
I've read many positive things about Supermicro MOBO's. So, I am now planning on going this route. I have to decide the specific MOBO model. Supermicro MOBO's I've been considering:
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182252
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182254
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182262
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182251

Main differences in above boards:
Some are 6x Sata2. Some are 4x Sata2 / 2x Sata3.
Mixture of Intel LAN (Intel 82579LM & Intel 82574L)
IPMI 2.0 (Some have it, some don't)

Questions re: MOBO:
Is there one Supermicro board in particular that stands out? One of the above 4, or something else?
How useful is IPMI 2.0 (I know next to nothing about this).
Type of Intel LAN matter?

CPU:
Planning on cheapest option that works with Supermicro board.
Perhaps i3-2100.
I am unsure, but the Pentium chips may also work with the above Supermicro MOBO's. Perpahs a G620. If a G620 will work, will this be enough power? Or stick with the 2100?

Case:
Down to 3 options:
Fractal Design XL
Lian Li PC-A70F
Norco RPC-2212

Norco has hot swap x 12. Nice when a drive fails. But it is $100 more. Other 2 are similar.
Main qualifications for case: Protect / cool drives (#1). Second, easy to replace drives when upgrading or failure.
It seems to me that Fractal and Lian Li case would be better with dust removal. As such, are hard drives better protected in these 2 cases? Recomendations?

Memory:
Plan on 8Gb.
Some of the Supermicro boards are ECC only memory. Some appear to be ECC "Supported". Cost difference is small. Is ECC memory prefered?

Power supply:
Not yet researched.


See any compatibility problems with above hardware and FreeNAS 0.8 with ZFS? Thanks in advance for all the help.
Chad

 

[sunflashx]

Supermicro tip for the boards you're looking at, they only take UNBUFFERED ECC such as:
http://www.newegg.com/Product/Product.aspx?Item=N82E16820139262

Any processor these MBs will accept will be enough.

If you don't mind stepping down to 10 drives Antec makes a really nice case that takes 6 internal drives, and can fit 4 more with a cheap 3 to 4 adapter. 5 star review out of 2660 reviews on newegg. I own 2 of them. $84 shipped for both items.
http://www.newegg.com/Product/Product.aspx?Item=N82E16811129042
http://www.newegg.com/Product/Product.aspx?Item=N82E16817993002

Motherboard, I'd get the X9SCM-F. IPMI, 2 GigE, 2x SATA3, 4x sata2. IPMI mandatory on my next NAS, every time I want to mess with it I have to drag it out of the basement and set it up in my office to get a monitor on it.
http://www.newegg.com/Product/Product.aspx?Item=N82E16813182253

To the best of my knowledge only SSD drives really take advantage of SATA3, and none of it matters if you're only doing SMB file sharing.

 

[Milhouse]

I recently did some basic disk performance tests on an N36L (AMD dual core 1.3GHz "Atom-class" processor) with 4GB RAM and 4x1TB Samsung drives, comparing RAIDZ1/RAIDZ2, 512B and 4KB sectors, with and without read ahead (vfs.zfs.prefetch_disable=0):

Code:

WRITE     RAIDZ1          RAIDZ2
512B      212,143,286     148,067,931
4K        211,027,799     145,136,615

READ      RAIDZ1          RAIDZ2
512B      134,556,578     101,815,843
4K        191,722,812     108,278,443
512B+RA   417,207,180     320,201,198
4K+RA	  422,350,077     341,636,028

As you can see, in terms of read performance these SATA2 disks should easily max out a single and (for reads at least) even dual GigE network connection. Read performance should only increase with additional drives (and more powerful CPU, with more memory). Write performance may increase with a faster CPU etc.

So no, I wouldn't worry about SATA3, the only benefit you may see here is when scrubbing drives which may finish a little earlier than with SATA2 drives, but other than that I don't think you will see any significant benefit.

If you can afford it, ECC would be preferred.

 

[cpotter638]

Thanks for the help Sunflashx & Milhouse.

Why the X9SCM-F (over the X9SCL+-F)? Extra PCI-E & 2x SATA 6.0 worth the $20?

Re: memory, I'll go ECC. I've read the Supermicro boards are picky with memory? Know picky how? Picky = need ECC? Or picky with Brand? Should I stick with the Kinston Sunflashx suggested, or any brand OK?

Re: hard drives & case:
Only problem is I already have the 6 drives. If I'm thinking about things correctly, when adding drives in the future, would want to create a new vdev with raidz2 (matching redundancy of initial vdev). If adding just 4 drives, this would yield me 2 drives of data 2 of redundancy. As such, would like to stick with a case that can hold 12. So case selection still up in the air.

Thanks for the comments & data re: Sata3.

Thanks again.

 

[cpotter638]

In addition to questions on most recent post, I have an additional question:

Quote from FAQ section:

"21) I'm trying to reflash my USB flash disk to update FreeNAS but... various problems, what can I try?
For some reason reformatting, zeroing disk doesn't always seem to work. Try wiping your disk described in #20 above. IMPORTANT, versions after 8.01 beta3 require a minimum of 2GB flash drive. Some drives say they're 2GB but are actually 1.8GB when formatted. They appear to flash ok, but then fail to boot."


Given above, I'm probably better off going with a 4Gb USB stick. Being that OS will be installed on this, I certainly want this to be stable / reliable. Are there any favored brands or models of USB stick for FreeNAS? Also - I assume it is recommended to purchase 2x USB flash drive and back up OS to 2nd flash drive?

Thanks again for all the help.

 

[Milhouse]

Given above, I'm probably better off going with a 4Gb USB stick. Being that OS will be installed on this, I certainly want this to be stable / reliable. Are there any favored brands or models of USB stick for FreeNAS? Also - I assume it is recommended to purchase 2x USB flash drive and back up OS to 2nd flash drive?

I've never had a problem with 2GB Verbatim MicroDrive USB memory, and I've used quite a few without fail.

I use a cron job to regularly back up the /data/freenas-v1.db to a directory on my storage pool (runs once a week). Should I ever need to replace a failed USB memory stick it's just a case of dd'ing the original xz image to the flash memory and then uploading the most recent freenas-v1.db configuration via the GUI.

A second (even third) memory stick is useful for upgrades and testing nightlies - given the low price, this is eminently feasible. Using the dd command under Unix/Linux to make a backup of your current boot disk prior to upgrading is always a good idea, although as long as you have a current config not really essential (provided you accept the slightly increased time to recover a working system when using only a config).

 

[cpotter638]

Millhouse - Thanks for the info re: flash drive. Very helpful.



Trying to finalize decision on MOBO. Favoring the X9SCM-F.

Re: X9SCM-F:
Uses 82579LM LAN. Apparently, this is a newer Intel NIC. Some OS's do not have supporting drivers. Is 82579LM supported in FreeNAS?

Options re: CPU
Confirmed with Supermicro - Core i3 series, Pentium G series, Xeon E3 series, & Intel Celeron series supported by X9SCM-F. However (per Intel Ark) - only Pentium 350 (not available to public) and Xeon E3 series support ECC RAM.

My question - ECC RAM obviously recommended for business servers. But how beneficial is this for a home NAS? Is it worth the added processor purchase cost (Xeon E3 series) as well as added energy consumption (E3 series with Max TDP of 80W).

Thanks again.


Thanks.

 

[Milhouse]

My question - ECC RAM obviously recommended for business servers. But how beneficial is this for a home NAS? Is it worth the added processor purchase cost (Xeon E3 series) as well as added energy consumption (E3 series with Max TDP of 80W).

For a home NAS? No, probably not unless the cost difference is less than £60/$100.

 

[cpotter638]

For a home NAS? No, probably not unless the cost difference is less than £60/$100.

I plan on sticking with the Supermicro X9SCM-F MOBO. Main advantages are Supermicro brand (from what I've read) and IPMI 2.0 support. If I understand this correctly:
X9SCM-F only takes ECC RAM. So would have to buy ECC RAM (added cost of ECC RAM is negligible to me)
X9SCM-F supports several Intel CPU, including both Celeron & Xeon
Only socket 1155 processors that supporte ECC RAM are the Xeon's

Main 2 options:
Intel Xeon E3-1230
Supported by X9SCM-F
Supports ECC RAM.
Max TDP = 80
$239

Celeron G540
Supported by X9SCM-F
Does not support ECC RAM. If I undestand correctly, I would have to buy ECC RAM to be compatible with MOBO; however, ECC benefits could not be utilized by CPU
Max TDP = 65
$67

Given usage of home NAS, which option recommended?

 

[b1ghen]

Performance wise I'm sure the Sandy Bridge Celeron will be enough, the only reason to go with the Xeon is ECC support. Too bad the Pentium 350 is not available to buy for end users.
I might be tempted to throw the Pentium G620T or G630T in the mix for a low power system though since I am allergic to anything with the Celeron name.

 

[survive]

Hi cpotter638,

Remember that the wattage measurements show the maximum watts the chip can\will draw when you are running it full bore.....I would bet that most of the time the chips would be within a couple of watts of each other.

Supermicro boards are picky about RAM so I wouldn't get the super OC gamers memory and expect it to work. Honestly you want memory that is more generic than anything...look for memory that runs at normal voltage with normal timings or take a look at Supermicro's site for a list of approved memory & get something off that list. You can find the list for each board under the "Links & Resources" on the right side of each boards page.

Here is Supermicro's page about the difference in the NICs:

http://supermicro.biz/support/faqs/faq.cfm?faq=11847

-Will

 

[cpotter638]

Am I making the following correct assumptions:

CPU such as Pentium G630T is not "ECC supported" per Intel Ark (confirmed).
X9SCM-F compatible with G630T (confirmed with Supermicro)
X9CSM-F needs ECC RAM to boot


So, even thought G630T is not "ECC supported", it will boot with ECC memory, correct? Only problem is that ECC feature of memory will not be used by CPU (but it will boot), correct?

 

[cpotter638]

Will:

Thanks for the info. If I'm following you, overall wattage savings between Celeron / Pentium & Xeon is minimal (given chip will be primarily near "idle speed")? So only real difference is pricing ($150 to $200 more) for Xeon CPU. Benefits of ECC ram worth $150-200 on a home NAS?

I've read previously about Supermicro and picky memory. I was planning on going with either Crucial memory (on Supermicro's approved list) or comparable Kingston version (seems to work well with Supermicro MOBO on many forum postings I've seen). Initially, was looking at getting 1x 8Gb stick. However, 2x 4Gb sticks are cheaper (often significantly cheaper). Is CPU more efficient with 2x 4Gb than 1 x 8Gb?

Thanks for info re: NIC's.

 

[b1ghen]

Benefits of ECC ram worth $150-200 on a home NAS?

I personally went the AMD route with a ASUS MB with ECC support and an Athlon II CPU since I wanted ECC on a limited budget, it has higher power consumption but I did the math and to make up for the difference in price would take longer than the life of my NAS anyway. The other downside is that finding an AMD board with Intel NIC's is basicially impossible so if you really want good performance you need to get an Intel NIC as well. I just kept with the Realtek on the AMD so far and it gets the job done but is very choppy in performance. Will probably put in an Intel NIC eventually.

Is CPU more efficient with 2x 4Gb than 1 x 8Gb?

You will have double the memory bandwidth since those CPU's use dual channel memory architecture, it will not make any noticeable difference in most cases though. But to utilize the platform to the max modules should be installed in pairs.

 

[jgreco]

Some quick comments. Note that I've been building VMware ESXi nodes out of the X9SCL+ boards with E3-1230. My motivations for picking that particular board are different than a NAS-builder's... out-of-the-box compatibility with VMware drove me away from the boards with the newer Intel ethernet part. The C204-based boards (extra slots, SATA-III) might be more desirable for your use.

The Supermicro IPMI is really nice. Remote management capabilities have been required around here for years, we used to have to arrange it externally with KVM gear, but these days, an extra twenty bucks gets you the version of the board that can be remotely controlled, remotely monitored, etc. It's not as slick as HP iLO, but it is fully functional - though you might need a Windows PC for some of it. Being able to load a CDROM into the machine from your laptop or desktop is really nice.

Be exceedingly careful with memory. You will not find an affordable (for a home user) way to put more than 16GB (4x4GB) in this box, the 8GB sticks are around $400/EACH. If you think you've found something that'll work, you're wrong. We're all hoping to see RAM prices for compatible memory drop this next year. I do suggest that you use the Kingston, Crucial, etc., memory configurators to help you identify compatible parts, and be willing to pay a mild premium for the recommended parts. You do not need to use Supermicro's "tested and listed" parts but you do need to listen to this advice, or you will end up burning money in return shipping and restocking fees, see? You will NOT be using 1x8GB as a memory configuration unless you also burn hundred dollar bills for kicks.

As for power. You are in for a treat. More details in the hardware forum, see detailed post at http://forums.freenas.org/showthread.php?3971-I-finally-got-around-to-setting-one-of-these-up . But I built some X9SCL+'s with E3-1230's, 16GB RAM, an extremely efficient power supply, and a few other thoughtfully-selected components like variable speed fans. The base platform ended up running at about 41 watts, idle, 85ish at full tilt CPU. Not too useful to have a diskless box, so after adding a dual ethernet, an LSI 8-port RAID controller, some Seagate Momentus XT 500GB's, some 120GB SSD's, and some 240GB SSD's, I had an 8-drive platform for VMware that idles at about 60 watts, peaks a bit over 100 watts.

But you have to bear in mind that I blew some money on energy efficiency. Spent $150/each on the most efficient lower wattage power supply I could get my hands on, and then some money on good four-pin fans, which actually idle at a speed low enough to cause the motherboard IPMI to alarm, so you've got to twiddle those settings too...

My take on using "lower power" CPU's is that it probably isn't worth it. This thing, running with 20 VM's, only barely inches up by 10 watts. I don't intend to heavily load these hosts, and so they have a good amount of headroom for when a VM does need some extra speed. For a NAS, this would translate to the ability to run things like high levels of compression, and future stuff like encryption and dedupe without having to worry about it. I keep thinking about whether to order in one of the low power CPU's and I keep having trouble convincing myself to do it.

 

[cpotter638]

b1ghen:
Thanks for the comments re: AMD / ECC & memory. I will probably continue with 2 memory sticks.

The Supermicro IPMI is really nice. ...though you might need a Windows PC for some of it. Being able to load a CDROM into the machine from your laptop or desktop is really nice.

How easy is it to configure IPMI? I briefly looked online last night, but didn't find a "cookbook" approach. This will be my 1st experience with a server-type box and FreeNAS. Before I purchase a MOBO with IPMI, want to make sure it is not too difficult to set up.

Be exceedingly careful with memory. You will not find an affordable (for a home user) way to put more than 16GB (4x4GB) in this box, the 8GB sticks are around $400/EACH. If you think you've found something that'll work, you're wrong. We're all hoping to see RAM prices for compatible memory drop this next year. I do suggest that you use the Kingston, Crucial, etc., memory configurators to help you identify compatible parts, and be willing to pay a mild premium for the recommended parts. You do not need to use Supermicro's "tested and listed" parts but you do need to listen to this advice, or you will end up burning money in return shipping and restocking fees, see? You will NOT be using 1x8GB as a memory configuration unless you also burn hundred dollar bills for kicks.

Certainly don't mind paying a mild premium for memory that actually works. I was planning on using either the Crucial memory (tested and approved by Supermicro) or the equavilent Kingston memory (verified on the Kingston site and also working for other users with same MOBO per forum posts).

Found 8Gb stick of supposedly compatible Kingston memory for $175 (approved per Kingston website):
http://www.provantage.com/kingston-technology-kvr1333d3e9s-8g~7KIN91M6.htm

Still more than 2x price of 2x 4Gb stick. Think I'll stick with 2x 4Gb (probably the Crucial memory).

As for power. You are in for a treat. More details in the hardware forum, see detailed post at http://forums.freenas.org/showthread.php?3971-I-finally-got-around-to-setting-one-of-these-up . But I built some X9SCL+'s with E3-1230's, 16GB RAM, an extremely efficient power supply, and a few other thoughtfully-selected components like variable speed fans. The base platform ended up running at about 41 watts, idle, 85ish at full tilt CPU. Not too useful to have a diskless box, so after adding a dual ethernet, an LSI 8-port RAID controller, some Seagate Momentus XT 500GB's, some 120GB SSD's, and some 240GB SSD's, I had an 8-drive platform for VMware that idles at about 60 watts, peaks a bit over 100 watts.

But you have to bear in mind that I blew some money on energy efficiency. Spent $150/each on the most efficient lower wattage power supply I could get my hands on, and then some money on good four-pin fans, which actually idle at a speed low enough to cause the motherboard IPMI to alarm, so you've got to twiddle those settings too...

My take on using "lower power" CPU's is that it probably isn't worth it. This thing, running with 20 VM's, only barely inches up by 10 watts. I don't intend to heavily load these hosts, and so they have a good amount of headroom for when a VM does need some extra speed. For a NAS, this would translate to the ability to run things like high levels of compression, and future stuff like encryption and dedupe without having to worry about it. I keep thinking about whether to order in one of the low power CPU's and I keep having trouble convincing myself to do it.

Good news re: power. Lower power CPU's also better on the budget. However, it seems that if I want Intel LAN, ECC ram, & IPMI I'll have to spend more on the processor (such as E3-1230). What about E2-1220 vs 1230? From what I can tell, $30 more gets you 2x threads. I presume this is worth it as I don't think I've seen a forum post yet with anyone running a 1220.

I'll have to search for an efficient low watt 2U power supply to work with the RPC-2212.

Thanks again for all the help.

 

[jgreco]

IPMI is abused to mean a few things. This board sports a web server that will report sensor readings (fan, temp, voltages, etc), the event log, allow you to configure alerts, date and time (NTP client too), local, LDAP, AD, or RADIUS for user accounts to access it, network settings, SMTP settings (for alerts etc), SSL cert, and ports. Now none of that involves the operating system you're running; this is all independently implemented by the system board BMC/IPMI system. It works when the server is powered off (not unplugged though haha). You can also get Java-based KVM-over-IP, or serial-over-LAN, and remote power control. There are also options for remote media, like you can mount a DVD on your laptop, log in with the KVM-over-IP to the server, and then tell the Java client to present that DVD to the server, which then appears as a USB device to the server.

Now I've kind of focused on the newbie user aspect here. It basically means you might never need to connect a keyboard and mouse to your machine. But the IPMI system also allows for direct querying and configuration of your system. For example, ipmitools will allow you to modify the thresholds that the BMC uses to identify alarm conditions. For serious uses, this is where the magic is at. You need to be able to twiddle your system's bits. ;-)

Anyways. Interesting on the 8GB stick. Looks like it's brand spanking new. At least it isn't *so* outrageously priced.

No thoughts on the 1220 vs 1230 that you haven't already had. Pretty sure we eliminated the 1220 because we wanted the better featureset of the 1230. For a NAS, I think the question would be whether or not it'd be smarter to go for one of the 20W TDP CPU's that are available. I know I've talked incessantly about that but I have yet to convince myself to do anything about it. We are very happy with the way these ESXi boxes have turned out, both power-wise and performance-wise. For what you pay for a server, the ability for the CPU to rev up, burning more watts when there's demand for service, that's just a total win, and my perception is that the arbitrary limit placed on the system by a 20W CPU does not make sense financially, when the system already costs so much to build to begin with.

 

[cpotter638]

IPMI is abused to mean a few things. This board sports a web server that will report sensor readings (fan, temp, voltages, etc), the event log, allow you to configure alerts, date and time (NTP client too), local, LDAP, AD, or RADIUS for user accounts to access it, network settings, SMTP settings (for alerts etc), SSL cert, and ports. Now none of that involves the operating system you're running; this is all independently implemented by the system board BMC/IPMI system. It works when the server is powered off (not unplugged though haha). You can also get Java-based KVM-over-IP, or serial-over-LAN, and remote power control. There are also options for remote media, like you can mount a DVD on your laptop, log in with the KVM-over-IP to the server, and then tell the Java client to present that DVD to the server, which then appears as a USB device to the server.

Now I've kind of focused on the newbie user aspect here. It basically means you might never need to connect a keyboard and mouse to your machine. But the IPMI system also allows for direct querying and configuration of your system. For example, ipmitools will allow you to modify the thresholds that the BMC uses to identify alarm conditions. For serious uses, this is where the magic is at. You need to be able to twiddle your system's bits. ;-)

Thanks for the detailed info re: IPMI. If I'm following you, IPMI will be set up via MOBO jumbers, BIOS, & Ipmitools. Is basic config relatively straight forward for a newbie?

No thoughts on the 1220 vs 1230 that you haven't already had. Pretty sure we eliminated the 1220 because we wanted the better featureset of the 1230. For a NAS, I think the question would be whether or not it'd be smarter to go for one of the 20W TDP CPU's that are available. I know I've talked incessantly about that but I have yet to convince myself to do anything about it. We are very happy with the way these ESXi boxes have turned out, both power-wise and performance-wise. For what you pay for a server, the ability for the CPU to rev up, burning more watts when there's demand for service, that's just a total win, and my perception is that the arbitrary limit placed on the system by a 20W CPU does not make sense financially, when the system already costs so much to build to begin with.

You've talked me into the 1230.




Re: Case:

Want 12 hot swap. Fill with 6 drives for now. Ability to add 6 more in future. Was favoring Norco RPC-2212. Only problem - RPC-2212 only takes 2U PSU (not standard PSU). So:

Anyone have a suggested 2U power supply for my build? Was looking for something that is energy efficient, but not extremely cost prohibitive. Perhaps something "Gold rated". However, not coming up with much given need for 2U PSU if I go the RPC-2212 route.

OR

Should I change to a 4U Norco case (such as RPC-4220 or 4216) and go with standard PSU? Obviously 16 or 20 drives is more than I need. But this does allow standard PSU, allows full height add-on cards (I think), and probably makes the build easier given more room.

Thanks again.

 

[survive]

Hi cpotter638,

If you got the space, I'd go for the bigger case. It's always nice to have some room to work, it should support bigger fans for quieter cooling, no need to worry about finding low-profile cards...

If you are getting a Supermicro board you might also want to take a look at their cases. They are expensive, but go together very nicely.

-Will

 

[b1ghen]

If you have the space for it I would go for a 4U unit, preferably one which supports 24 drives since then you could increase your storage pool with another 2 vdevs of 6 disks if needed in the future. Or easily add some cache or ZIL drives to speed things up if needed as well. More expansion options is never a bad thing.
Also if the PSU should break sometime in the future finding a 2U PSU with short notice is probably not the easiest.
And as survive said using normal profile cards and larger diameter fans is always nice as well, especially the fans unless it's going to be in a data center environment where the noise is a non issue.