Advice needed for a (potential) new FreeNAS user...

[Charles Frank]

Hi folks,

First time poster, though I've been reading these forums for a few weeks now, trying to make up my mind - which is where I hope some of you kind folks come in :) Please bear with me, this might get a little long-winded...

For a few years now I've been using a NAS in the form of a Netgear ReadNAS Duo - but to say it's getting a little long in the tooth is putting it mildly - I'm a big-time consumer of media and my collection outstripped it a LONG time ago (I currently have nearly 200 blu-ray disks of media burned sitting under my desk...) so I want to upgrade to something with a LOT more grunt. Initially I looked at offerings from Synology, then moved on to QNAP (I liked the fact their boxes had a lot more processing power and memory than Synology's for transcoding and plug-ins, plus HDMI out) but then I ran across these forums and you lot scared me lol - talk of potential catastrophic data loss if you weren't using ECC RAM, RAID arrays failing etc. made me look long and hard at the options (I don't have the option of replicating my data off-site or even to another NAS so data integrity and RAID robustness became my major concerns at this point).

I went back to looking at my 'off-the-shelf' options from Synology and QNAP for more Enterprise-level gear - and while they exist (I've been seriously considering a QNAP TS-EC880 Pro) they are NOT cheap - that QNAP TS-EC880 Pro is over 1500 pounds bare - add another 140 pounds for an additional 16Gb ECC RAM and 6x3Tb HGST NAS Drives for around 625 pounds and you're looking at the better part of 2500 pounds...the Synology options are even more costly. Now I have the money - but it would be a serious investment in a serious bit of kit. That's the downside (as I see it). The upside is that it IS an off-the-shelf solution - plug the memory in, plug the drives in, switch it on and away you go - all the hard work has been done for you, tech support is a phonecall or e-mail away if you run into any problems and with a major company supporting it the unit has an easy-to-use GUI-based Linux OS and there are plenty of plug-ins available and it SHOULD run flawlessly for years.

Now on to FreeNAS. The downside is that YOU need to do the work. Now I have no problems building it - I'm a computer tech by profession so I love tinkering :) - but I'm nowhere near as confident with the FreeNAS OS and software installation itself - I have ZERO *ix experience (unless you consider installing a couple Linux distros via their GUIs and typing the odd command into an Apple's terminal app 'experience') so my main concern is spending money on the parts and then NOT being able to get it up and running in a stable manner (or get it running at all...). The upside of FreeNAS is cost - it's a lot cheaper to get a decent - powerful even - server built. And with that in mind, here's what I priced up as my build machine:

Fractal Design Node 804 case £75.00
Supermicro X10SL7-F £236.00
Kingston ValueRAM KVR16LE11/8KF x2 £133.00
Seasonic S12G-500 PSU £75.00
HGST Deskstar NAS 3Tbx6 £625.00

As for the processor, I've got 2 options - both are OEM Xeon parts that I can get. The first option is an Intel Xeon E3-1240L V3 - this is a 25W(!) part that could be passively cooled. The second option is an Intel Xeon E3-1265L V3 - this is a 45W part that would need a (modest) active cooler. Price-wise there's not a lot in it - the 1240 is £213 and the 1265 is £222.

All-in (with the more expensive processor) I'd be looking at £1366. Add another £34 for bits and bobs to round it up to a nice number and let's call it £1400 - or less than that QNAP is BARE...

...So there's my dilemma - right now I've basically talked myself into an impasse - I want the QNAP for ease of use and support, but I want FreeNAS for the cost savings and the satisfaction of building my own system - and I just can't make my damn mind up - can any of you kind folks help me out here?

Merry Christmas to one and all!

Charles.

 

[danb35]

The hardware you've picked sounds good, though the Kingston RAM is getting a bad reputation, and most here recommend avoiding the L CPUs, as they don't really save you any power. You don't need an integrated GPU, which I think the 1265 has.

Something else to consider, although shipping would be expensive, is the FreeNAS mini. That'll give you the ease of use and support, considerable cost savings compared to the QNAP, and all the FreeBSD and ZFS goodness.

 

[Ericloewe]

The hardware you've picked sounds good, though the Kingston RAM is getting a bad reputation, and most here recommend avoiding the L CPUs, as they don't really save you any power. You don't need an integrated GPU, which I think the 1265 has.

Something else to consider, although shipping would be expensive, is the FreeNAS mini. That'll give you the ease of use and support, considerable cost savings compared to the QNAP, and all the FreeBSD and ZFS goodness.

This pretty much. You'll need active cooling for the HDDs anyway, so might as well go with a non-underclocked processor and have all the performance you're likely to need. The iGPU is absolutely useless on a server board, so save some cash and get a Xeon without a GPU.

Regarding RAM, I recommend you read the Supermicro X10 RAM guide - it explains the whole situation.

FreeNAS does require a bit more work than an off-the-shelf thing, but it's not harder. You're just expected to make a lot of decisions that were hidden from you or were simply not presented - some of them rather important, like ECC RAM.

Oh, one more thing: The Seasonic G-Series is a bit better than the S12Gs for only a little more cash. Best PSUs up to 500W (well, excluding the Seasonic Platinum Fanless models - those have the Seasonic Platinum goodness in a smaller capacity, plus the Seasonic Platinum price).

 

[Charles Frank]

OK, can swap out that Kingston kit for a Crucial one - CT4484984 - that they guarantee works with that motherboard. It's a bit more expensive and I can only seem (with a quick search) to buy it direct from Crucial UK. And a standard Xeon is doable - though I was looking at making as quiet and energy-efficient a server as possible, hence the L chip - though I agree the iGPU of the 1265 would be fairly pointless. The standard chips, however, use between 80 to 84W - significantly more than the 1240L's 25W. Was trying to get the machine below 100W if possible (one of the advantages of the off-the-shelf NAS boxes is they are very energy efficient).

One other thing - while reading these forums some users reported being a bit concerned about how hot the LSI RAID chip was getting - is that really an issue? The case I am looking at has good airflow which should help, but how hard would it be to either:

a) Replace the existing heatsink (assuming I could find a suitable - possibly actively-cooled - replacement AND I could get the existing heatsink off without prying the chip off the motherboard...)

b) Jury-rig a spare fan inside the case to direct extra airflow over the LSI chip (assuming that would even be effective).

 

[Charles Frank]

The hardware you've picked sounds good, though the Kingston RAM is getting a bad reputation, and most here recommend avoiding the L CPUs, as they don't really save you any power. You don't need an integrated GPU, which I think the 1265 has.

Something else to consider, although shipping would be expensive, is the FreeNAS mini. That'll give you the ease of use and support, considerable cost savings compared to the QNAP, and all the FreeBSD and ZFS goodness.

I did look at the FreeNAS mini - unfortunately it only has 4 HDDs and I was looking at a mininum of a 6x3Tb drive RAID-6 (if I bought a QNAP) or RAID-Z2 (if I go FreeNAS) so I can get plenty of storage and 2 drive redundancy (should the proverbial hit the fan...).

 

[Charles Frank]

I think the thing that's bothering me the most about going the FreeNAS route is the initial set-up and configuration - compared the what I've seen (and tried with the online demos) it at least SEEMS a lot more daunting than either Synology's DSM or QNAP's QTS OSes... Some of the build videos on Youtube show even experienced users having to faff around with configs and command line stuff just to get their machines booting up, let alone configured - THAT's what's really making me hold fire on going down the FreeNAS route - a lack of confidence I can get it up and running. One of the reasons I've picked the Supermicro motherboard is because not only have a lot of users had positive experiences with it, but the FreeNAS OS itself seems to have fewer problems with that particular combination of hardware...

 

[Ericloewe]

OK, can swap out that Kingston kit for a Crucial one - CT4484984 - that they guarantee works with that motherboard. It's a bit more expensive and I can only seem (with a quick search) to buy it direct from Crucial UK. And a standard Xeon is doable - though I was looking at making as quiet and energy-efficient a server as possible, hence the L chip - though I agree the iGPU of the 1265 would be fairly pointless. The standard chips, however, use between 80 to 84W - significantly more than the 1240L's 25W. Was trying to get the machine below 100W if possible (one of the advantages of the off-the-shelf NAS boxes is they are very energy efficient).

One other thing - while reading these forums some users reported being a bit concerned about how hot the LSI RAID chip was getting - is that really an issue? The case I am looking at has good airflow which should help, but how hard would it be to either:

a) Replace the existing heatsink (assuming I could find a suitable - possibly actively-cooled - replacement AND I could get the existing heatsink off without prying the chip off the motherboard...)

b) Jury-rig a spare fan inside the case to direct extra airflow over the LSI chip (assuming that would even be effective).

My server and my (ancient - Core 2 era) pfSense Router draw less than 100 Watts at idle to low loads.

Don't be deceived by TDP - that's only the thermal limit. The processors will draw roughly the same power at idle, L or no L (I say roughly because no two CPUs are the same). Difference is that the L will throttle lower than the non-L, artificially limiting performance (and thus power consumption at load - in exchange for crummier performance.)

Given decent airflow (which should be there, given the hard drives you'll be installing), the SAS controller will be cool enough. No need for jury-rigging.

I think the thing that's bothering me the most about going the FreeNAS route is the initial set-up and configuration - compared the what I've seen (and tried with the online demos) it at least SEEMS a lot more daunting than either Synology's DSM or QNAP's QTS OSes... Some of the build videos on Youtube show even experienced users having to faff around with configs and command line stuff just to get their machines booting up, let alone configured - THAT's what's really making me hold fire on going down the FreeNAS route - a lack of confidence I can get it up and running. One of the reasons I've picked the Supermicro motherboard is because not only have a lot of users had positive experiences with it, but the FreeNAS OS itself seems to have fewer problems with that particular combination of hardware...

No, that is not my experience.

The CLI is only for a very limited set of operations (manually checking drive SMART data, getting text-based pool status... and that's basically it).

There's also a new initial setup wizard, which is said to help new users (haven't checked it out yet).

 

[Charles Frank]

My server and my (ancient - Core 2 era) pfSense Router draw less than 100 Watts at idle to low loads.

Don't be deceived by TDP - that's only the thermal limit. The processors will draw roughly the same power at idle, L or no L (I say roughly because no two CPUs are the same). Difference is that the L will throttle lower than the non-L, artificially limiting performance (and thus power consumption at load - in exchange for crummier performance.)

Given decent airflow (which should be there, given the hard drives you'll be installing), the SAS controller will be cool enough. No need for jury-rigging.



No, that is not my experience.

The CLI is only for a very limited set of operations (manually checking drive SMART data, getting text-based pool status... and that's basically it).

There's also a new initial setup wizard, which is said to help new users (haven't checked it out yet).

Dammit Eric, you're starting to talk me around to giving this a go lol ;) BTW, just noticed your sig - that Sharkoon case looks ok - I was looking for a midi tower with LOTS of 5.25" bays for Icy Dock or StarTech hotswap caddies before I looked at the Fractal Design case - I'd rather a midi-tower with hotswap bays just for ease of access and swapping drives (if necessary) but there doesn't seem to be many decent ones out there that can support 2-3+ caddies. I had been looking at the Aerocool VS-92 Black Edition Midi-Tower too. I'm guessing with your Sharkoon case you had to pop out one of the front fans to free up space for one of those Icy Dock bays? What is the fan noise like with those hotswap bays btw - they use 80mm fans, right? What's their airflow like over the drives?

Thanks for your help :)

 

[Ericloewe]

After a ton of research, I was left with a few options for a case with tons of 5.25" bays:

• Sharkoon T9
• Zalman MS800
• A Lian-Li full tower case that would end up costing as much as a Supermicro rack chassis
• CM Storm Trooper (extra large, but still only 9 bays)
• Antec Ninehundred and twelvehundred

I ended up limiting it to the first two, as the Lian-Li was crazy expensive, the Storm Trooper was unnecessarily large and the Antecs were full of quirks due to their ancient design.

In the end, the MS800's upper opening made me hesitate, as the plan was to have a rackmount-like cooling setup. In retrospect, it might be a better option, though. The backplate opening is much larger, which should allow the server boards to receive backplates without removal. I also had trouble installing the hotswap bays in the Sharkoon T9, but it's hard to tell if it's the chassis' or the cages' fault. The fan controller on the MS800 is totally redundant, though.

Regarding cooling, I removed all fans and populated the two remaining positions (after the drive cages) with Noctua NF-F12 IndustrialPPC 3000 PWM fans. At full blast, they're crummy hair dryer loud. They're quite reasonable once the motherboard is properly controlling them. Of course, there's also the two 80mm fans from the cages. They seem to be a decent ball bearing model and they're not too loud. Airflow has been good enough to keep the drives at a maximum of 32ish degrees Celsius during burn-in testing (I was pleasantly surprised, as I was expecting to have to buy more powerful fans). I also covered the (ventilated) PCI-e slot covers with plastic, since there's nothing in the area that needs cooling.

Relevant photos here.

 

[Charles Frank]

After a ton of research, I was left with a few options for a case with tons of 5.25" bays:

• Sharkoon T9
• Zalman MS800
• A Lian-Li full tower case that would end up costing as much as a Supermicro rack chassis
• CM Storm Trooper (extra large, but still only 9 bays)
• Antec Ninehundred and twelvehundred

I ended up limiting it to the first two, as the Lian-Li was crazy expensive, the Storm Trooper was unnecessarily large and the Antecs were full of quirks due to their ancient design.

In the end, the MS800's upper opening made me hesitate, as the plan was to have a rackmount-like cooling setup. In retrospect, it might be a better option, though. The backplate opening is much larger, which should allow the server boards to receive backplates without removal. I also had trouble installing the hotswap bays in the Sharkoon T9, but it's hard to tell if it's the chassis' or the cages' fault. The fan controller on the MS800 is totally redundant, though.

Regarding cooling, I removed all fans and populated the two remaining positions (after the drive cages) with Noctua NF-F12 IndustrialPPC 3000 PWM fans. At full blast, they're crummy hair dryer loud. They're quite reasonable once the motherboard is properly controlling them. Of course, there's also the two 80mm fans from the cages. They seem to be a decent ball bearing model and they're not too loud. Airflow has been good enough to keep the drives at a maximum of 32ish degrees Celsius during burn-in testing (I was pleasantly surprised, as I was expecting to have to buy more powerful fans). I also covered the (ventilated) PCI-e slot covers with plastic, since there's nothing in the area that needs cooling.

Relevant photos here.

Wow - VERY tidy installation there Eric :) The Icy Box bays get good reviews on Amazon and look the part, as do StarDock bays - but it looks like I'm not as limited with cases as I first thought - like I said, hotswap bays in a decent midi-tower would be my preference over the 804 case, more for ease of access and (possibly) better cooling than anything else...

 

[Ericloewe]

Wow - VERY tidy installation there Eric :) The Icy Box bays get good reviews on Amazon and look the part, as do StarDock bays - but it looks like I'm not as limited with cases as I first thought - like I said, hotswap bays in a decent midi-tower would be my preference over the 804 case, more for ease of access and (possibly) better cooling than anything else...

Thank you for the kind words. It was painful - the motherboard and CPU power cables were barely long enough to reach the (unusually-located) connectors, connecting both cages with a single SATA chain required a half-hour's worth of elbow grease.

An advantage of the Sharkoon T9 is that the top can be disassembled, which was useful to get cables through the relatively narrow holes near the top.

It's important to note that hotswap bays are an absolute necessity if you want to hotswap drives. Not because of ease of use, but because regular SATA connectors don't longer pre-charge pins than regular pins - this is needed on device and host side for hotplugging SATA devices (as a crazy person an adventurer a scientist discovered a few months back).

 

[Charles Frank]

OK, couple more questions and I'm probably going to pull the trigger and get this build's components ordered - the QNAP is starting to look less 'necessary' and FreeNAS is looking more appealing :)

For installation purposes I see 9.3 can use 2xUSB 2.0 flash drives of 8Gb+ and mirror them, so if one goes bad your system can still boot. I take it I just need something like this for the motherboard and then a couple (decent) 8Gb+ sticks? Then install to the sticks from a third usb stick in an external socket that's had the .iso written to it (as I assume you can't write the .iso to one of the internal sticks as they need to be wiped during the install process). If one of the sticks DOES go bad, can you simply replace it and have FreeNAS image the good stick onto the new one?

Finally, I understand from reading the forums that it's important to match the IT firmware revision for the LSI chip to the driver version on FreeNAS. I believe FreeNAS is still on driver version 16 in 9.3 - the latest IT firmware and driver for the supermicro board is up to version 19 on their ftp site. Just stick with firmware 16 for now until the devs update the FreeNAS drivers? Or if you're feeling REALLY daring can you have a go at updating the drivers yourself (there is a version 19 FreeBSD driver on the Supermicro site). And yes, as a TOTAL noob I realise I'd probably make a complete mess of this even if it WAS possible (with some heavy-duty Googling on my part) - I'm just curious :)

Oh, one last thing - I understand that IPMI is godly for configuring FreeNAS :) How do I find the IP address of the machine though when I initally boot it up? The supermicro utility can browse my network for it from my Windows machine?

Think that's it :)

 

[Ericloewe]

OK, couple more questions and I'm probably going to pull the trigger and get this build's components ordered - the QNAP is starting to look less 'necessary' and FreeNAS is looking more appealing :)

For installation purposes I see 9.3 can use 2xUSB 2.0 flash drives of 8Gb+ and mirror them, so if one goes bad your system can still boot. I take it I just need something like this for the motherboard and then a couple (decent) 8Gb+ sticks? Then install to the sticks from a third usb stick in an external socket that's had the .iso written to it (as I assume you can't write the .iso to one of the internal sticks as they need to be wiped during the install process). If one of the sticks DOES go bad, can you simply replace it and have FreeNAS image the good stick onto the new one?

Yeah, that's pretty much it - but instead of a third USB drive, you can remotely mount the ISO with the IPMI Java viewer.

You'll have to follow the specific replacement instructions, though (also goes for data drives). Nothing out of the ordinary.

Finally, I understand from reading the forums that it's important to match the IT firmware revision for the LSI chip to the driver version on FreeNAS. I believe FreeNAS is still on driver version 16 in 9.3 - the latest IT firmware and driver for the supermicro board is up to version 19 on their ftp site. Just stick with firmware 16 for now until the devs update the FreeNAS drivers? Or if you're feeling REALLY daring can you have a go at updating the drivers yourself (there is a version 19 FreeBSD driver on the Supermicro site). And yes, as a TOTAL noob I realise I'd probably make a complete mess of this even if it WAS possible (with some heavy-duty Googling on my part) - I'm just curious :)

The current P16 FreeBSD driver is said to be better than the P19 driver, due to changes that were implemented by the community after its release. In any case, updating the drivers involves recompiling FreeNAS.

Oh, one last thing - I understand that IPMI is godly for configuring FreeNAS :) How do I find the IP address of the machine though when I initally boot it up? The supermicro utility can browse my network for it from my Windows machine?

Think that's it :)

I never got the Supermicro utility to work (yay Java /s).
There are two options:

You can set a static IP in BIOS or you can get the IPMI MAC from a sticker and/or the BIOS and look for it on your DHCP server. I'm not sure right now, but you may also be able to see the current IP in BIOS.

 

[Charles Frank]

Yeah, that's pretty much it - but instead of a third USB drive, you can remotely mount the ISO with the IPMI Java viewer.

You'll have to follow the specific replacement instructions, though (also goes for data drives). Nothing out of the ordinary.



The current P16 FreeBSD driver is said to be better than the P19 driver, due to changes that were implemented by the community after its release. In any case, updating the drivers involves recompiling FreeNAS.



I never got the Supermicro utility to work (yay Java /s).
There are two options:

You can set a static IP in BIOS or you can get the IPMI MAC from a sticker and/or the BIOS and look for it on your DHCP server. I'm not sure right now, but you may also be able to see the current IP in BIOS.

Okay, I'm going to go ahead and get the parts ordered and give this a go - if it all goes horribly wrong I'll blame you :p Hehe :) That Icy Dock hotswap cage - do the Supermicro motherboards support the drive fail indicator on them?