BUILD Complete newbie FreeNAS build

[Kosta]

Hi guys,

After reading quite a lot about building, hardware, ZFS, if DIY or prebuilt, I have come to conclusion, that for my needs and my level of knowledge, a FreeNAS is most likely the best solution.

My requirements:
- storage server for all my photos/videos/movies/documents with lots of space and expandability
- reliability with redundancy and failure safety (yes, most important data will be saved externally too, most likely 2nd 4TB NAS, the one I have now)
- possibility of future upgrade without having to invest 1000€+
- low power usage
- ability to run bittorrent and other similar programs (sabnzbd, sickbeard etc...)
- real-time 1080p mkv transcoding to mobile device(s), but one at the time (most probably Plex)
- Gigabit network, speeds around 100 MB/s wished and hopefully achieved (single CAT6 cable)
- sleep ability, but would like the power up ability like the current cheap DLINK NAS has, by accessing it, it wakes up (no WOL)

Currently provided networking hardware is ASUS RT-N66U router, Cisco SE2500 unmanaged switch, fully CAT6 based network, including CAT6 patchbay and wall-connectors (all CAT6 certified), self-made. Quite some knowledge about unix-based commands, not complete newbie here (also have OSX, and using Terminal quite much), however no unix expert. I find my way around when I need stuff, Google is much help and forums too. I am no stranger to tinkering and self-configuring, if it's not overly too complicated.

So, after some research, I have finally set up something that I think should serve for the time come:

Server MB Super Micro MBD - X9SCM - F mATX Intel LGA 1155
Intel Core i3 - 3240T 2,9 GHz ( Ivy Bridge ) Sockel 1155 - tray
Scythe Katana 4 SCKTN - 4000
Kingston D3 16 GB 1333 - 9 REGP Dx4I K2 KVR (KVR13R9D4K2)
ServeRAID M1015 SAS, SATA Controller
4x WD 8,9 cm ( 3,5 " ) 4 TB SATA3 WD40EFRX 64 MB Red
LC - Power 300W LC7300 80 + Silver Shield 12 cm V2,3 retail
NZXT SOURCE 210 ELITE Midi - Tower

This build goes total for 1379€. A little higher than planned, but it should provide for everything I want in my requirements.

So, how does this build sound?
The memory is not found on the compatibility list of Supermicro, however they list very little tested and compatible memory. Is this memory OK? For the beginning, as I start with 16TB pool, I take 16GB RAM. Later possibly expansion to 32GB.
The mainboard, there are many choices, I went for C204, as the board is 20€ cheaper than C202, and I see no disadvantages.
I have chosen the 3240T as it apparently has no disadvantage over 3220T when it comes to power usage on idle, and it provides a little more power when needed. The price difference is also merely 15€, so not really an issue. However, if I take 3220T, I could buy the boxed, and save money on Scythe too. However is 3220T enough for transcoding 1080p MKVs? I read somewhere that 3Ghz is recommended.
M1015 comes recommended as the FreeNAS SATA card, I know I have to mod it, have no problems with that (there are no eBay articles on this one, and none cheaper), cost me €151. I like the idea of not running the stuff off the board.
4 WD Reds are for the beginning (RAIDZ), I would like to start with 1 drive redundancy, and later expand the zpool with additional 4x 4TB drives (2nd RAIDZ). However, if I understand correctly, if I do it like that, I will have 2-drive redundancy if only 1 drive from 1st vdev fails and 1 drive from the 2nd vdev, but if 2 drives in the 1st vdev fail, my whole zpool is going down and I am fucked. How high is the probability that two drives die at the same moment? The whole server will be in the cellar, which is a fairly cold room, so the heat shouldn't be any issue. The case will also be well cooled. Noise is not a problem. Buying 8x WD Reds with 4TB is a no go. Buying 8x 2TB Reds might be an alternative. I wanted to avoid buying 2GB disks if I already plan on going 4TB versions. What do you think?
PSU - that should be quite OK PSU for this server? I read it's recommended and quite good when it comes to power usage.
Case - well, I wanted 19inch rack with slots from the front, but I can't find anything suitable in the priceclass that I can afford.

I hope I planned this carefully and hope you can give me some advice!

 

[cyberjock]

Looks good all around. Here's my comments:

1. RAM will probably work. That's Kingston, and I don't see any reason why it won't work. 16GB is a good start with upgrade potential when you need it.
2. Motherboard is exactly what I have. Great choice IMO.
3. If cost is a factor consider the g2020 CPU. No AES-NI but will definitely perform for you unless you want to transcode videos with plex or run some other jail that is very CPU intensive.
4. For a case, you can find out if temps are an issue by setting up the temp monitoring part of SMART. Below 40C is the place to be, so set the critical to 39. :)
5. RAIDZ1 isn't a recommended setup. See my sig, then go with RAIDZ2. :)

 

[Kosta]

Thanks cbyerjock,
The CPU will then remain the 3240T, as transcoding (currently using Plex) is a must. I frequently watch movies or series in the train while traveling or in the gym on the trainer. I travel 2hrs per day.
Temps - great advice, will see to that. The build is likely to happen in December, so planing ahead.
OK, RAIDZ2 it is. So if I understand correctly, with RAIDZ2:
If I go initially with 4x 4TB, I will have only 8TB of space available to me, while 8TB go to parity. Vdev1.
Then I add additional 4x 4TB disks, Vdev2, and what happens then? Then I still have 8TB parity, and 24TB usage space? Can I expand like that? Or do I get 16-16, as it is two Vdevs?

And... should I install FreeNAS or NAS4Free? I know I am on FreeNAS forums, but I am kinda confused what's right. Both seem to be the same thing, one is 8.x kernel, and the other is 9.x kernel...

 

[cyberjock]

With a RAIDZ2 you'll get only 8TB usable space. Normally we recommend a 6 disk RAIDZ2 to make it more cost effective per GB. If you add a vdev that's a whole second set of redundancy, so if you went with 4x4TB you'd add only 8TB more space for a total of 16TB. Now you see why we recommend building it big and build it right the first time. Adding more space later is painful and expensive.

As for FreeNAS vs NAS4Free, that's totally a personal choice and not one that I can guide you on. But both run a 9.x kernel I believe(not a NAS4Free user so I could be wrong).

If you plan to do transcoding you may need a more powerful CPU than an i3. I'm not completely sure on this, but you may be forced to get a beefier processor later.

 

[Kosta]

OK thanks cyberjock, makse sense. I will then definitely go for 8x 2TB.

How about I forget the whole cheap case thing, as I was thinking of getting cases with removable slots (3x5.25 -> 4x3.5) and PSU from above, however it would cost more than it would cost getting something like this:
http://www.supermicro.com/products/chassis/2u/825/sc825tq-563lp.cfm
or
http://www.supermicro.com/products/chassis/4U/743/SC743TQ-865.cfm

I would probably prefer the 2HE case, but I'd like to hear the opinions too.

I will then probably try both and see that I settle for one.

On i3 and transcoding - according to Plex forums, i3 is quite enough for single transcoding job, and some say it can even do two.

However, there is an info I got on another forum, so wanted to double check: I have been told that it doesn't matter if I get 3240T or non-T, the idle power usage will be the same. The only difference will be that T has less power and is thus less power-hungry when it comes to higher or full loads. Is that correct?

 

[Kosta]

And here is another alternative I found out about:
http://www.supermicro.nl/products/motherboard/Xeon/C220/X10SAE.cfm
(or the X10SAT, as I have no need for PCI slots, and the connectors on SAT are horizontal, making it better for the horizontal case)

8 SATA slots, so I could save myself the M1015, and save quite a lot of money.
So the current build looks like this:
Supermicro CSE-825TQ-563LPB
Intel Core i3, 2x 3500 MHz, L2 Cache
Kingston 16 GB DDR 3L 1333MHz ECC Reg CL 9 DIMM DR x4 1,35 V Intel (this is a single module, but it's lower V - less heat, less power consumption, cheaper)
Supermicro MBD-X10SAE-O

It's 800€ without disks, that looks quite good as a build, and I would possibly plan to buy a 19inch case for all the stuff in the cellar. Is something like that recommended for the server? I see it everywhere, however I can imagine server being "hot" inside... experiences?

 

[cyberjock]

You do need to keep in mind that if you are running Plex as a plugin your server is also doubling as a ZFS server and file server. It's not quite an apples-to-apples comparison, but if they say you can do two normally you will probably be safe with doing one.

 

[Kosta]

OK, and this doubling, is bad, or just CPU intensive or how should I understand it?

And I only need one

 

[cyberjock]

Doubling? I'm just saying you'll need more CPU power than what you'd need for "just plex". Depending on various factors you might be able to do one stream, or you might not be able to do any. As I said above, there will be other things running on the system besides just "an idle OS". Reads and/or writes to ZFS does require CPU power. So you'll need more than "just plex" because you'll be reading the move off the pool and then transcoding it with Plex.

Keep in mind I keep saying "Transcoding". If you're just dishing out the streams as-is the risk of not being able to stream is far lower. If you aren't familiar with what transcoding is I recommend you google it and/or visit the plex forums.

 

[Kosta]

Ah OK, I wasn't aware that reading the data requires so much CPU power, especially in ZFS case. I have no experience with that.

If the CPU power is a worry, I was thinking of going the Xeon, but my highest worry is the idle wattage. So the question is, do i3 and Xeon produce the same idle power consumption? I read some threads that Xeons park the core quite quickly, and actually while not needed, it doesn't use any power if not needed. Unfortunately, I can't find any reliable results for power consumption of i3's or Xeons on idle. It is a must that my system, without a single disk, idles at about 30-35W, not really much more. In that case I can expect a good power bill reduction. However the price of Xeons is also mostly double, so I would get it only if power is needed, not to lower the power consumption. But it also must not grow upwards (on idle). I am aware it is a quad-core and it uses more power when needed. However I am OK with that when I need to use the server.

I am very familiar with transcoding, been doing it for about over 2 years now, since I got the iPhone 4s. Different programs, until I found out about Plex couple of days ago. And it is on their site that I found what CPU I need for transcoding (i3-3220 as a min, and ATOM 2700 won't cut it) -> I was thinking of going 1813+ Synology box initially. But the current transcoding has been done on a 2600K @ 4.6Ghz, so no real problem running it CPU-wise.

 

[cyberjock]

My e3-1230v2 system does 35w idle with no disks and 32GB of RAM. So the whole idea that idle power is really is is quickly debunked. ;)

 

[Kosta]

Alright, so would the e3-1220v3 be a much better alternative, but it would do much easier on transcoding, as it is a quad I guess? I am pushing and pushing the costs up, and I really wanted to stop at 1500€. I can turn around if I take cheaper case and a PSU up to 80€ for example. I found out that getting the rack component is actually stupid, so I turned back to server towers, and will just mount it onto the wall or put on the floor. And even if I get for instance Chenbro SR11269 and a Seansonic S12G 550W, I save some 50€, so fine, I get the Xeon.
But seriously, I have a woman also on my side, who is really more than only understanding, and the whole story began with about 1000€ for the server, now we are at 1500€ because it is better to build it complete, but going for 2000€ is not an option.
Only additional cost would be the UPS, however I better not tell her about that anways... :D

 

[Kosta]

And concerning UPS, this would be enough?
APC Back UPS PRO USV 900VA

 

[Kosta]

And concerning sleep, since you have the Supermicro... how does this server actually sleep, like a normal PC? You go into sleep mode, and have to wake up by WOL command or is it like my current DLINK DNS-323 NAS, which has an ability to sleep and when I try to access the share via computer, it wakes up (takes a while, but it does wake up). In other word, it's enough for me to VPN into the network and basically do anything that's in connection with the server, be that access via SMB or browse, it will wake up. No additional commands necessary.
How does SM handle that, what is possible? Thanks.

 

[cyberjock]

My e3-1230v2 system does 35w idle with no disks and 32GB of RAM. So the whole idea that idle power is really is is quickly debunked. ;)

What I meant to say is that "So the whole idea that idle power is lower is quickly debunked".

If you are asking me about the sleeping thing, I don't sleep my server. FreeNAS doesn't sleep in the S states as far as I know. It does use the C-states and P-states if your CPU supports it. You could conceivably shutdown your server and then wake it up remotely via WOL data packet assuming FreeNAS/FreeBSD supports WOL for your hardware.

 

[Kosta]

Yes, I got the meaning of the upper sentance ;-)

Well, this means that the DIY server with FreeNAS or whatever other software is not capable of "standby" what Synology and QNAP systems call their low power usage states? Afaik, these states are probably very much the same like a standby of a normal computer, however they wake up without the magic packet (WOL), just by accessing the NAS. And WOL is what I would like to avoid, because it's simply annoying. I tried to tell my computer to wake up not only for magic packet, and that caused it basically to wake up all the time without my input, so I am at a loss how they do it with the prebuilt NAS...?? C/P states are nothing else but lowering the frequency and Vcore. That brings a little, but definitely not 1W standby mode.

EDIT: What I think I am talking about is the S3 sleep state. I guess that's the mode those prebuilt NAS's are using.

 

[cyberjock]

Well, the Synology devices don't use your standard x86 based CPU. Usually they are custom CPUs on custom hardware.

QNAP uses Intel CPUs for some of their models. Many look like ARM custom CPUs.

I will agree that what you are describing is roughly equivalent to S3 state though. FreeNAS wasn't really designed with S-states because it was expected you'd want 100% uptime. Not sure why you'd want to make it wake up for every packet, but WOL does work well for servers for me in the past with magic packet. I even had a .bat file that you ran from any computer on my LAN and it woke up the server. I have never done this with FreeNAS because constantly cycling hard drives seems to wear them out more quickly than just leaving them on all of the time.

 

[Kosta]

Synology uses ATOM and i3 processosors for many high(er) end devices.

QNAP also, for instance the i3-3220, 8xHDD NAS, €1100.

OK, but you do use HDD shutdowns?? Leaving both NAS AND HDDs running all day is a no-go, as long as NAS is doing absolutely nothing (I am building it, amongst others, with BT in mind). I would want then the HDDs to shut down. My greens are now 3 years and no failure signs (DNS-323 with 2x 2TB green, sleep and HDD turn off set to some half an hour I think), so I reckon REDs would hold out even longer/better...

 

[Dusan]

OK, but you do use HDD shutdowns??

You can set HDD Standby timers to spin down the disks. The WD REDs are a bit tricky to set up when you also use SMART. More details here: http://forums.freenas.org/threads/ata-idle-wd-red-drives.11773/#post-68854

 

[titan_rw]

OK, but you do use HDD shutdowns?? Leaving both NAS AND HDDs running all day is a no-go, as long as NAS is doing absolutely nothing (I am building it, amongst others, with BT in mind). I would want then the HDDs to shut down. My greens are now 3 years and no failure signs (DNS-323 with 2x 2TB green, sleep and HDD turn off set to some half an hour I think), so I reckon REDs would hold out even longer/better...

This is a debate akin to 'what religion is better'.

Whatever you prefer is what's best for you.

I personally never have any drives / desktops / servers sleep. They are 100% on all the time, all day, every day. Even laptop's that I may only use once or twice a month are set to 'never sleep'. The only thing I don't have run 24/7 is monitors. And my desktop monitors I manually turn off when I know I'm going away for more than a few hours. I still set windows to never turn them off.

I have some hard drives that are 15 years old, and have been spinning continuously since new. And they still work great. On the other hand, I'm sure there's people that have drives set to sleep after 5 minutes, and end up having 1,000's of power cycles a year and never have problems. It's all up to your preference.