BUILD New Build - Seeking Opinions and a few clarifications

[Gilley7997]

I have been roaming around the forums and reading all the stickies and a few specific posts about hardware that I was looking at. There is a lot of excellent information here for people getting started. Kudos to you all!

Let me give you a brief reason for needing to do something. We currently have 2 desktops in the house and both machines hard drives are well over 85% capacity at this point and growing rapidly with a new baby in the house. I currently have a small 2 Bay Synology 211j that we picked up many years ago that is used as a backup location and also serves small bits of media out. It has 2 x 2TB Drives in a mirror currently and has worked out thus far for what I wanted it for. The little box just doesn't have enough horse power to serve as file server to the entire house and definitely doesn't have enough horse power to be used as a Media Server.

Looking at other Retail NAS solutions they just don’t have the horsepower and for the money they are asking, I just feel that I can do better.

My goals for changing the household environment:

• Move all data off of our machines and store in a central shared location. Data includes home videos, pictures, documents, large music library, Optical Disc Images and such.
• Current data is approx 500GB with an expected 200GB of growth per year
• This data is important. (A.K.A my wife would kill me if it would disappeared)
• I would also like to store system images of the workstations on this
• Have an onsite backup of the FreeNAS box.
• Have room for expansion as at some point I want to add a virtualization environment and would like to use this system to host the storage for that, but not share the same shared file system pool.
• Either use DLNA or Plex to source the media out to the rest of the house.
• ***FUTURE*** Offsite Backup either through a service or FreeNAS solution somewhere else.

Planned Hardware:

• Motherboard: Supermicro X10SL7-F (Plan to use the Onboard LSI controller for future expansion)
• Processor: Xeon E3-1231v3
• Memory: Crucial 2x8GB DDR3-1600 Unbuffered ECC - CT2KIT102472BD160B (Seems to work with this board)
• Storage: 6 x 1TB Western Digital Reds(WD10EFRX) in a RAIDZ2
• USB: 2x16GB USB2.0 SanDisk Cruzer Fits (Mirror the boot image)
• Power Supply: Seasonic SSR-550RM 550W
• Case: Fractal Design R4 (Seems I can find a way to nicely get 12 3.5” Drives in this case if I wanted/needed too.)
• There will be a UPS involved. I have been following the posts concerning the CyberPower stale data problems so still considering what exactly to do here.

Planned Implementation:

• FreeNAS system will be Central Storage for all files, and system images
• OnSite Backup solution – RSYNC to a single drive in the Synology. This backup drive will not be redundant in any way.
• No current offsite backup solution except for a USB hard drive stored off site.

Questions:

1.) Reading some of cyberjocks posts about data corruption, bad memory and backups has made me a little apprehensive about FreeNAS as a whole. It’s probably still less risky than storing the information on the individual desktops. Is FreeNAS stable enough to be used as the safe source for all my files?

2.) LSI Controller on motherboard. I believe I found posts from jyavenard and Sir Robin concerning the process of flashing the Onboard LSI to IT mode. There was also discussion of whether it was necessary to do on this board. I didn’t see that discussion resolved. Anyone have any input on this? If flashing is still recommended is version 16 still correct and this has to come from SuperMicro?

3.) Backups: I think the above backup plan will work for my needs. I have 2 disks of redundancy in the system itself and a single copy on the synology. I have tested the rsync functionality to the synology from another linux system and believe I have that working correctly. Is there a better way to do this? Also, when I do decide on the Offsite backup option, I am tempted to run the offsite backup from the synology itself and take the load off the FreeNAS system, but is that an acceptable way to do that, or should I always make the offsite backup from the FreeNAS system directly? What do other people use for their offsite solution?

4.) I would like the ability to if I need a little extra storage for something that isn’t part of this environment to be able to just drop a hard drive into the NAS system and create it as a separate zpool and use it. This type of usage would be for nothing permanent. I don’t see a problem with this except of course shutting the system down and running the risk of bumping cables or causing some other issue.

5.) 550W Power Supply enough for this (if I end up maxing the 14 Hard Drives) or is it to much?

6.) I don't have an immediate need so I haven't looked much into this yet but if there is something I should include in the build that is difficult to add later to be able to run Plex and or other Jail Add-ons in the future please feel free to let me know.

7.) Anything else I should be thinking about or considering? Any issues with the hardware above?

Thanks for all the information you all have provided thus far via other sections of the forums. I am looking forward to your feedback.

 

[Bidule0hm]

1. Yes, as long as you follow the recommended hardware (you've already done it) and software config (SMART, scrubs, emails, ... properly configured) in the stickies

2. Yes, you must flash it to the IT mode version 16. It's pretty simple, I've done it in 10 minutes (+ 1/2 hour of search before to be sure on how to do it right), I just needed a USB stick with the update files and I used the EFI shell. I believe you can do it from the FreeNAS 9.3 shell but I'm not sure about that, search a bit to confirm ;)

3. I don't recommend to do the offsite backup from the local backup but directly from the source (less things can go wrong plus the synology likely don't have any kind of data corruption detection let alone correction...)

4. You're right ;)

5. Yes, just verify that the 12V rail have enough current for the drives to spinup (count 2A per drive)

6. I don't see any problem (obviously if you want to transcode 50 HD streams simultaneously there might be a little problem with the current hardware... )

7. No, except for the drives: 1TB is very small and it will cost you far more than 3 or 4TB drives for the same storage space.

And, yes, this RAM works with this MB, look at my signature ;)

 

[Gilley7997]

5. Yes, just verify that the 12V rail have enough current for the drives to spinup (count 2A per drive)

So if I use that as a calculation I could actually get away with he 450w version of that power supply that Rated Maximum Load of 37A if I ever wanted to run all 14 drives.

I think that's asking the power supply to work a little to hard so I will stick with the 550. Thanks for the thought though made me actually go find a calculator and see what it said. It guesses about 465W required, so 550 sounds good to me.

7. No, except for the drives: 1TB is very small and it will cost you far more than 3 or 4TB drives for the same storage space.

This point I have gone back and fourth on quite a bit. Pricing out the Hard Drives 4x2TB($383.16) vs the 6x1TB ($389.94) drives cost comes 6 dollars of each other. I know at 6 drives for RAIDZ2 you are at least getting 66% utilization for the storage with possibly some performance help, figured I might as well go this way. Let me know if there is any flaws in my thought process.

Right now I figured that 4TB is about what I need for about the next 10 years of my important files. The board has enough expansion room if I need to add storage for a different purpose I can add another pool to handle that. If a disk fails I will probably replace it with a higher capacity drive, one at a time until either the pool expands, or when it comes time that I need more space I only have so many drives to replace.

 

[Bidule0hm]

"Right now I figured that 4TB is about what I need for about the next 10 years of my important files." The thing is, when you've plenty of space you tend to fill it, be it 1, 10 or 100 TB... :D

More seriously; you've included the 80% rule in your calculation?

More drives = more power, it might be important (electricity isn't free...).

Performance? even a single drive can saturate a gigabit link; if you don't have 10G link it's useless to include the performance in this choice ;)

At the end it's your choice, so do as you want but choose wisely ;)

 

[Gilley7997]

Bah...Now you made me research which has made me come up with another question...

Based on Amazon's current pricing for WD Red Drives and planning on a RAIDZ2 Configuration
6 x 1 TB Drives = $389.94 = 4TB Useable Price per TB = $97.49
4 x 2TB Drives = $383.16 = 4TB Useable Price per TB = $95.79
4 x 3TB Drives = $459.96 = 6TB Useable Price per TB = $76.66

Clearly the better deal here is the 4x3 combination. So the question for me comes down to this.

Is a 4 Drive RAIDZ2 Configuration capable of saturating a Gigabit ethernet connection?

It appears according to the FreeNAS manual that you can't configure 2 interfaces on the same subnet so I won't be getting any higher network speed that way by directing my machines to one IP and my wifes workstations to another. Unless I wanted to put us on different subnets which I could do, but I really really don't want to do that.

I don't have the infrastructure in place to do link aggregation so I won't be trying to increase speed over that.

If going to a 6 drive configuration won't actually help my performance because the bottleneck is my network then the 4x3TB option is a better option.

 

[Bidule0hm]

Seems like you've missed my line about the performance, i'll put it here too: even a single drive can saturate a gigabit link; if you don't have 10G link it's useless to include the performance in this choice ;)

 

[Gilley7997]

Seems like you've missed my line about the performance, i'll put it here too: even a single drive can saturate a gigabit link; if you don't have 10G link it's useless to include the performance in this choice ;)

Nope didn't miss it, read it as soon as I hit the post button on mine, it's like you were reading my mind and answered the question as I was typing it...;)

 

[Bidule0hm]

No big deal ;)

I forgot to put this in my previous post: be aware that 1TB is not equal to 1TiB (but to roughly 0.9TiB) and that there is some overhead due to ZFS (between 5 and 10% from what I can see on my server), so a 2TB drive doesn't mean you can put 2TiB of data on it.

 

[Gilley7997]

Just for the sake of argument, are there any metrics anywhere that I missed, when reading and searching the forums that show performance?

Just curious, maybe sometime in the future I will have the infrastructure to do Link Aggregation, my router currently can, the switches between us and it can't though. :)

Or maybe I will try some network acrobatics and add one more subnet to the network just to see what happens....but then again I might not, but I like numbers so just curious if there are numbers our there on what it can and can't do. I do have a series of test machines that do exist on a seperate subnet anyway so will probably use that second NIC for that purpose. (That is also where my future Virtualization hardware would sit :) Might as well use it if I have it.)

So I guess there are instances where I could be coming at this with 2 Gigabit Ethernet Connections. Could I still fill those?

Or is there nothing I can do short of running Fiber that's going to get me to those speeds that I would max out a 4 disc array or even 1 disc since that's the example you are using :)

 

[Bidule0hm]

Yes, here for example: https://forums.freenas.org/index.php?threads/performance-testing-with-dd.351/ there is also a similar, bigger and more recent thread but I can't find it right now.

Edit: found: https://forums.freenas.org/index.php?threads/notes-on-performance-benchmarks-and-cache.981/ :D

 

[Gilley7997]

So what your saying based on what I read, is I should be good with my 4 x 3 TB RAIDZ2 unless I'm coming at it with at least something that would get me the equivalent of at least 4 GigE connections. ;)

Well I think you have altered one of my decisions. (Well before this post by the way ;)) Thanks for the info.

 

[Bidule0hm]

I can't predict what speed you'll get but you can estimate based on other member's speeds who have a similar setup :)

No problem ;)

 

[Ericloewe]

A few comments:

Hardware is textbook. The PSU will easily handle your 14 drives, with room to spare. A G-450 would be tight, so no point in downgrading.

 

[Rilo Ravestein]

First of all; big thumbs up for doing your homework!

Regarding the UPS

• There will be a UPS involved. I have been following the posts concerning the CyberPower stale data problems so still considering what exactly to do here.

The "data stale" problem is only that message. The UPS itself works just fine. I also have that "problem" and I did quite some tests and my NAS shuts down nice and easy when the power is unplugged from the UPS and the batteries go below the given percentage.

Aslo, with every new build I see, I just have to say this: Really think through your BACKUP plan. Think about what would happen in case of a break in, fire, water leakage, lightning, or whatever else might occur and your NAS/data is lost. Don't think of it as some future plan, think of it as part of your current plan..

 

[Gilley7997]

The "data stale" problem is only that message. The UPS itself works just fine. I also have that "problem" and I did quite some tests and my NAS shuts down nice and easy when the power is unplugged from the UPS and the batteries go below the given percentage.

Yep, I have read that post, it's an interesting little communication hiccup, and I have also seen the work arounds. I just haven't decided exactly how I want to configure this yet down in the basement. Cabling from my entire house runs to that particular location. Modem and Router are available there as well as an access point. So was trying to determine how much I wanted to power with the UPS since this is also where the Synology and the FreeNAS box will be and most likely the virtualization hardware that I plan to build at some time. I haven't spent as much time researching this as of yet. That's why there are not any specifics on it, not sure if there is a solution that could handle everything or if I should look at individual units.

Aslo, with every new build I see, I just have to say this: Really think through your BACKUP plan. Think about what would happen in case of a break in, fire, water leakage, lightning, or whatever else might occur and your NAS/data is lost. Don't think of it as some future plan, think of it as part of your current plan..

I agree with you entirely, I guess this is where I will still be taking a USB drive backup offsite periodically. This is my current process I just want a more automated process, espcecially as this continues to grow. I have a little time to sort this out...who knows it may involve me talking my inlaws into letting me build a FreeNAS system for them at there business that they can use and I can drop a pool in theres and they could do the same to mine.

One more question, as I have been exploring the 3TB drives now, The WD Red 3TB's don't seem to be doing very well in the rating department. The Seagate 3TB NAS Drives are about the same price. I know everyone likes the HGST drives, but for a premium. Anyone have any direction on this. I see in the signatures that everyone uses a little of everything. Just curious the experiences out there.

 

[Rilo Ravestein]

Yep, I have read that post, it's an interesting little communication hiccup, and I have also seen the work arounds.

There seems to be some light at the end of that tunnel

 

[Starpulkka]

For extra safety lately i have been multiparring priceless data and parring those par data. I recommend to look at it. Had one currently windows hdd gone real bad, and sofar i managed to recover only 80% from it. I had automatic md5 checksums but its not useful to know that file is tinybit corrupted and you cant repair it. As for multipar you could check file and even repair it.

-offtopic
Have learned from it, i now have live smart detection and reporting scripts on windows, so it not going to happen again. What is strange is why windows does not have this live reporting and monitoring by default. (Sure windows will notify when disk is toast like 2 days late).

 

[Rilo Ravestein]

One more question, as I have been exploring the 3TB drives now, The WD Red 3TB's don't seem to be doing very well in the rating department. The Seagate 3TB NAS Drives are about the same price. I know everyone likes the HGST drives, but for a premium. Anyone have any direction on this. I see in the signatures that everyone uses a little of everything. Just curious the experiences out there.

HGST Deskstar NAS 3TB are only 5,00 Euro more expensive than the WD red 3TB in my area. I use those (see my signature), i'm happy with them, but they are not even one year old yet.

 

[Gilley7997]

The HGST Drives are about 10 dollars more per drive right now. I will have to consider that.

 

[Gilley7997]

....Or maybe I will just have to follow cyberjocks guide and get the WD Green 3TB drives and use the wdidle3 tool....any thoughts on that?