New to FreeNAS, need help on hardware

[rockhead006]

Hi all,

I currently have 3 x D-LINK 320 dual bay NAS enclosures. And also 3 x backup ones (kept offline most of the time for backups). I have a range of 1.5TB, 2TB and 3TB drives in them.
They are getting old now, and the fans are constantly on (and not quiet either) as the thermostat in them are broken.

I don't have any RAID set up, as I don't like the idea of one drive dying and then being at risk of losing all my data is another drive dies. It's not worth it, as hard drives are (relatively) cheap and so I'd prefer to go with no RAID. Just standard disks.

I currently have a total of about 14TB storage and it's almost maxed out. So I would like to (eventually) go for large HDDs (4TB each), and also have some drives used for backups only. I won't need to backup everything, so perhaps 5 of the (8) drives used as primary storage, and the remaining 3 used for backups.
I use the system mainly for storage of movies/tv-series (purchased, ripped from DVDs, etc), music, as well as normal file storage.

I'd like to replace all of the above with a FreeNAS system (DIY built), with perhaps 8 bays. I don't want to have a rack mount system as they are too deep for where I want to keep it. I don't have a full sized cabinet to put them in. So a system with a standalone case would be best. And not with a really loud fan either if possible.

My problem is where do I start with hardware. I don't want to spend a tonne of money (if possible). I'd like to use my existing HDDs (SATA, 3.5") for now.
So I'd need a (suitable) case, some sort of motherboard (with perhaps integrated CPU/GPU, memory(?), NICs). And also a way to connect all the HDDs.
Either by eSATA (and multipliers?), or standard SATA. I don't need massive transfer speed either, so gigabit network (either single or multiple NICs), and enough speed to stream to perhaps 2 devices at once (in 1080P or 4K for movies).

So I'd appreciate it if someone (perhaps someone who has built a similar system) could point me in the way to some possible hardware I can look into.

Also if possible I'd like to run OpenVPN server on it. But this is not essential as I have a Raspberry Pi which is doing this currently.

Many thanks in advance.
Chris

 

[rockhead006]

This is what I'm thinking so far:

CASE: Fractal Design Node 804 Case for Computer - Black
https://www.amazon.co.uk/Fractal-De...00JBBH93K/ref=cm_cr_arp_d_product_top?ie=UTF8

MOBO: Supermicro Micro ATX DDR4 LGA 1151 Motherboards X11SSM-F-O
https://www.amazon.co.uk/Supermicro...531323472&sr=8-1&keywords=Supermicro+X11SSM-F

CPU: Intel Pentium Dual-Core G4400 3.3 GHz Processor CPU
https://www.amazon.co.uk/Intel-Pent...&qid=1531325667&sr=8-1&keywords=Pentium+G4400

MEMORY: Patriot Memory Extreme Performance 8GB DDR4 2133MHZ Memory Module (8GB DDR4 2133 MHz PC Memory Modules/Server, 288-pin DIMM, 1 x 8 GB)
https://www.amazon.co.uk/Patriot-Me...d=1531326134&sr=8-6&keywords=8GB+DDR4+2133Mhz

NEW HDDs (up to 8): Seagate BarraCuda - 4 TB internal hard drive, Silver,ST4000DM004
https://www.amazon.co.uk/Seagate-Ba...rs&ie=UTF8&qid=1531323032&sr=1-1&keywords=4tb

PSU: Not sure yet. Will need to work out the wattage required.

What do you think? Is there any possible issues that you can see with FreeNAS (or anything else)?

Thanks

 

[warllo]

Would you be willing to use some used gear from ebay? I recently did a build very similar and it worked great. I saved a significant amount by using some used gear. I see based on your link your not in the US. That might diminish some of the cost savings but it may still be worth a look.

 

[rvassar]

That's not ECC memory. Also... You can save quite a bit of $$ sticking with DDR3 as well.

 

[Chris Moore]

Many thanks in advance.
Chris

If you are looking to get into a NAS for a small investment...

I don't want to spend a tonne of money

Why are you looking at new hardware?
You can pickup used gear that will work fine with FreeNAS and save. A few years down the line, then you can buy this to upgrade the gear you buy now.
I am not saying that you must buy this one, but have a look at this system board:
https://www.ebay.com/itm/SUPERMICRO...on-Server-Motherboard-w-Heatsink/273335376327
This one will do the job and only cost around $50
Then you could go with this CPU:
https://www.ebay.com/itm/Intel-Xeon...d-Core-LGA-1155-CPU-Processor-km/173397510479
For only $58 and between the two you would have saved a bit and that is before even looking at memory.
Additionally, this gets you a quad core Xeon that is much more capable if you want to run something like a jail with Plex media server.

 

[garm]

Also, it’s worth noting that you will not be able to keep the data on the drives unless they are members of a ZFS pool already. And your attitude about storing data on descrete disks is going to be counter productive as with FreeNAS you need to use ZFS. ZFS is wing clipped if you do not have vdev redundancy, it might work fine for a workstation, but is less then ideal for long term data storage.

 

[Chris Moore]

Either by eSATA (and multipliers?)

Don't even try. Not reliable.

I currently have a total of about 14TB storage and it's almost maxed out. So I would like to (eventually) go for large HDDs (4TB each), and also have some drives used for backups only. I won't need to backup everything, so perhaps 5 of the (8) drives used as primary storage, and the remaining 3 used for backups.

Sounds like you need to read the primer on ZFS because after what @garm said, I looked again I think you misunderstand how FreeNAS works.

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Did you read the manual?
http://doc.freenas.org/11/freenas.html

Updated Forum Rules 4/11/17
https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/

 

[rockhead006]

Thanks. I will look into 2nd hand, but after doing a quick search I couldn't find any.

I live in Ireland, but can order from the UK also.

I was thinking of spending up to about £500 for everything except the hard drives. Which I think what I listed would be in budget, just.

Regarding the quad core, would this be necessary. As all this system will be doing is sharing files.

I'm willing to go RAID (or ZFS). As I mentioned before I never really trusted RAID. I've had it in the past (RAID5) and on of the drives failed, so I bought a new drive and started rebuilding the RAID, but then another drive failed (probably due to the heavy read/write of the rebuild) so I lost all my data. Since then I've just gone for a dual setup of offline drives matching online ones, and doing a manual backup every month.

Also what do you mean about the memory not being ECC? Is ECC memory required? What's the difference?

Also I could downgrade the memory to DDR3. But one person said to upgrade the system (CPU) and another said to downgrade (the memory). Does this system need to be upgraded or downgraded for my purposes?

I guess the only other thing which I would like to run on this system is Emby (a media browser/steaming system). I'm not sure if it runs on FreeNAS or not. But if not it means I'll have to have another system running for it, which is not ideal.

 

[rockhead006]

Don't even try. Not reliable.

Sounds like you need to read the primer on ZFS because after what @garm said, I looked again I think you misunderstand how FreeNAS works.

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

Did you read the manual?
http://doc.freenas.org/11/freenas.html

Updated Forum Rules 4/11/17
https://forums.freenas.org/index.php?threads/updated-forum-rules-4-11-17.45124/

Ok for ZFS, I'd for for individual vdevs and vpools. One for each drive. So basically no RAID. It sounds the safest.

Emby is supported in FreeNAS. So that's good.

Also I won't be going for eSata/multiplier. As the motherboard I selected has 8 SATA ports already.

 

[Chris Moore]

Ok for ZFS, I'd for for individual vdevs and vpools. One for each drive. So basically no RAID. It sounds the safest.

You may as well use some other filler if you are not using ZFS the way it is intended. There are plenty of options out there. I would rather see you use something else than use FreeNAS wrong.

 

[pro lamer]

If you need live 4K transcoding x2 (comparing to direct play streaming) you may need 2x decent discrete GPUs especially for HEVC (if this is the case) and I'm not sure if FreeNAS allows GPU passthrough (since I guess a VM/jail is needed here)... So a separate rig for transcoding might not be a bad idea.

EDIT: it depends also on the fps, bitrate...

 

[garm]

No, you need to read the links @Chris Moore gave you. First, you wouldn’t know it but your data is most likely already degraded. You just don’t know it because you don’t use ZFS. ZFS will tell you when a single bit on a disk is flipped and correct it, if you have redundancy.
Redundancy isn’t risking data, it’s the opposite.

You don’t have to use RAIDZ for redundancy, I don’t. I use almost exclusively mirror vdevs. There is no such thing as a vpool. ZFS is built as a storage pool of large numbers of drives. These are arranged in virtual devices (generic *nix term for among other things hard drives). A virtual device consists of one or more storage devices. If you have two mirroring each other you have a mirror vdev. This is the bare minimum of redundancy in a single vdev. A pool may consist of any number of vdevs.

 

[Chris Moore]

I'm willing to go RAID (or ZFS). As I mentioned before I never really trusted RAID. I've had it in the past (RAID5) and on of the drives failed, so I bought a new drive and started rebuilding the RAID, but then another drive failed (probably due to the heavy read/write of the rebuild) so I lost all my data.

This is why we suggest always using RAIDz2 as a minimum configuration. If you will read the presentation I linked to, it will explain many of these concepts, but RAIDz2 can suffer 2 drive failures without data loss.

 

[rockhead006]

Thanks for your responses so far. They have been very useful.

I would much prefer to have a copy (separate drives dedicated for backups) like I have currently. Where I can selectively choose what I want to have backed up, rather than just backing up everything. However, if this is not possible (recommended) in FreeNAS, then perhaps I should put on Linux instead.
I just thought as FreeNAS was a dedicated NAS OS, then it would be able to handle this sort of setup, and would likely be better performance than running Linux (with a lot of stuff I don't need installed on top).

So my first preference would be: Totally individual drives, not linked to each other with RAID or anything. Which allows selective (manual) backups, and also varying drive sizes and replacing of drives without a rebuild.
My second preference would be: RAIDz2 (which I believe is like RAID6). Not great, as it's also a waste (backing up everything) and also has the risk of losing all data if 3 drives go before I can complete the rebuild (it's not unheard of). Also not sure how this would work with different sized drives, as I'd likely start off with my existing drives (1.5/2/3TB) and gradually replace them for the 4TB ones, when I buy them.

If I have to go down the RAID option. I guess I could dedicate 7 drives to the RAIDz2 (RAID6) (5 usable). And then have the remaining 1 drive for (a second) backup of important data (non-RAIDed). Which would give me 20TB of usable data on the RAID, and 4TB of non-RAIDed data/drives.

I'm looking at this memory, as it's ECC:
Kingston KTM-SX421/8G 8 GB DDR4-2133 MHz ECC Memory Module

Regarding the transcoding of videos. I'd say it would more likely be only one 4K transcoding at once. Is it possible to do transcoding at all on FreeNAS. I hope so, as I currently have a pc running 24x7 doing the Emby stuff and all my NASes (3) running 24x7 storing the files. I was hoping to just have one system do it all. Is FreeNAS a VM? If FreeNAS can't, then I guess I'll have to go for Linux (or Windows).

I assume all the other hardware is ok. Again, I just need to look into how much power I need (what size PSU). I also need to look into a drive for the OS. I see USB is standard for FreeNAS, but I believe some people use SSD. Or perhaps even SD(?). What would be best here? Knowing that I'm going to be using up all my SATA ports on the motherboard with the NAS drives. So will have no free SATA port. So I assume USB would be best. Or how about SD over USB (using a USB to SD adapter), is that supported/better than standard USB?

I'm thinking 450W PSU should cover it. So perhaps this one:
Corsair CP-9020049-UK VS Series ATX/EPS 80 PLUS Power Supply Unit, 450 W
Please let me know if you think my system would require more than 450W, as there are larger ones available in the same type.

I plan to buy all this stuff (with just one of the drives for now) in the next day or so. It's all coming in at about £485. So not too expensive. Especially when you see the pre-built ones for like £1500-£3000 (not including drives).

Thanks

 

[Chris Moore]

I would much prefer to have a copy (separate drives dedicated for backups) like I have currently.

You have a misunderstanding about what RAID is. RAID (any kind of RAID) is not a backup. Not ever.
If you look at my system description in my signature (that I need to update) you will see that there is a main storage pool and a backup pool. The main storage pool consists of two RAIDz2 vdevs of 6 drives each and the backup pool consists of a single RAIDz1 vdev of 4 drives. You can have your RAIDz2 and your backup also. In ZFS, the RAIDz level is the redundancy that keeps your data from being corrupted and protects you from disk failure, but it is still not a backup.

Where I can selectively choose what I want to have backed up, rather than just backing up everything. However, if this is not possible (recommended) in FreeNAS, then perhaps I should put on Linux instead.

What I copy to the backup pool is fully configurable. I use the rsync utility to update the information in my backup pool on a schedule and I am able to pick what is backed up and when it is done.

I just thought as FreeNAS was a dedicated NAS OS, then it would be able to handle this sort of setup, and would likely be better performance than running Linux (with a lot of stuff I don't need installed on top).

FreeNAS is great and wonderful and I love it dearly, but it is not always the answer for everyone. The way you have described wanting to use it is not the way FreeNAS or ZFS (which is the file system of FreeNAS) is designed to work. You would be setting yourself up for failure and I would rather not have that failure associated with FreeNAS in any way. In other words, when you loose data because you didn't use FreeNAS properly, I don't want you to blame FreeNAS.

So my first preference would be: Totally individual drives, not linked to each other with RAID or anything. Which allows selective (manual) backups, and also varying drive sizes and replacing of drives without a rebuild.

That is just not how FreeNAS is meant to work.

My second preference would be: RAIDz2 (which I believe is like RAID6). Not great, as it's also a waste (backing up everything)

Again, RAID (of any kind) is never a backup. I keep spare drives on hand (in every size I use) so that I can replace a drive as soon as it starts giving me problems and that prevents me from needing to worry about

and also has the risk of losing all data if 3 drives go before I can complete the rebuild (it's not unheard of).

which doesn't happen often if you are not using drives that are rubbish because they are poor quality to begin with or very old.

Also not sure how this would work with different sized drives, as I'd likely start off with my existing drives (1.5/2/3TB) and gradually replace them for the 4TB ones, when I buy them.

You can only access the capacity of the smallest drive in the vdev. Again, if you look at my system in my signature, you will see that one of the vdevs in my main data pool is 6 drives that are 2TB each. Way back, years ago, I started with 6 x 1TB drives. As time went on, I replaced the 1TB drives with 2TB drives and when they were all replaced, I had access to the full capacity of the 2TB drives. Then, I took the 1TB drives that I had removed, and created a second vdev of 1TB drives. This gave me an immediate total pool size increase and over time I replaced all the 1TB drives again with 2TB drives. As time went on, I have replaced the 6 x 2TB drives in vdev-0 (which started as 1TB drives) with 4TB drives and when I had all 4TB drives in the vdev, I had access to the full capacity of those drives. It does require some capacity planning and time to go from 1 point to the next.

If I have to go down the RAID option. I guess I could dedicate 7 drives to the RAIDz2 (RAID6) (5 usable). And then have the remaining 1 drive for (a second) backup of important data (non-RAIDed). Which would give me 20TB of usable data on the RAID, and 4TB of non-RAIDed data/drives.

If that is how you choose to configure your system. Many people do something like that.

Also I could downgrade the memory to DDR3. But one person said to upgrade the system (CPU) and another said to downgrade (the memory). Does this system need to be upgraded or downgraded for my purposes?

I guess the only other thing which I would like to run on this system is Emby (a media browser/steaming system). I'm not sure if it runs on FreeNAS or not. But if not it means I'll have to have another system running for it, which is not ideal.

The reason I suggested DDR3 is because it is often able to be found for less money than the cost of DDR4. If you wanted to use something like Plex to stream video from your NAS, you would want a stronger CPU to handle the transcoding.

Regarding the transcoding of videos. I'd say it would more likely be only one 4K transcoding at once. Is it possible to do

Yes

Is FreeNAS a VM?

Although it could be done that way, it is not what I was suggesting and it would take more powerful hardware.

I assume all the other hardware is ok.

I am not sure which hardware you are talking about because you had suggested some and there were other suggestions made. I don't know what you have in mind to buy at this point, so we probably need to discuss that some more if you decide to go the FreeNAS route.

Knowing that I'm going to be using up all my SATA ports on the motherboard with the NAS drives. So will have no free SATA port.

For the data drives, I would suggest a SAS controller, and use the SATA ports for things like the boot drives or other such.

SAS PCI-E 3.0 HBA LSI 9207-8i P20 IT Mode for ZFS FreeNAS unRAID
https://www.ebay.com/itm/162862201664
Price: US $69.55

I would suggest one of these for the boot drive. It will last as long as the server, if not longer:

Intel-SSD-DC-S3500-Series-2-5-SATA-6Gb-s-20nm-MLC-80GB-SSDSC2BB080G4
https://www.ebay.com/itm/283017365544
Price: US $29.99

You might need to read this:

Don't be afraid to be SAS-sy
https://forums.freenas.org/index.php?resources/don't-be-afraid-to-be-sas-sy.48/

Also, you will need forward breakout cables from the controller to the drives. I like these because they are a bit more durable than some of the slim ones that I have accidentally broken before:

Lot of 2 Mini SAS to 4-SATA SFF-8087 Multi-Lane Forward Breakout Internal Cable
https://www.ebay.com/itm/371681252206
Price: US $12.99

So I assume USB would be best. Or how about SD over USB (using a USB to SD adapter), is that supported/better than standard USB?

I would stay away from any USB as it is not (in my experience) reliable over time when using it the way it is used in FreeNAS.

I'm thinking 450W PSU should cover it. So perhaps this one:

It depends entirely on the number of drives and the specifics of the other hardware you put in, but I would always suggest going for a larger supply than you expect to need, so it doesn't need to be replaced a few years later when you need a bigger one. I have a 550 watt supply in a system that only draws about 120 watts and my FreeNAS supply is able to crank out 1300 watts but the system only draws around 350.
Here is a guide that will help you figure what you need:

Proper Power Supply Sizing Guidance
https://forums.freenas.org/index.php?threads/proper-power-supply-sizing-guidance.38811/

I plan to buy all this stuff (with just one of the drives for now) in the next day or so. It's all coming in at about £485. So not too expensive. Especially when you see the pre-built ones for like £1500-£3000 (not including drives).

If you are going to setup the RAIDz2 that would be recommended when using drives larger than 1TB, you need to have 4 drives at the minimum and I would suggest going with 6. You can't dynamically change a vdev once it is created. Did you read the presentation I gave you a link to?

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

If you are not going to take the time to learn about how FreeNAS and ZFS work, you are just asking for trouble that will bite you in unpredictable ways. You could loose all your data, for example. It has happened before. I am not trying to rain on your parade, but I would like to either set you up for success or see you move along to something else

then I guess I'll have to go for Linux (or Windows).

 

[rvassar]

I would stay away from any USB as it is not (in my experience) reliable over time when using it the way it is used in FreeNAS.

I ran for a couple weeks on a SSD installed in a USB 3.0 enclosure, connected to a USB 3.0 PCIe HCI. I ended up in that state after multiple USB thumbdrives got kicked out of the boot pool on the host's USB 2.0 HCI. The thumb drives all tested OK after FreeNAS "failed" them. Thumb drives are generally junk, but don't get the idea that using a SATA SSD in an enclosure is any better. It's USB itself that's the problem. @Chris Moore warned me, and I didn't listen and had to prove it to myself... So I wanted to jump in here and say "He's not kidding!"

 

[Jorsher]

I'm definitely not one of the experts here, but I think I see where your thoughts are.

However,
1) You do want ZFS, especially if any of the data is irreplaceable like family photos. ZFS protects from bitflips/data-rot/bit-rot/whatever. Data slowly corrupts, and virtually all other filesystems are blind to it. You get no notification and repairs aren't made. If you're backing up to cloud storage or another offline drive, that corrupted file is just replicated in those locations. The file integrity is gone. Now, it's very likely you will never notice the corruption, but if you care about protecting your data -- then you should care about the integrity of it as well.
2) You want ECC ram for the same reason you want ZFS filesystem. ECC ram verifies data integrity. I avoids copying corrupted data to a disc from memory. If you aren't using ECC ram, you're losing some of the benefits of ZFS.
3) RAID is more economical. Why not build two RAIDZ2 or RAIDZ3 arrays, and have them mirrored to a duplicate array? It will be cheaper than buying double the usable storage, and you get to use the great ZFS :) I have one array now, soon to have a second, and back only my important data to two cloud storage providers. Downloaded media and software, I can always download again. My personal, irreplaceable data gets synced to 3 other locations FROM the integrity-protecting ZFS arrays.

Read the hardware guide. It doesn't get extremely detailed, but it tells you what you need to know and explains some of the "why." These other members will be able to offer much more help, as you can probably see above, but you need to at least understand the basics.

 

[Chris Moore]

also has the risk of losing all data if 3 drives go before I can complete the rebuild (it's not unheard of)

I have been using FreeNAS for a longish time (as of 2018) it is about 7 years, but I have been administering storage systems where I work since 2001 and over the years, it happened once or twice that we lost a storage array because another drive failed during the rebuild. It can happen and that is the reason that we went (years ago) to using RAID-6 at the minimum because it allows for 2 drive failures to happen without data loss.
You are correct that RAIDz2 is similar to RAID-6 in that regard, but ZFS also stores a checksum for each data block so it can verify that what the disk gives it back is the same as what was stored and ZFS will correct the data as long as it has redundancy so that you get back what you stored.

Here is a video presentation you might be interested to watch which covers some of these things: https://youtu.be/m4FMjyrdD10

In my RAIDz2 array at home, I have had a couple of instances over the years where I had a second drive start giving errors either before I was able to replace a failed drive or while I was replacing a failed drive and that is why I have a backup pool. All my important data is stored in the backup pool in addition to being stored in the main pool. The redundancy is all for my convenience. I don't have to worry about loosing data because I have a redundant array and I have an entire backup array.

I don't claim to be the one with the biggest servers, because I know I am not, but I try to have a reliable place to keep my data. It is like a chain though and your system is no stronger than the weakest link and the week link is usually the drives. Solving that weakness is the reason that ZFS was developed.

 

[Chris Moore]

It's USB itself that's the problem. @Chris Moore warned me, and I didn't listen and had to prove it to myself... So I wanted to jump in here and say "He's not kidding!"

I appreciate the vote of confidence. :)
I always try to guide people in a direction that I think will be problem free. I want the experience to be a good one.

 

[Chris Moore]

It's not worth it, as hard drives are (relatively) cheap and so I'd prefer to go with no RAID. Just standard disks.

It is because drives are cheap that putting them in arrays does make sense. Take a drive like this:
https://www.ebay.com/itm/Hitachi-Ul...0Gb-s-Hard-Drive-HUA722020ALA330/382122140604
You get 6 or 8 of those (lets say 8) and that costs $384 with free shipping. Put those in a RAIDz2 array and you have 8TB of storage that is absolutely reliable as long as you keep a single spare drive available for when you have a drive fail. Notice that I said when, not if, because it is always to be expected that a drive will fail, it is just a matter of time. So, we add another drive and our cost is up to $432.
The comparison is this:
https://www.newegg.com/Product/Product.aspx?item=N82E16822146142
That is an 8TB HGST NAS drive for $249. They are fairly reliable, but if you only have one, you risk a single disk failure taking all your data. If you want to be able to backup that 8TB, you need another drive, another $249 and that would bring you to $498 and still, you only have 2 drives and that doesn't protect you from 2 drives of failure. Suddenly, that array doesn't look so bad. You get the same usable storage and you would need 3 failed drives before you loose your data, but you can have a spare on hand for a total cost that is less than the cost of 2 of the 8TB drives, so really it would be even better.
If you are really paranoid, there is even RAIDz3, where you have 3 parity drives in the pool and in that configuration, you would need to have a 4th drive fail before you loose your data and you can still have spares on hand.
With ZFS, you can replace two drives at the same time. I have done it, but only because I had a full backup and I wasn't scared of loosing my data if the pool failed.