Cheap, Home Build?

[Harrisonm65]

Not clear why you think not using server-grade hardware is better. It isn't generally much more expensive. It's generally quite a bit cheaper than the prosumer gamer stuff..

I never mentioned that not using server grade kit was better - not sure where you got that from! I was just commenting that from what Id read on here so far, not many people had used consumer grade kit or reused redundant kit they had lying around. I was just curious if there were any like that and what their experience of using non server grade kit had been.

 

[jgreco]

I never mentioned that not using server grade kit was better - not sure where you got that from!

Isn't that really what you asked, though? Seemed to be the gist of it.

I was just commenting that from what Id read on here so far, not many people had used consumer grade kit or reused redundant kit they had lying around. I was just curious if there were any like that and what their experience of using non server grade kit had been.

The difference between a reasonable desktop board and a reasonable server-grade board is usually a handful of dollars.

The reason you generally don't want a reasonable desktop board is that the PC market is a race to the bottom. Board manufacturers competing in the desktop market are under severe pressures to cut as many corners as possible. This usually includes including crap you don't want, such as Realtek ethernet controllers, and crap you don't need, such as sound card hardware, along with much less stringent design of the BIOS, which can leave you in a situation where you are missing things that might be needed for virtualization, PCIe bifurcation, etc.

And that's not even considering the super-cheap desktop boards, you know, the ones that suck even at running Windows.

The reason you don't want a super-cheap generic craptastic desktop board is that they will not be designed for 24/7 use, because inexpensive components that are good enough for six hours a day for a year will get the manufacturer through the warranty period, but your NAS is going to be on 24/7 for five or ten years, and because of all the additional corners cut.

The conclusion here:

If your metric is simply "FreeNAS can be loaded onto it and it serves files" then there's a huge amount of options to pick from. You can buy the cheapest $29.99 300W PSU and the cheapest $69.99 mainboard with 8GB of non-ECC RAM and pile a bunch of WD SMR disks into a $49.99 tin can chassis and declare success. Never mind that you'll blow the PSU the first time you sleep all the drives and they spin up in unison, or that the Realtek ethernet can't break a few hundred megabits consistently, or any of the other common problems.

However, if what you want is a high quality NAS, there are all sorts of finer points to consider. That's what all the hardware guides on this forum are about. We kind of assume that if you're coming to ZFS, maybe you care about your data and you want your NAS to be a pleasant, trouble-free place to store your files, rather than being a constant educational expensive misadventure in learning things the hard way.

 

[Harrisonm65]

Isn't that really what you asked, though? Seemed to be the gist of it.

No, it really isnt. Youve just presumed that. I quite clearly stated that I was just interested in the experiences anyone had from building a home use FreeNAS device with old parts they already had lying around, maybe from an old PC build etc. That was it. Ive read some - not all, granted - of the hardware guides and appreciate what they say. However, I was merely trying to see if anyone had gone down the FreeNAS route on home grade equipment and how they had found the journey. Nothing more, nothing less.

 

[Yorick]

anyone had gone down the FreeNAS route on home grade equipment

Everyone and their grandma's dog. And the experience is:
- It usually works
- When it doesn't, it's probably that Realtek Ethernet, which can be replaced with an Intel i210 or similar add-in board
- If you have a memory failure, you won't know - lots of time spent chasing that down
- Ryzen in particular requires C6 states / CoolNQuiet be disabled, some other tweaks to make sure PSUs don't spin down
- Some REALLY old boards won't even boot the installer, though that's never been chased to root cause AFAIK
- Some REALLY crappy SATA on-boards are crappy SATA on-boards, though that shouldn't be too common any more. Do not ever trust SATA add-ins.
- If your on-board SATAs aren't reliable (rare but happens), you can sidestep the issue with an M1015 off eBay, and going through the learning curve of cross-flashing it to P20 IT mode, then using SAS to SATA breakout cables. With cables, maybe USD 50? Thereabouts.
- If you use USB sticks as boot media, you're likely going to regret it. If you use USB 3 sticks at USB 3 speeds, you will come to regret it.
- 9 out of 10 issues with FreeNAS not working right are hardware related
- IPMI and iKVM are god sends. If you've never used it, and experience it, you will not EVER want to build on a consumer board without it, ever again
- What you do depends on where you are in your socioeconomic journey. A lot of us are in the enviable state of being gainfully employed and having enough disposable that we can buy cheap server stuff off eBay. A day or two spent troubleshooting a bad RAM stick is more costly to us than buying DDR3 Registered ECC on eBay for 15-20 a stick. And, that might not be where you find yourself. Or, you just want to get a feel for FreeNAS first, and splurge for that 350-400 dollar "used server kit" build down the road. Those are all completely legit things to do. You're already doing the right thing by asking questions and thinking it through so that building on consumer kit is a conscious, deliberate choice, not just a "well it was there" default that might be regretted later.

For what it is worth, while I agree with the greybeards here that server grade kit is way nicer and saves you so much trouble, I don't have religion about it. I think you can absolutely build on consumer kit if that's what you already have. Know the potential trouble spots ahead of time and accept that you may need to deal with them.

When it comes to buying new or used: A server grade build won't be appreciably more expensive new, and is dirt cheap used. There, going with "server grade" as the default makes a lot of sense.

 

[elorimer]

Over the last year I have fooled with three different variations of FreeNAS builds on mostly leftover parts. I've learned a lot, had a lot of fun, and made mistakes and bad decisions. One of my bad (but not terrible) decisions was to put this all on a J5005-ITX motherboard I picked up for $150. It does everything I want with ease (files and Plex server), and with six drives idles at around 28 watts (40 watts transcoding, 75 watts for the two seconds the drives spool up). But at that exact point I should have jumped to a used server grade board and ECC memory. Another mistake was to get a Coolermaster 130 case, which was nifty but limited me to an ITX and three HDDs. That was physically wrong. I've drug out of the closet an old tower and moved everything there. Another was to start with a 3 drive z1 raid instead of the 4 drive/2vdev mirror I have now.

I even did a backup server on a $15 ECS 1ghz board, which did fine in that very limited role, except that it didn't reliably POST. Nothing to do with FreeNAS.

What I have learned is that the pool (and its backups) is king. The most awesome thing about FreeNAS is the ability to do a new install and move the pool over. Which I intend to do when I see the right used server board.

 

[Harrisonm65]

@elorimer this is exactly the sort of answer I was looking for. As this is my first step into anything FreeNAS related I wanted to keep my budget as small as possible, but I also know I'd be better off with certain specs. Im opting for 4 drives in z2 setup when I finally purchase them, and while I like the ITX size cases, I knew I wanted more drives plus a little more cooling and airflow, so re-used an old NZXT tower I had lying around.

Thank you though for sharing your experiences, both good and bad. Always nice to see what lessons people learned along their own journey.

 

[Yorick]

You’d be better off with 4 in mirror for IOPS and extending, and a little better off with 4 in raidz2 for pool resilience.

I like 6 in raidz2, minimum 5, to make that worthwhile. Keep in mind a raidz cannot be made wider after the fact

 

[elorimer]

Im opting for 4 drives in z2 setup

Before you settle on this, something I learned was that you need a backup, and backup of the backup. That can be of datasets on the pool, so you don't have to have backup space equal to the size of the pool, or have the backups in the same place, or have backups with the same performance. So if you have an 8TB pool, you might be able to back it up onto a 4tb external, or two 2tb drives.

So if you are contemplating 4 4TB drives in z2, you will have an 8tb pool, 2 parity drives and no backup. You might be better off with 3 drives in z1 (also 8TB) and one to backup the pool. I ended up destroying my 3 drive z1 pool for the 4 drive mirror (to extend later, as @Yorick notes, by changing only two drives instead of three, and also informed by having a drive in the pool fail). I used my backups to restore the datasets to the new pool.

So figure out the backup of the pool at the same time you design the pool.

 

[Harrisonm65]

Im only currently playing with approx 3.5TB of data. This is why I though 4 x 4TB drives in z2 would give me double the storage Im using so room to add more date, plus the resiliency. @elorimer ultimately I will purchase a large capacity external for backups, but for now I am going to repurpose some 2TB drives I have around here in caddies to back up to. I know that will be a little clunky having the backups over multiple drives but Ive not seen yet any problems doing it this way, other than the inconvenience to me. If Im wrong thinking that though then please let me know. Its all a learning curve.

 

[elorimer]

We're both in roughly the same place in the shallow end of the pool (pun intended). I have about the same amount of data in my pool but I did it with 4 4TB drives in two vdev mirrors. I have another 2TB single drive pool for certain backups. I've divided my primary pool into datasets mostly for different backup situations, but actively snapshotted.

So one dataset is media, largely static with additions from time to time and a few deletions, around 1.7TB. I have that backed up to a 2TB external and occasionally bring it out to update.

Another dataset is a cloud sync backup of my OneDrive. I don't need a backup of that dataset.

Another dataset holds the backups of the PCs in the house. That is backed up to to Backblaze. Probably unnecessary

The last dataset is the active share of the PCs in the house: documents, downloads, etc. That changes actively and is vulnerable to accidental deletions, ransomware, and so on. That is actively snapshotted and replicated to that 2TB pool. If I lose this drive I have a spare I could replace it with.

So I have backups over multiple places, which works fine for me, I think, but would be a lousy outcome in an office setup.

 

[Constantin]

The rate at which stuff is written to disk will vary by camera, resolution, compression, and other factors such as the use of guard zones. I’ve installed a DVR at a school with an alleged capability of running 16 2K cameras and as best as I can tell, the two purple hard drives inside are not striped because adding a second one didn't trigger the need to reformat the whole array. Drives marketed towards DVR application (i.e. WD Purple, Seagate Surveillance, etc.) are not physically distinguishable from other 3.5 drives though the firmware is likely different.

That said, if you want high quality recordings, be prepared to pay for it. For example, I found that dedicated hardware DVRs (i.e. Hikvision, et al) are frequently not up to the task of running best quality into every port at once for the cameras they are marketed to. The vendors appear to be practicing price tiering as a function of throughput per port as well as the number of physical ports you can plug into. At the school, we had to upgrade to a higher-port model to enable the local cameras at high-quality, low compression (which is the only way you get usable content at night when the signal is noisy).

Both DVR models use the same number of drives (2), so I don’t think it’s the drives that are the bottleneck. Rather it is the limits imposed by the hardware or software inside the DVR. Guard zones are also your friend. They allow your DVR to observe continuously but only write to disk the events that trigger the movement detection system +/- 10 seconds. That cuts down on drive activity and increases the time your system will cover before overwriting old content.

TL;DR: You don't need a $$$ NAS system to cover the needs of a DVR application. However, I would research carefully the hardware/software running it as some hardware DVR solutions are documented so badly that you will be pulling your hair out in no time. It is kind of telling that these DVRs typically include a auto-reboot function that allows you to force the thing to reboot daily... and thanks to our experience it now does.

 

[Harrisonm65]

@Constantin wrong thread with your reply above?

 

[Constantin]

Nah. Just a reply for posts #14 and all that. Previous page. Didn't see you guys had moved on to page 2.

 

[alamoudi.moh]

@Harrisonm65, @elorimer

i think we are all doing it the same way :D start with something reasonable and then move to better server grade.

I like the idea of the backups that you are setting, which i was thinking about it and good i have someone who is doing it already.

 

[bigdadda06]

I built my original system with a motherboard and drives I had laying around.
Then later i "upgraded" it to with 2nd hand motherboard & CPU from eBay with 6 SATA ports plus some extra 2nd hand HDDs.
I've had a couple of the HDDs die, of course, but then i replace them with other 2nd hand drives from eBay.
With the "proper" Raid-Z configuration, SMART and email alerts setup I usually get plenty of warning when a drive is dying so have time to order another from ebay.

And now with info on how to check and mark bad blocks on the disk before putting it in the system, that process works even better.

 

[Harrisonm65]

And now with info on how to check and mark bad blocks on the disk before putting it in the system, that process works even better.

Where is that info please? Thats something Id like to read up on. Im putting new drives in but you can never be too careful I guess.

 

[Redcoat]

Where is that info please?

Here's one place https://www.familybrown.org/dokuwiki/doku.php?id=fester112:hvalid_hdd - see Resources, Fundamentals, Uncle Festers Guide.

Here's another https://www.ixsystems.com/community/resources/hard-drive-burn-in-testing.92/. In that one, also check out the jgreco resource referenced in the first sentence.

 

[Harrisonm65]

Here's one place https://www.familybrown.org/dokuwiki/doku.php?id=fester112:hvalid_hdd - see Resources, Fundamentals, Uncle Festers Guide.

Here's another https://www.ixsystems.com/community/resources/hard-drive-burn-in-testing.92/. In that one, also check out the jgreco resource referenced in the first sentence.

Brilliant, thanks for that.