Small Home-NAS build: searching for advice

[Stevie_1der]

Hello everyone!

I'm currently thinking of building a small home-NAS, and FreeNAS sounds as what I need.

Purpose:
Network storage and backup for 2 Windows PCs and some Android smartphones.
Current data amount is about 3TB in total, so I think I should be fine with 6 to 8 TB of NAS net storage (please correct me if you thinkt this is too small).
Maybe I could add Plex support, but not as a primary goal, just as an additional goodie.
No DB, mail, or OwnCloud servers, or stuff like that are planned.

I want a rather compact build, so I'm planning with a 4-bay case and Mini-ITX.

So I searched and came up with the following configuration:
Case: Supermicro SuperChassis 721TQ-250B (4-bay Mini-ITX with 250W Flex-ATX PSU)
Mainboard: Supermicro A2SDi-4C-HLN4F (Atom C3558, 4x2.2GHz)
RAM: Samsung DIMM 16GB, DDR4-2400, CL17-17-17, ECC (M391A2K43BB1-CRC)
Boot drive: SanDisk Ultra Fit 32GB, USB-A 3.0 (SDCZ430-032G-G46) USB thumb drive
Storage: WD Red or Seagate IronWolf

Now for the storage configuration, what would you recommend?
Mirror with 2 disks (leaves space for a later upgrade for another mirror) or RAIDZ1 with 3 disks, or even RAIDZ1 with 4 disks or RAIDZ2 with 4 disks?

To save some bucks, I could change the Mainboard to an ASRock Rack with a Celeron G3930 (2x2.9GHz).
I think both the Atom C3558 and the Celeron G3930 will be sufficient, is this right?
But I could easily pick another CPU instead for the E3C232D2I, like for example a Pentium Gold G4560 (2x3.5GHz), Core i3-7100 (2x3.9GHz), or a Xeon E3-1225 v6 (4x3.3GHz) if it would be needed.
This could be the case for transcoding with Plex.
Does anyone know if the Plex plugin supports hardware transcoding, especially with a C232 chipset (without graphics ports)?
If not, would it work with a C236 board?
I have searched a little, and it seems the current Plex plugin for FreeNAS 11.2 doesn't support hardware transcoding at all, so I think I can forget that for now and go with a G3930 or G4560, these should be able to transcode 1080p in software, if needed.

What would you pick for the task, Supermicro Atom C3558 board, or ASRock Rack + Celeron/Pentium?

And now some more detailed questions:
The Supermicro case has a backplane built in, will the HDD LEDs only function if the sideband header and/or activity LED header is populated?
If yes, has one of my selected mainboards a suitable connection for that?
I guess no, but I hope I'm wrong...

What about the USB thumb drive as boot device, will that last or is it likely to die after a short time?
I can read very mixed opinions about that, from "works since years" to "dies every few weeks".
So I thought of using a SSD (or 2 in mirror) instead.
But even the cheapest SSDs have minimum 120GB, and it would be a waste if I only put the boot files onto them.
Is it possible to use the remaining space for something else, like for example the system dataset?
That would have the additional benefit that I could send the data HDDs to spindown mode (like after 1 or 2 hours of idle time).
Or it could be used for the plugin jails.

I'm looking forward to your answers and advice!

 

[Yorick]

Boot: USB 3.1 stick might overheat. Sandisk improved heat flow over the 3.0 model, and still. I'd go with the USB 2.0 model, and use two in a mirror. https://www.amazon.com/SanDisk-Cruzer-Low-Profile-Drive-SDCZ33-032G-B35/dp/B00812F7O8/ref=sr_1_5

You can get 40GB SSD drives off eBay. One or two of these are a good choice, if you have the room in the chassis for it. https://www.ebay.com/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=intel+320+40gb&_sacat=0

Plex and hw transcoding - I'm not 100% certain. I think that's a work in progress. There is a GPU driver that has to be enabled manually, though I haven't found the knob to do so. I also don't have a Xeon with a GPU, that might be why :).
I'd expect that hw transcoding will be supported within this year or so.

Rule of thumb is 2,000 passmark per stream for software transcode. The Atom has a passmark of 2,500 - you should be good for one transcoded 1080p stream. I wouldn't bet on anything more.

Nice case. I wouldn't know anything about backplanes or LEDs, can't help there.

Drives - looks like you'll be doing 4TB x 4, in raidz2? Leaving you with 8TB usable?

I've seen WD Red on eBay for 75 each with shipping, if used drives don't freak you out.

 

[KrisBee]

And now some more detailed questions:
The Supermicro case has a backplane built in, will the HDD LEDs only function if the sideband header and/or activity LED header is populated?
If yes, has one of my selected mainboards a suitable connection for that?
I guess no, but I hope I'm wrong...

Did you read the manual at https://www.supermicro.com/manuals/chassis/Mini-tower/SC721.pdf ?

AFAIU, conncetion to the sideband header is only required for SAS2, not SATA disks. I have a Asrock E3C226D2I installed in an Ablecom CS-M50 case which is basically the OEM version of the SC721TQ-250B as used in the FreeNAS mini. I'm using SATA drives and the front HDD leds all light up on disk activity for each populated drive bay.

 

[Stevie_1der]

Boot: USB 3.1 stick might overheat. Sandisk improved heat flow over the 3.0 model, and still. I'd go with the USB 2.0 model, and use two in a mirror. https://www.amazon.com/SanDisk-Cruzer-Low-Profile-Drive-SDCZ33-032G-B35/dp/B00812F7O8/ref=sr_1_5

You can get 40GB SSD drives off eBay. One or two of these are a good choice, if you have the room in the chassis for it. https://www.ebay.com/sch/i.html?_from=R40&_trksid=m570.l1313&_nkw=intel+320+40gb&_sacat=0

I just thought USB 3.0 would significantly increase boot speed, but if USB 2 sticks are more durable then I'll take one of those instead.

The Intel 320 is an even better idea, I didn't think of looking for used ones. But it seems they are sold quite cheap, the 40GB version is under 20€, and even the 80GB is at about only 25€.
Thanks for the hint!
I think this will be way better than some new cheapo SSD.

Stupid idea: Could I mirror a stick with an SSD?
That would increase integrity and read speed (at least I guess so), and only use up one SATA port...

Plex and hw transcoding - I'm not 100% certain. I think that's a work in progress. There is a GPU driver that has to be enabled manually, though I haven't found the knob to do so. I also don't have a Xeon with a GPU, that might be why :).
I'd expect that hw transcoding will be supported within this year or so.

Rule of thumb is 2,000 passmark per stream for software transcode. The Atom has a passmark of 2,500 - you should be good for one transcoded 1080p stream. I wouldn't bet on anything more.

Ok thanks, then I will keep that in mind for later.
But I think I'll go with the Celeron G3930, this has a passmark of 3,000, is even cheaper than the Atom, and I can upgrade it if the need should arise...

Drives - looks like you'll be doing 4TB x 4, in raidz2? Leaving you with 8TB usable?

Well, it would somehow hurt me to "waste" half the capacity, sounds quite bad if you hear it first...
RAIDZ1 with 3 or 4 disks is no good?

Did you read the manual at https://www.supermicro.com/manuals/chassis/Mini-tower/SC721.pdf ?

Yes I read it, it mentions SES-2, and that is a bus used for communication between backplane and HBA, the HBA can set for example the LEDs, and the backplane can report drive presence, temperature, fan error and stuff like that to the HBA.

I have done some research on that topic, as this got my interest.
The backplane can be set to SGPIO mode (which is another, more simple protocol only for drive presence and LED status afaik) by setting some jumpers.
The Supermicro A2SDi-4C-HLN4F comes with a SFF-8643 to 4xSATA + SGPIO cable, there seems to be no separate SGPIO header on the board for the SATA connectors.
The ASRock Rack E3C232D2I has no SFF connector, but there are two SGPIO headers on the board. I suppose one for SATA0-3, and one for SATA4-5(7) (the C236 WSI has 8 SATA ports). But there is no SGPIO cable included.

AFAIU, conncetion to the sideband header is only required for SAS2, not SATA disks. I have a Asrock E3C226D2I installed in an Ablecom CS-M50 case which is basically the OEM version of the SC721TQ-250B as used in the FreeNAS mini. I'm using SATA drives and the front HDD leds all light up on disk activity for each populated drive bay.

Thanks, I'm glad to hear that the LEDs work even without the cable!

 

[Yorick]

Stupid idea: Could I mirror a stick with an SSD?
That would increase integrity and read speed (at least I guess so), and only use up one SATA port...

Well, it would somehow hurt me to "waste" half the capacity, sounds quite bad if you hear it first...
RAIDZ1 with 3 or 4 disks is no good?

I think you could mirror an SSD with a stick, for a boot device. Speed of the boot device matters some, but not a lot.

raidz1 is perfectly fine if you're happy to take the risk of a two-disk failure. Resilvering a 4-disk raidz1 of 4TB per disk should take about 4 hours, give or take. Particularly if the data on that drive also exists elsewhere - as it should anyway, disk redundancy doesn't negate the need for backup - I think a raidz1 with 3 or 4 drives of 4TB each is an acceptable risk. Though you'll see some disagreement on that.

The concern with raidz1 is that two-disk failures aren't all that uncommon, particularly when drives get bigger and resilvers take longer.

 

[Stux]

I think I’d suggest starting with a 2 disk mirror in your situation. Leaves room for expansion without replacing drives, and you can even use bigger drives later if you do expand

 

[Yorick]

I think I’d suggest starting with a 2 disk mirror in your situation. Leaves room for expansion without replacing drives, and you can even use bigger drives later if you do expand

To expand, that'd be another mirror. If in the same pool, that gives no better resilience than raidz1, while using the same amount of disks for parity as raidz2. It can be the right choice, no doubt, depending on circumstances. I'm just not seeing the circumstances right now where it would be.

 

[Stevie_1der]

Hmm, more difficult than I thought, really.

I've done some more search here in the forum and found some interesting stuff from the links from Chris Moore's signature:
ZFS RAID size and reliability calculator 2017-05-17
ZFS Drive Size and Cost Comparison Spreadsheet

So I played a little with the numbers (I assume they are correct) and came up with the following results.
When I take the MTBF of a single disk of 1 million hours (1.0E6), the mean time to data loss would be the following:
A mirror of 2*6TB would give 5.9E9 hours, 2*8TB would give 5.6E9 hours (about 6000 times "better" than a single disk).
A RAIDZ1 of 3*3TB would give 2.0E9 hours, 3*4TB would give 1.9E9 hours (about 2000 times "better" than a single disk).
A RAIDZ2 of 4*3TB would give 5.8E12 hours, 4*4TB would give 5.3E12 hours (about 5.5million times "better" than a single disk).
A RAIDZ3 of 5*3TB would give 1.4E16 hours, 5*4TB would give 1.2E16 hours (about 13 billion times "better" than a single disk).
And just for completion, a mirror of 3 disks would be around 2.6E13 hours, and a mirror of 4 disks would give incredible 8E16 hours (about 9 trillion years, that is 660 times the age of the universe!).
Yes, these numbers are not to be taken literally ("my HDD fails after 1 million hours, not earlier"), they are just mathematical propability numbers.

But all that only covers data loss due to failing drive(s), and not deletion, theft, fire, asteroid impact and others.

So a backup is still needed, but what is the most practical solution?
I'm thinking of that for quite some time now...

• External HDD or a HDD in a hot swap cage:
  Only feasible for up to ~10 or 12GB of data, quite expensive for bigger capacities if you want to have multiple copies, and a single point of failure if the single backup HDD gets damaged.
• DVD/BD/M-Disc:
  Only suitable for (relatively) small amounts of data, but can be performed additionally for extremely valuable data.
• Tape devices:
  LTO tapes are cheap, the tape drives are not.
  RDX drives are cheap, the cartridges are far more expensive than a HDD of the same size.
  So not really an option unless you can get an LTO tape drive that fell off the back of a truck...
• Yet another NAS (with minimum hardware):
  As it is just a backup, it could be without (or with minimized) redundancy.
  But it consumes more space, energy and administration than just "dumb" backup media.
• Online storage:
  perfect as an off-site storage, but only suitable if you have a good internet connection.
  For me in good old Germany, the Third-World-Country in regards of broadband internet access, it would be impossible with my 16/2 MBit DSL (I would have to move elsewhere to get more bandwidth)...

 

[Yorick]

One more option: External drive cage connected via HBA. Chris Moore has more to say about that, no doubt.

 

[Stux]

I tend to use the second level nas backup plan.

Don’t even need to be co-located necessarily.

As you understand you need a backup, even if only to protest your data from yourself ;)

And that means the MTBF is not so critical. The important thing is you will *know* if you have to restore something from backup... and that’s unlikely anyway.

 

[Stevie_1der]

I just thought a little, and I begin to like the secondary NAS idea more and more...

How about a HP Gen7 N36L as backup NAS?
I found some in ebay with RAC (Remote Access Card) and 8GB ECC RAM for about 200€, is that a reasonable price?
Or should I go for a N54L instead?

Then I can put 3*4TB RAIDZ1 into that, and use it as an intermediary primary backup.
Maybe I'll rethink my plans for primary NAS until then and go with 6 disk RAIDZ2 instead...

 

[Ixian]

With your use case I'd look to something like Unraid for your primary media server, and use FreeNAS as your backup server. You'll find the former easier to deal with in general (particularly if you are new to all this) and cheaper to expand - by far - and the latter will give you good local data resilience as your level 2 backup.

Don't forget the 321 backup rule - 3 copies of a file, 2 stored locally, one offsite. Obviously you decide what files are worthy of belonging to all 3 - for example a lot of my media files (DVR recordings and such) I don't backup at all, because I don't care, and those files I do care a little more about I replicate to my second FreeNAS server. And then the files I really care about are also backed up to a Backblaze B2 bucket (which FreeNAS supports).

I don't want to start an Unraid vs. FreeNAS debate - because I think the comparision is stupid if you look at it apples to apples - and I no longer use it myself but in my experience it's a nice and easy setup for "home media server" use cases. I wouldn't trust backup files to it, really, which is why I recommend, if you do go with a second server, making that one FreeNAS and using it for backup duty/cloud syncing.

 

[Stux]

The problem is you can’t trust the primary copy to unraid if you can’t trust it for the backup either.

With non zfs you don’t know when a file becomes corrupt and needs to be restored from backup.

I believe there are file integrity plugins for unraid.

 

[Ixian]

True, but I was going off the assumption primaries would be quickly replicated to the backup minimizing the long term risk. Also, to your point: there are file integrity plugins for Unraid and in this use case they would work well. That's because the plugins just maintain a checksum record of the data as written - they can't fix any problems that may develop later, just tell you if a file changed. Recovery depends on you having a backup somewhere else, which in this case would be the FreeNAS server.

I don't know if this is an ideal use case, and I run with two FreeNAS servers myself, however I have used Unraid in the past and for a pure "store and serve all my media" use case it is pretty decent, and a lot more flexible due to using non striped arrays. Use it for bulk media that you probably don't care as much about backing up/protecting (for the most part) and takes up the most space and use FreeNAS for the important stuff, which for many won't be nearly as large in size.

 

[Stevie_1der]

I think I'll definitely stay with FreeNAS.
My planned NAS system should be a NAS in the very first place.

The whole Plex thing is just like the icing on the cake. Yes it would be cool if I could stream some movies to the TV, or audio books to my stereo.
But if this won't work, I'll just get a Raspberry or a Fire TV Stick with Kodi.

What about my plans with the HP Gen7 N36L?
Would that be ok, or is this thing just too outdated?

 

[Ixian]

When you say "NAS" what does that mean to you? Does media comprise 75% or more of your total storage footprint? There are valid reasons for looking at a non-striped array setup the biggest being they are easy to grow in to.

You're starting on the bottom end hardware wise and looking to scale up as your needs dictate. There's nothing wrong with that nor is it uncommon but FreeNAS, or more specifically ZFS, tends to skew more towards planning and building ahead, and tends to be more resource intensive - it's doing a lot, software wise, to warrant it. And you're on the light side, hardware wise, for both the systems you've mentioned. Not that it won't work, but it'll be harder/more expensive later to scale up.

The 36l has a fairly weak Athlon II Neo and I believe maxes out at 8GB memory. That is the barest of minimums for FreeNAS/ZFS, even in a backup server role. Expansion is also tight and it's a proprietary case. I feel like as cheap as it is you will not only grow out of it quickly but discover it doesn't meet your needs today.

You haven't said much about why you feel FreeNAS and ZFS are a good fit for you and your needs/use case. It's obvious to most everyone here why it's good for us - I mean, I have two FreeNAS systems and have been running them for half a decade - but you haven't outlined why you think it is good for you. Your use case will help us guide you a lot better.

 

[KrisBee]

The 36l has a fairly weak Athlon II Neo and I believe maxes out at 8GB memory. That is the barest of minimums for FreeNAS/ZFS, even in a backup server role. Expansion is also tight and it's a proprietary case. I feel like as cheap as it is you will not only grow out of it quickly but discover it doesn't meet your needs today.

The N36L supports 16gb - https://forums.freenas.org/index.php?threads/n36l-with-16gb.10050/ and with a small mod you can fit 6 HDD in the N36L, e.g: https://www.ebay.co.uk/itm/HP-ProLi...h=item3b2a1cbf65:g:0aYAAOSwYbdcWJaS:rk:1:pf:0

Anyway, if you want to know how a microserver with 8gb can perform, ping @Mannekino who has a N40L with 8GB. (PS his recent post re: slow replication is solved by combining the use of netcat with zfs send/receive)

 

[Mannekino]

Anyway, if you want to know how a microserver with 8gb can perform, ping @Mannekino who has a N40L with 8GB. (PS his recent post re: slow replication is solved by combining the use of netcat with zfs send/receive)

Well I still have to confirm that :) hopefully this week, if I'm not too tired tonight when I get home from work. I've used my Microserver N40L for about 5 years as a FreeNAS server. Only used it for local network storage (my media content) and Transmission. For that it worked just fine.

 

[Stevie_1der]

When you say "NAS" what does that mean to you? Does media comprise 75% or more of your total storage footprint? There are valid reasons for looking at a non-striped array setup the biggest being they are easy to grow in to.

Well for me it means a place to store my data in the very first place, in a way to prevent data loss.

What do I want to store?

• Videos and photos taken with my phone and personal documents.
  These are important and irreplaceable.
• Ripped and tagged music CDs and a few movies.
  These could be replaced with some effort.
• Downloaded copies of installed software.
  Most of it could be replaced.
• System backups (not including the data) of my PCs.
  Needed to restore in case someting breaks, without the need to manually reinstall every single thing.
• I work at home, as a software developer. Source code is maintained and backupped in remote Git repo, but I also have some hundreds of GB input and output data.
  Some of that could be replaced, some have to be kept, at least for a while.

As I said in my first post, total amount is about 3TB at the moment, only slowly growing.

You're starting on the bottom end hardware wise and looking to scale up as your needs dictate. There's nothing wrong with that nor is it uncommon but FreeNAS, or more specifically ZFS, tends to skew more towards planning and building ahead, and tends to be more resource intensive - it's doing a lot, software wise, to warrant it. And you're on the light side, hardware wise, for both the systems you've mentioned. Not that it won't work, but it'll be harder/more expensive later to scale up.

Yes I know that I'm a bit at the bottom end, but there are many enthusiasts here I guess.
To be honest, there was a time where I always was on the top end, and I spent a lot of money on stuff that I never really needed.
I don't think I really need a redundant cluster of dual-CPU systems with 256GB RAM, 4x 10GB NICs and 48 12TB-SSDs in mirrored RAIDZ3 (though it would be kinda cool to have something like that *Tim-Taylor-grunt*).

But I don't need any power for virtualisation, Domain Controller, mail server or something like that.
I don't want any huge and loud big towers or racks any more, but rather something compact and quiet.
So I thought a 4-bay mini-ITX case with a Pentium G4560 and 16GB of ECC RAM would be more than sufficient.
If a small 6 or 8-bay case existed (like the Ablecom CS-T80, but this has no known sources afaik), I'd love to go for that instead.
But the next thing to 4-bay mini-ITX with more bays is a full size tower with a huge depth.
The only exception is the Silverstone DS-380, but the plastic drive carriers look cheap, and it has no real backplane with individual LEDs and stuff, and there are some cooling issues reported.

The same with HDD space, I don't want to spend money on drives that are filled by the time when double the size cost half the price than today's space costs now...

The N36L was planned as an additional backup for my NAS-in-question only, and only for the important stuff when space were full.
So it would only receive a (weekly?) backup snapshot of the main NAS and stay powered off for the rest of the time.

Having written all that, how much power and space would I need?
I'm a little insecure now...

 

[Ixian]

I'd do a needs assessment because that will provide answers to the other questions - sounds like you are halfway there.

Since I've spent my professional life running enterprise software projects I tend to organize most of my home IT planning the same way I do when grooming feature requests, backlogs, etc.:

First, organize your needs - what you want your local NAS system(s) to do - in to broad buckets:

1. Must have
2. Should have
3. Nice to have
4. Don't care to have

Everyone overlooks #4 but I find it very useful to define what I'm *not* going to do/need when planning a project.

Second, sort your files in to categories using the 321 rule as follows:

1. Files that are easily replaceable, or lack of replacement would be a minor inconvenience at worst
2. Files that are not as easy to replace but not critical - losing them would be a large inconvenience but not the end of the world
3. Files that are irreplaceable

1 will probably be your largest category in terms of size. In my case it's things like DVR recordings from Emby or movies I've downloaded, etc. Not hard to replace and in a lot of cases I may not even care about replacing them.
2 is a little harder to define, lots of people just skip right to 3 which is OK, but review it anyway. In my case it's things like Bluray disks I've ripped from my collection, time machine backups, and such. Stuff I could replace, or ultimately could stand to lose, but would be a pain in the ass to deal with, basically. Enough so that it's worth expending the time and money to store them on another backup system locally.
3 is the stuff you really don't want to lose. Media wise for most people this is personal pictures, home movies, tax documents, and the like but totally up to you; your old game saves, hard to find movies, whatever is important enough to you that it's worth paying to store it somewhere outside of your house in addition to 2 places in it.

Your needs + Category 1 determines what your primary system should look like, Category 2 your local backup (system, external drive, whatever) and 3, for me, is cloud storage - Backblaze B2 in my case. I.e. category 3 files are stored on both my local systems and in the cloud. And so on.