First NAS build, from the scratch. Any advise is welcome.

[karl_azytzeen]

Hi people,
let me sum up: I want to build a centralized storage for my house. I found that Freenas is the right way to go, so here I am.

My net landscape:
ISP-> FTTH 100/50 Mb With rebranded FRITZ!Box 3390 under the layer of ISP management.
Devices Connected to the net.
2 x Windows Desktop (1 multimedia high end, 1 multiporpose for net surfing and office)
5 x Laptop (Windows: 2 multimedia, 2 gaming) (Macbook Air)
6 x Phones (5 Android, 1 Iphone
3 x Tablet ( Windows, Android and iPad)
3 x Ebook reader
1 x Multimedia TV
2 x TV Box
2 x Printer/scanner
Obviously they are not working at the same time, but the net is overpopulated. Only wired devices are: High end desktop, TV and TV box. The rest are on 2.4 and 5 ghz Wi-Fi

My problem:
I use to keep imporatant files synced at least between two computers with resilio, plus everybody has its own dropbox, drive... and some high capacity flash drives, and me always asking for them to do their backups in external drives. But some recend dead disks has shown me that this system is no longer (it was working past my own computers??? i doubt it) working.
I want to have a centralized and automated backup system, with at least the most important devices synced (2 desktop, 2 laptop, 3 phone).
Also streaming/serving multimedia files to devices (i had no plans for 4k, so they will be 1080 for the next few years).
Keep system images plus, file archive for or photos etc.
Some other nice functins that i wuld love to centralize, but i dont even know if it can be done, music archive/player, ebook archive (any calibre plugin...?) and probably man other things i cant imagine.

But, by now, all the functions i can achive are a bit overwhelming to me, so i will start with the bulding project.

The ardware:
I had been lurking and reading the forums for a few weeks so I THINK, that i had something near to be able to work. ANY ADVISE is welcome and really apreciated as I can be totally wrong with my choices.

Board X11SSL-CF: it has the aditional SAS so i can populate with more disks aftewards.
Ram Kingston kvr21r15s4/16 (starting with one, yes lacking for the amount of TB i had but in a few time i will add another one. Does not apear in the board compatibility, so i will like to hear your sugestions, but kingston seems fine to me.
CPU: Xeon 1230 V6. I thinks is more horsepower than enought for what i want, but i dont mind if any bigger is needed.
Disk: I will start with 4x WD Red 8tb for the "important" files, and 2x WD shucked WD80EZAZ to keep multimedia and non-important files.
Case: Node 804, small enought and able to be populated with 10 disk. I see it to be easy modable to fit a lot more disk if i print new brackets, keeping an adecuated cooling. I had a few spare noctua fans over here.
PSU: I love seasonic, but to keep cost down, i will use a 600W LC-Power 650GP3 (i know jhonnyguru hates them, but this update will arrive in a few months, but i need to keep cost downs while investing in the most important hardware.
CPU cooling: I will start with stock cooler, i dont mind to delid the processor to keep it cool. If needed i can update later to any better cooler.
Boot drive: Here i had a 60Gb ssd that seems nice for it, but i read here that most people uses USB3.0 drives... can anyone explain me why they are better choice?

Regarding to the disk distribution, RaidZ1 seems the best way to go for the 4 disk. Can i later expand (without removing the data) to Raid Z2 just by adding 4 disk more? This way i can start with 20tb and later exand to more safe 40tb of usable space. For the non-important files, JBOD or just keep them independant?

Any idea on how to improve that?
PS: sorry for my english, but that's a nice way to practice a bit this too.

 

[Chris Moore]

Disk: I will start with 4x WD Red 8tb for the "important" files, and 2x WD shucked WD80EZAZ to keep multimedia and non-important files.

If you have six drives, just put all six together in a single RAIDz2 vdev (array) so all your data is protected. The storagepool can be subdivided with datasets if you need to keep the data apart for some logical reason.

i don't mind to delid the processor to keep it cool.

That should not be needed. The NAS is not a gaming system and the CPU should not be working so hard that temperature would be a problem. Most of the time, processor utilization will probably be under 10%.

Boot drive: Here i had a 60Gb ssd that seems nice for it

That is actually a perfect fit. The only reason that some people use USB is to save the SATA port for use by a hard disk drive. If you have a small SSD, use that because it will be more reliable in the long term.

RaidZ1 seems the best way to go for the 4 disk. Can i later expand (without removing the data) to Raid Z2 just by adding 4 disk more?

No, which is why I said to go with RAIDz2 to begin with. What you can do is have a RAIDz2 vdev in your storage pool and add another RAIDz2 vdev to the storage pool later to expand the capacity. At this time, it is not possible to change from RAIDz1 to RAIDz2 without removing the data first.

For the non-important files, JBOD or just keep them independent?

RAIDz1 is never suitable for drives larger than 1 TB.
I would put all the drives together under a single pool. If you put six drives at 8 TB each into a ZFS pool, protected by RAIDz2, it will give you about 43 TiB of raw capacity, but after redundancy of RAIDz2 and the overhead of checksum data and the 20% space reservation that is recommended to preserve performance of the pool, you are only able to put about 22 TiB of data into that pool. If you need 40 TB to start, you need more drives or larger capacity drives.

It might help you to understand this a bit better if you are able to review the following resources:

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://forums.freenas.org/index.ph...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Terminology and Abbreviations Primer
https://forums.freenas.org/index.php?threads/terminology-and-abbreviations-primer.28174/

 

[karl_azytzeen]

Hi Chris, thank you for your quick answer.
I took a look at both documents, and while they are very informative and comprehensive, i maybe have now more questions than answers.

If you have six drives, just put all six together in a single RAIDz2 vdev (array) so all your data is protected. The storagepool can be subdivided with datasets if you need to keep the data apart for some logical reason.That should not be needed. The NAS is not a gaming system and the CPU should not be working so hard that temperature would be a problem. Most of the time, processor utilization will probably be under 10%.

I read in the source you linked, that NTFS is not stable. I almost do everything with windows, does it had any implication that i didnt understood correctly? Cand this be solved going with exfat instead, for example?
Another thing, windows sends and recieve files with Samba, single threaded does this means i cant send or recive multiple files at the same time? Im missing something here? (looks really weird for a high performance data transmission standpoint).
When the machine is fully is working, i had asumed i can create private folders for each user, password protected, there is any trouble about having an admin account that can acces and manage all folders andits contents? Having password locked folders means that they are cyphered?

Regarding to the drives, looking at the reddit threads, the EZAZ drives are suposedly HGST Helium 10 with 256Mb of cache and reduced spin speed, but needs TLER manually activated on boot. VS the WD RED that has native TLER but only half the cache (does this impact in the ZFS environament performance?)
Since i read the first backblaze statistics i start getting HGST drives and they never failed me. And they are cheaper too.
What is the best route to go?

That is actually a perfect fit. The only reason that some people use USB is to save the SATA port for use by a hard disk drive. If you have a small SSD, use that because it will be more reliable in the long term.

Yep. but, can i later move the OS to another drive (USB) if i need the sata interface to add more drives?

No, which is why I said to go with RAIDz2 to begin with. What you can do is have a RAIDz2 vdev in your storage pool and add another RAIDz2 vdev to the storage pool later to expand the capacity. At this time, it is not possible to change from RAIDz1 to RAIDz2 without removing the data first.
RAIDz1 is never suitable for drives larger than 1 TB.
I would put all the drives together under a single pool. If you put six drives at 8 TB each into a ZFS pool, protected by RAIDz2, it will give you about 43 TiB of raw capacity, but after redundancy of RAIDz2 and the overhead of checksum data and the 20% space reservation that is recommended to preserve performance of the pool, you are only able to put about 22 TiB of data into that pool. If you need 40 TB to start, you need more drives or larger capacity drives.

After reading the pdf, and with that in mind (please correct me if wrong), going for 4x8Tb RaidZ1 with such big drives is an idea heading for disaster. Im absolute noob, but there is any better number of drives vs usable space ratio? If i understood correctly, i will create a pool, which inside will have the vDev containig my disks, this vDev is the one that will be on RaidZx.
To expand avaible space i can: 1- Vdev cant be expanded in number of drives, but i can replace the drives with larger ones, one at a time with the sistem integrating it (resilvering???), when all drives are replaced the system will expand with the increased added storage space.
2- Add a new vDev.

Looking for futureproof expandability, what is the better way to go? If i understood, i will be safer with an 10 (should be 11?)drives raid z3(any 3 disc can fail without loss), instead of 2vdev x5 raidz2 (2 drives per vdev fail without loss), but also the first one have 64% of usable space vs 56 with the other option (gross numbers of the zfs storage calcultor I found).
Correct me if wrong but that 8% is the penalty that must be paid if you dont have another storage to save data while recreating the pool and vdev with the new drives?
Plus, is a good idea to mix new and used drives?? (i read about get them in different batches to avoid manufacturing issues that can kill all drives in a short time window).

And talking about the rest of the hardware, im missing something with the board, i can get a cheaper one that will work better for me? I read some old post about X10 boards with kingston ram not working properly, is this issue still a problem?

Thanks everyone for its time :)

EDIT: Edited to improve its readability and some weird phrases i wrote.

 

[IQless]

Yep. but, can i later move the OS to another drive (USB) if i need the sata interface to add more drives?

The easiest way is just to do a fresh install to a USB stick and upload the config file (make sure to backup your config file!). I would not advise you to switch back to a USB stick, if you have room for it, add an HBA instead :)

1- Vdev can't be expanded in number of drives, but i can replace the drives with larger ones, one at a time with the sistem integrating it (resilvering???), when all drives are replaced the system will expand with the increased added storage space.
2- Add a new vDev.

1: Correct,you can not expand an existing RaidZ* vdev (at least at the moment).
2: Also correct.

Looking for futureproof expandability, what is the better way to go? If i understood, i will be safer with an 10 (should be 11?)drives raid z3(any 3 disc can fail without loss), instead of 2vdev x5 raidz2 (2 drives per vdev fail without loss).

This often comes down to how you want to start. Do you intend to go with 10-11 drives from the get-go? Or do you plan on starting with 5-6 drives?
Often we see that people start with something like a 6drive RaidZ2, and later on expand the pool with another 6drive RaidZ2 when they need/can afford it.
Multiple vdevs in a pool also give you greater performance, which for some people is necessary.

Plus, is a good idea to mix new and used drives?? (i read about get them in different batches to avoid manufacturing issues that can kill all drives in a short time window).

There is no problem mixing new and used drives, but there are multiple things to remember. The vdev will only be as fast as the slowest drive, so if you mix 7200rpm drives with 5200rpm drives the performance will be as if the vdev was filled with only 5400rpm drives.
Also, make sure to burn in the drives, both new and old.

 

[karl_azytzeen]

Hi, the project is starting to become a more defined thing.

The motherboard will be x11ssh-ctf. coming with integrated sas 3008 and intel x550 nic is long term saving over more simple boards that will need aditional cards later. (a 10g nic ASUS XG C100 is in the way to upgrade my main desketop for direct atachment to the nas).
I'm in the process of getting the drives from diferent supliers, but is slow as im waiting for special offers (165-180€ each drive) where i can save some $$. I had 5 WD80EZAZ by now, and ordering 2 more over the next phew days, and 2 more in october if possible. I think that my best bet to start with reasonable performance is to have 2 vdev, raid 10 each, with 8 disk in total, and have an spare one. The upgrade path here will be: create a new zpool, with 8 new drives 8tb each in z3, move the data to the new pool and create a new vdev with all the old drives joining it to the new pool? Any issue with this?

The processor i had spotted is the xeon 1230 v6.
Also i take a look at diferent infiniband nics en ebay that are dirty cheap, but im not sure about driver compatibility, founding HP, dell, and other server brands, but im not able to find if there is compatible drivers, as vendor webs are outdates or with broken links.

My main question that still ramains is: Wich memory choose??????
I can get this models:
Kingston
KTH-PL424/16G
KVR24R17D8/16MA
KVR24E17D8/16MA
KVR21R15D4/16
KVR24R17S4/16

Crucial
CT16G4RFD824A

EDIT: I will get this ones: Kingston KVR24E17D8/16MA
Also inserted amazon links to the parts im using, to suport this freat forum!

 

[The Wallet Inspector]

Hi,
Was looking at getting that board as well. (Supermicro X11SSL-F)

Anyone know if the kingston ram kvr24r17s4/16 is compatible with it? It's DDR4 and the MHz range looks ok but cant see it listed on the Kingston website.

 

[Johnnie Black]

Anyone know if the kingston ram kvr24r17s4/16 is compatible with it?

No, that's registered RAM, note the R in the model number, you want unbuffered ECC, for Kingston RAM there will be an E in the model instead, e.g.:

KVR24E17D8/16

 

[The Wallet Inspector]

Ohhh, right. Got it, thank you.

 

[karl_azytzeen]

Im still in the process of getting the parts, mainly the mainboard as it is out of stock.
I recomend you to chek the kingston compatibility page as if is a lot more complete than the supermicro counterpart.

 

[The Wallet Inspector]

Thanks Karl, yeah i went with 2 sticks of KVR24E17S8/8. 16gig should be enough for now.

 

[karl_azytzeen]

Small update, this is still alive, after some troubles finding the board, stock shortages, ludicrous prices movements, etc...
Finally the thing arrived this weekend!!

 

[pro lamer]

Hi, are the WD80EZAZ working fine for you?

Sent from my phone

 

[karl_azytzeen]

Sorry, i had been busy. Im pending on moving to a new house, onece everithing is settled i will work on the server again.
Thanks for your intrest, i hope they will work fine, anyway will update any new.