First server build

[julong]

Hello, I have started to build a new server and I am starting to get overwhelmed by it. I have questions mainly about raid configurations and if I need a raid controller. Also if I can use a V2 Cpu on the motherboard I am thinking of buying. All my HDD will be segate exos x12tb. I am planning on starting with 4 of them and want to do raid 6. But I cant find a motherboard that supports raid 6; only raid 0,1,5,10. Eventually I want to get more HDD as I am running out of space quickly. I also want to be able to do 4k streaming.
Already bought
Case - Fractal Design define R5
Power Supply- Seasonic Focus Plus 850 gold full modular
Fans - noctura 2x 120mm and 1x140mm
Planning to buy
motherboard - supermicro atx X10SRL-F-O (says only supports raid 0,1,5,10) ( do i need a raid controller if so which one) - how do I get more than 10 hdd attached to this motherboard
Ram - I want to use ECC but need a recommendation
Thermaltake water 3.0 extreme s 240mm liquid cooling cpu system
CPU - Xeon E5-1620 v3 @3.50 -passmark -9766- not sure if enough for 4k streaming
CPU- Xeon E5-2643 v2 3.30 passmark 12957 - not sure if supported
HDD - segate exos x12tb


I would greatly appreciate any help here. Let me know if I am missing anything. I am thinking of using freenas. using Plex media server for streaming.

 

[Basil Hendroff]

If you're planning on using FreeNAS, you should not be running hardware RAID at all. https://www.freenas.org/blog/a-comp...are-design-part-i-purpose-and-best-practices/

 

[jsherm101]

Welcome to the FreeNAS community.

A few things to note that hopefully help:

• FreeNAS uses ZFS as its filesystem. ZFS offers a lot of great features and lets you pool your drives for redundancy much like RAID. RAIDZ2 is what is "RAID6" for comparison (2 drive redundancy). However, as mentioned above, you do not use a RAID card/controller with FreeNAS. The FreeNAS operating system and CPU handle the pools directly.
• I highly, highly recommend reading this presentation from @cyberjock which goes over a lot of this, including some of your questions around whether or not to use ECC RAM. The TL;DR is to treat your system config as well as you want to treat the data it will ultimately store. ECC RAM adds reliability and is recommended at 1GB per TB.
• My (weak) understanding of transcoding is that there aren't any reasonable CPUs on the market that will reliably transcode 4k content. At around 14-16,000 Passmark I think you can reliably transcode a single stream, but really GPUs are far better designed for this work and FreeNAS (or FreeBSD rather) doesn't have the kind of driver support needed for GPU support. You're better off setting up devices that can directplay the HEVC 4K content (for example, most GeForce and Radeon graphics cards can directplay 4K content, as can the NVIDIA Shield TV) - The Plex forums are a better place to understand what kind of computing requirements you'll need for 4K transcoding. Just keep in mind the same CPU is being used upstream for your FreeNAS installation.
• You'll want to leverage additional SAS or SATA controllers via PCIe to add to the number of disks you can connect to the computer. I would highly recommend carefully checking brand names to ensure compatibility with FreeNAS

 

[julong]

Welcome to the FreeNAS community.

A few things to note that hopefully help:

• FreeNAS uses ZFS as its filesystem. ZFS offers a lot of great features and lets you pool your drives for redundancy much like RAID. RAIDZ2 is what is "RAID6" for comparison (2 drive redundancy). However, as mentioned above, you do not use a RAID card/controller with FreeNAS. The FreeNAS operating system and CPU handle the pools directly.
• I highly, highly recommend reading this presentation from @cyberjock which goes over a lot of this, including some of your questions around whether or not to use ECC RAM. The TL;DR is to treat your system config as well as you want to treat the data it will ultimately store. ECC RAM adds reliability and is recommended at 1GB per TB.
• My (weak) understanding of transcoding is that there aren't any reasonable CPUs on the market that will reliably transcode 4k content. At around 14-16,000 Passmark I think you can reliably transcode a single stream, but really GPUs are far better designed for this work and FreeNAS (or FreeBSD rather) doesn't have the kind of driver support needed for GPU support. You're better off setting up devices that can directplay the HEVC 4K content (for example, most GeForce and Radeon graphics cards can directplay 4K content, as can the NVIDIA Shield TV) - The Plex forums are a better place to understand what kind of computing requirements you'll need for 4K transcoding. Just keep in mind the same CPU is being used upstream for your FreeNAS installation.
• You'll want to leverage additional SAS or SATA controllers via PCIe to add to the number of disks you can connect to the computer. I would highly recommend carefully checking brand names to ensure compatibility with FreeNAS

That presentation is quite a lot to handle. So I am not wanting to use command codes cause I will undoubtedly mess something up.

So can I use raidz2 with FreeNas? How do I know if my motherboard is compatible with it? The motherboard I picked is a server board.
If I use raidz2 can I add HDDs to it or do I need to make new raidz2 when getting more HDD

So I will use the CPU - Xeon E5-1620 v3 @3.50 then since the motherboard says it supports Xeon E5=1600 v3/4 (should this CPU be good enough for multiple 1080 streamings?)

Again I want to use ECC ram but I need a recommendation for the board I am planning. Unless you guys have a better board to pick.

 

[Mannekino]

That presentation is quite a lot to handle. So I am not wanting to use command codes cause I will undoubtedly mess something up.

Hi, just my personal opinion but if you want to do FreeNAS "properly" you should be getting comfortable with using the command line. While the GUI is pretty neat you will have to use the CLI for fine tuning and the more comfortable you are with using the CLI the faster you can do stuff. Things like fine tuning fan thresholds, permissions for shares, editing configuration files for jails and more is done more easily through the CLI.

But it really all depends on your use case. If you just want to create a local storage system with some shares this can all be done through the GUI. Personally I see FreeNAS as something for enthusiasts (in a home use setting) who also want to learn more how it works under the hood and for that you will be needing the CLI.

So can I use raidz2 with FreeNas? How do I know if my motherboard is compatible with it? The motherboard I picked is a server board. If I use raidz2 can I add HDDs to it or do I need to make new raidz2 when getting more HDD

RAIDZ2 is a property of the ZFS file system, you can compare this to RAID6 from traditional RAID controllers. It's not dependent on the type of motherboard. The most important thing is that FreeNAS has direct access to the disks. You could view this as software RAID. You can attach the disks to the SATA ports of your motherboard or use a something like Host Bust Adapter (HBA). LSI SAS RAID controllers flashed into IT mode are quite popular among FreeNAS users. I have a Dell PERC H200 which is based on the LSI 9211-8i flashed into IT mode so FreeNAS has direct access (flashing it into IT mode disabled the RAID functionality of the card itself).

There are some great resources on the recommened hardware for FreeNAS you can find those here:

https://forums.freenas.org/index.php?resources/categories/hardware.1/

You could start with this one:

https://forums.freenas.org/index.php?resources/hardware-recommendations-guide.12/

 

[julong]

Hi, just my personal opinion but if you want to do FreeNAS "properly" you should be getting comfortable with using the command line. While the GUI is pretty neat you will have to use the CLI for fine tuning and the more comfortable you are with using the CLI the faster you can do stuff. Things like fine tuning fan thresholds, permissions for shares, editing configuration files for jails and more is done more easily through the CLI.

But it really all depends on your use case. If you just want to create a local storage system with some shares this can all be done through the GUI. Personally I see FreeNAS as something for enthousiast who also want to learn more how it works under the hood and for that you will be needing the CLI.



RAIDZ2 is a property of the ZFS file system, you can compare this to RAID6 from tradtional RAID controllers. It's it not dependent on the type of motherboard. The most important thing is that FreeNAS has direct access to the disks. A better way of looking at it that ZFS is a built-in software RAID system. You can attach the disks to the SATA ports of your motherboard or use a something like Host Bust Adapter (HBA). LSI RAID controllers flashed into IT mode are quite popular among FreeNAS users.

So I really want to use it for a media server. Mainly using it as a plex server and possibly for music around the home not much else really.

Ok so with Raidz2 I can just plug in my HDD to the motherboard (or when I run out of sata ports add a pci sata adapter?) So can I keep adding drives to the same Raidz2? or do I need to make a new raidz2 like I would have to using raid 6. Also as I get more exos 12tb hard drives would it be possible to change to raidz3 or no?

 

[Mannekino]

Installing Plex through the GUI worked fine for me. Pretty much worked out of the box but you do need to know some basics about creating datasets, linking a dataset to a jail and how to go about with permissions. This is covered pretty well in the documentation.

I'm no expert on ZFS but I believe it works like this. You physical disks are combined in what is called a VDev. A VDev has a certain RAIDZ level like RAIDZ1 or RAIDZ2 but it can also be a stripe or a mirrored VDev. One of more VDev can be used to create a storage pool. At this time it's it not possible to expand an already existing VDev (I believe this is in development). But I believe you can create a new VDev with new pyshical disks and add that VDev to an already existing storage pool thus expanding the available storage in that pool.

Depending on the type of board you can get quite a bit of SATA ports on a Supermicro board. Personally I like the HBA route because you can use the SAS to SATA breakout cables which provides a cleaner build in terms of cable management. Also with ZFS it's possible to move your disks to another system and import the ZFS pool. So lets say your system breaks down, motherboard dead. No problem you can take out the disks and attach them to another system and import the ZFS pool.

 

[IQless]

If I use raidz2 can I add HDDs to it or do I need to make new raidz2 when getting more HDD

At the moment, you can not add more drives to an existing vdev. That means, that if you have a 6x12TB RaidZ2 vdev, you can not just add 2 more to change it to an 8x12TB RaidZ2 vdev. If you want to do that, you will have to destroy the pool and recreate it as an 8drive vdev.
What you can do is extend an existing pool by adding another vdev. You should read the whole presentation @jsherm101 pointed out, and when you are done, read it again. It will serve you good in the long run to understand what ZFS is and what it requires.

When setting up your FreeNAS, it is better to do it right from the get-go, than to have to rebuild it later because you did not bother to do it right from the start. There is a lot of people that have made this mistake.

 

[julong]

Installing Plex through the GUI worked fine for me. Pretty much worked out of the box but you do need to know some basics about creating datasets, linking a dataset to a jail and how to go about with permissions. This is covered pretty well in the documentation.

I'm no expert on ZFS but I believe it works like this. You physical disks are combined in what is called a VDev. A VDev has a certain RAIDZ level like RAIDZ1 or RAIDZ2 but it can also be a stripe or a mirrored VDev. One of more VDev can be used to create a storage pool. At this time it's it not possible to expand an already existing VDev (I believe this is in development). But I believe you can create a new VDev with new pyshical disks and add that VDev to an already existing storage pool thus expanding the available storage in that pool.

Depending on the type of board you can get quite a bit of SATA ports on a Supermicro board. Personally I like the HBA route because you can use the SAS to SATA breakout cables which provides a cleaner build in terms of cable management. Also with ZFS it's possible to move your disks to another system and import the ZFS pool. So lets say your system breaks down, motherboard dead. No problem you can take out the disks and attach them to another system and import the ZFS pool.

I'll have to look into HBA is or what they do. Can you give a run down of it? Would it be better to just use HBA instead of sata on the board or use that then add HBA.

Also is what i have picked so far good for expandability and use for freenas?

Can anyone recommend good ECC ram for the motherboard I picked.

 

[Mannekino]

You are fine with using the SATA ports on the board. An HBA is just a storage controller that you plugin a PCIe slot and it provides you with additional SAS or SATA ports. I'm using two SATA ports of my Supermicro board for two SSDs and my HBA for the HDDs.

If this is your board I see you have 10 SATA ports which is quite a bit https://www.supermicro.com/products/motherboard/Xeon/C600/X10SRL-F.cfm
The board looks pretty good to me, I think it ticks all the boxes for a nice FreeNAS system.

I'm not comfortable giving any RAM recommendations I don't know enough about it but Supermicro has a list of tested memory for the board see the first link in the panel to the right.

 

[julong]

You are fine with using the SATA ports on the board. An HBA is just a storage controller that you plugin a PCIe slot and it provides you with additional SAS or SATA ports. I'm using two SATA ports of my Supermicro board for two SSDs and my HBA for the HDDs.

If this is your board I see you have 10 SATA ports which is quite a bit https://www.supermicro.com/products/motherboard/Xeon/C600/X10SRL-F.cfm
The board looks pretty good to me, I think it ticks all the boxes for a nice FreeNAS system.

I'm not comfortable giving any RAM recommendations I don't know enough about it but Supermicro has a list of tested memory for the board see the first link in the panel to the right.

Yeah that's the board. I figured 1 sata for SSD for my operating system. And I can afford about 5 exos 12tb right now. And later I'll add 4 exos 12 the once I fill up the first 5. Running the 5 HDD in raidz2 and the 4 in raidz2 should give me almost 60 tb before needing to expand.

So when I connect my HDD to the mother board after installing freenas I just tell it to run them as raidz2?

 

[Mannekino]

Yeah that's the board. I figured 1 sata for SSD for my operating system. And I can afford about 5 exos 12tb right now. And later I'll add 4 exos 12 the once I fill up the first 5. Running the 5 HDD in raidz2 and the 4 in raidz2 should give me almost 60 tb before needing to expand.

Be sure to check if the drives will work with FreeNAS I've read some things of those very high capacity drives not working properly but I don't know enough about it.

Also some things to take into consideration. It's generally not recommended to fill up the entire pool. I believe you shouldn't go beyond 70%. I don't know enough about this yet but I think it has something to do with ZFS being a Copy on Write filesystem but I still need to research this topic.

This will leave you with the following amount:

5x 12 TB = 60 TB - 2 x 12 TB for RAIDZ2 = 36 TB not going over 70% = 25 TB for the first VDev
4x 12 TB = 48 TB - 2 x 12 TB for RAIDZ2 = 24 TB not going over 70% = 16.8 TB for the second VDev

Effective total storage = 41.8 TB (and there is some overhead cost as well but I don't know the fine details about that).

See this comment for example https://forums.freenas.org/index.ph...-how-far-past-that-is-safe.39486/#post-244354

Look up the post history of that dude, he knows his stuff

Personally I'm using RAIDZ1 for a 4 drive VDev I find this an acceptable risk for the type of data I'm storing so I'm only losing one drive for parity.

So when I connect my HDD to the mother board after installing freenas I just tell it to run them as raidz2?

After installing FreeNAS you can use the GUI to create a storage pool, in this process you can configure the type of VDev.

 

[julong]

Be sure to check if the drives will work with FreeNAS I've read some things of those very high capacity drives not working properly but I don't know enough about it.

Also some things to take into consideration. It's generally not recommended to fill up the entire pool. I believe you shouldn't go beyond 70%. I don't know enough about this yet but I think it has something to do with ZFS being a Copy on Write filesystem but I still need to research this topic.

This will leave you with the following amount:

5x 12 TB = 60 TB - 2 x 12 TB for RAIDZ2 = 36 TB not going over 70% = 25 TB for the first VDev
4x 12 TB = 48 TB - 2 x 12 TB for RAIDZ2 = 24 TB not going over 70% = 16.8 TB for the second VDev

Effective total storage = 41.8 TB (and there is some overhead cost as well but I don't know the fine details about that).

See this comment for example https://forums.freenas.org/index.ph...-how-far-past-that-is-safe.39486/#post-244354

Look up the post history of that dude, he knows his stuff

Personally I'm using RAIDZ1 for a 4 drive VDev I find this an acceptable risk for the type of data I'm storing so I'm only losing one drive for parity.



After installing FreeNAS you can use the GUI to create a storage pool, in this process you can configure the type of VDev.

Thanks I'll keep that in mind. So after reading what he wrote. I should never delete a movie from my system? Not that I will but I was just wondering.

 

[Mannekino]

Well it is highly unlikely that you will end up in a situation where you have only ever written to the disk and never deleted anything. Also how are you going to handle for example downloads from a Torrent client and copying/moving those files? How are you going to deal with archived files? You going to keep everything indefinitely? Even the archives files after you extracted them? That would be a waste of space.

 

[julong]

Well it is highly unlikely that you will end up in a situation where you have only ever written to the disk and never deleted anything. Also how are you going to handle for example downloads from a Torrent client and copying/moving those files? How are you going to deal with archived files? You going to keep everything indefinitely? Even the archives files after you extracted them? That would be a waste of space.

I'll probably try to keep the majority if not all torrents. I like watching things again after a long time not watching. Also perhaps my kids will want them. Why Would i archive files?

Can I just move files to my server from a USB external HDD?

 

[Mannekino]

I'm talking about "archived" files, like RAR files you download and then extract. Afterwards generally you want to only keep the extracted files and delete the archived. What I'm trying to get at, and so is @jgreco is that it very implausable you'll end up with a server that has no deletions.

Unfortunately for you, with jails and things like transmission running

Plex runs in a jail

 

[julong]

Also should I used a SSD pci or SSD on a sata? For holding the OS

 

[IQless]

An ordinary SATA SSD is more than adequate for the OS.

 

[julong]

I'm talking about "archived" files, like RAR files you download and then extract. Afterwards generally you want to only keep the extracted files and delete the archived. What I'm trying to get at, and so is @jgreco is that it very implausable you'll end up with a server that has no deletions.



Plex runs in a jail

Maybe I don't understand I only have movies and I have never extracted any of them that I know. I will continue to use my laptop to get movies then transfer to Nas. Is it possible to transfer using USB external hard drive?

What do you mean Plex runs in a jail?

Also I would put Plex server on ssd that I hold my OS on correct?

 

[Mannekino]

Your board of choice supports Supermicro SATA DOM and @Chris Moore seems to be a big fan of that for the OS

https://www.supermicro.com/products/nfo/SATADOM.cfm

Otherwise just get a good small sized SSD

I'm using 2 USB drives in a mirrored setup for FreeNAS itself.