First Build - Advice Appreciated

Kallman

Cadet
Joined
Jun 15, 2018
Messages
8
Hello everyone!

After spending a couple of weeks looking at different options for a DIY NAS, hardware recommendations and the merits of FreeNAS vs. alternatives like unRAID, I've been convinced that ZFS seems to be the way to go. While I would have preferred if ZFS was more flexible in terms of expansion, the anti-bit rot features are a much higher priority than expandability.

The use case for this build is a machine capable of safely storing personal files as well as being able to handle 4-5 simultaneous PLEX transcodes. It might also be used to run some jails such as BT Sync, Owncloud, etc.

My (albeit limited) understanding of ZFS is that it will keep data safe on a software level, meaning the biggest threats are damaged hardware (disk failure or disaster) or simple user error. A properly configured array should mitigate the damage of disk failures, but I also understand that "RAID(Z) is not a backup" and would very much appreciate any tips for a backup solution, such as if it would be advisable to build a second FreeNAS machine to keep off-site, or if it would be enough to backup snapshots to one or multiple smaller, less expensive devices.

The Build

Case:
Fractal Design Define R6
By ordering trays from Fractal's spare parts shop, the Define R6 will be able to fit up to 11 HDDS and 4 SSDs.

CPU:
Intel Xeon E3-1270 V6
I realize is that this CPU is most likely overkill, given that the hardware guide recommends the E3-1230 V5 and similar CPUs for heavy transcode use, but I managed to find one almost new at a very good price, and decided to snap it up while available. This is therefore the only part of the build that is more or less set in stone.

Motherboard: Supermicro X11SSL-CF or Supermicro X11SSM-F
The choice of motherboard mainly comes down to the amount of drives that will be be used in the system. If the total amount of drives (including SSDs) is 14 or less, I can get away with using the X11SSL-CF without a separate HBA.

RAM: Crucial 32GB Kit (2 x 16GB) DDR4-2400 ECC UDIMM

SSD: 2-4x Samsung 860 EVO 250GB
One SSD will be used as a boot drive, and one will be used for jails. I'm not sure of either of these are worth mirroring, but if they both are I would end up with a max of 4 SSDs. This particular model also seems to be a bit overpowered, at least for a boot drive, but the price difference for lower-capacity SSDs seems to be negligible unless I choose to step down to a lower quality products (such as a WD Green 120 GB, which I've heard bad things about).

Storage: 11x WD Red 4TB
I'm currently thinking of using 11 4TB drives in a RAIDZ3-array, which should result in around 30 TB of redundant storage. The only issue with this is that it would be impossible to expand without replacing every single drive in the array. The alternative would be to use 6x6TB drives in a RAIDZ2-array, with the ability to add another 5xN TB vdev later on. The question is whether it's better to have 2 vdevs capable of surviving 2 drive failures each, or 1 vdev capable of handling 3 drive failures. I've also heard that 5 drives is not an ideal number for a RAIDZ2-array, but I'm not sure how much of an issue that would be in practice.

PSU: EVGA SuperNOVA 850 G2 or Corsair RM850i
Based on the following calculation, the ideal size of a power supply would be 800 W, but given that they appear to be a bit uncommon, I have instead decided to go with a 850 W PSU. This should also enable me to add in an extra fan later on.

CPU: 72 W
Motherboard: 25 W
RAM: 12 (2x6) W
SSD: 16 (4x4) W
Storage: 385 (11x35) W
SAS Controller: 10 W
Case Fans: 90 (3x30) W

Total: 610 W
1.3x Total: 793 W

While the general recommendation seems to be to use Seasonic power supplies, they don't provide a model with enough cables to support the number of drives that will be used in this build, and don't appear to sell additional cables. Both of these PSUs come with 10 SATA connectors, and sell additional cable sets.

UPS: APC SMC1500I Smart-UPS C 1500VA LCD 230V
I'm honestly not sure how important a UPS is, but since the it has to be able to handle the power draw of the PSU, this 900 W model seemed like a good choice.


Thank you in advance for all your help!
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
Seems like you have done your research. Good !

A few points
This is a good case ,but if you are ok with rack mount chassis -- then a rack mount will usually give you more drives in the same space (24 drives in 4U) and also the hot swap capability. If rack mount is not your thing, then R6 in a great case.
It is overkill, but since you have bought it already, don't go changing it
Since you already bought the CPU, your choices in board and RAM are also limited. I would go with the -CF so I can save the money on the HBA provided the cost difference between SSM-F and SSL-CF is not more than $40-$60. Because you can get a decent HBA for about that price
That's too large. Go with a 120Gb Crucial, Adata or SanDisk. These cost about $30-$35. No need for mirroring the boot drive or the jails SSD
If you choose the rack mount chassis, your drive/storage needs will change, but for R6 I would do a 6 drive RAIDZ2 and a 5 drive RAIDZ2 in the same pool
Either should work. But rack mount chassis usually include PSUs and some chassis have dual redundant ones. Something to think about.
Good choice. Check the NUT compatibility to be sure
 

anmnz

Patron
Joined
Feb 17, 2018
Messages
286
Go with a 120Gb Crucial, Adata or SanDisk
But beware, a number of us have reported problems recently with SanDisk 120GB SSDs. If I was doing it again I'd look harder for a used server-grade SSD in the 60-120GB range, e.g. Intel.
 

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626
CPU: Intel Xeon E3-1270 V6
I realize is that this CPU is most likely overkill, given that the hardware guide recommends the E3-1230 V5 and similar CPUs for heavy transcode
It may depend on the input video parametrs (bitrate, resolution, FPS, originally used encoding etc) and whether the concurrent streams involve same content/target encoding, too.

EDIT: I guess the transcoding involving higher bitrate/resolution (like 4K) videos can be named "very heavy" ;) And IIRC people advise plex forums for more details...
 
Last edited:

kdragon75

Wizard
Joined
Aug 7, 2016
Messages
2,457

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
Depends on how valuable your time is or the up time of the jail.
Meh. I have my jails on my storage pool. For the 4 plugins that I run, I really don't need them to be super-fast - so I don't need an SSD for jails. I got enough space on my storage pool still.
 

Kallman

Cadet
Joined
Jun 15, 2018
Messages
8
This is a good case ,but if you are ok with rack mount chassis -- then a rack mount will usually give you more drives in the same space (24 drives in 4U) and also the hot swap capability. If rack mount is not your thing, then R6 in a great case.

While I would like to go with a rack mount chassis, I can't think of anywhere to put it where the noise wouldn't become an issue.

I would go with the -CF so I can save the money on the HBA provided the cost difference between SSM-F and SSL-CF is not more than $40-$60. Because you can get a decent HBA for about that price

The price difference is closer to $90, but the price of a HBA seems to be close to $400 here, unless I'm looking at the wrong thing. If this is right, even getting an X11SSL-CTF would be cheaper than an X11SSM-F + HBA.

That's too large. Go with a 120Gb Crucial, Adata or SanDisk. These cost about $30-$35. No need for mirroring the boot drive or the jails SSD

But beware, a number of us have reported problems recently with SanDisk 120GB SSDs. If I was doing it again I'd look harder for a used server-grade SSD in the 60-120GB range, e.g. Intel.

Would 2 of these 120GB Crucial SSDs be a good choice? (One for boot, one for jails). I could order these along with the RAM from Crucial's website.

Otherwise, what would be classified as a server-grade SSD? Any Intel SSD, or specifically the "DC" series?

Good choice. Check the NUT compatibility to be sure

I'm sorry but I have no idea what this means. Is there a resource you could point me towards?


I have a couple of other questions:
1. For the SAS Controller built in to the X11SSL-CF, would this be an appropriate cable?

2. In order to use the Kaby Lake CPU, the BIOS will probably need to be flashed. I could probably solve that by borrowing a CPU from a friend, but another option is to buy a Skylake Celeron or Pentium for flashing the BIOS. This second CPU could then be repurposed for a machine to host backups of important data, i.e. no transcoding or other fancy features. If I go this route, would a Celeron, Pentium or i3 be the best choice?
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
Corsair RM850i

Would recommend the rm850x over the i. The i features are wasted on FreeNAS, and actually complicate the psu setup!, but if necessary I guess you could connect up a Windows vm to get to them!
 

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626
anywhere to put it where the noise wouldn't become an issue.
There has been a thread about quiet/silent rackmount in these forums.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
The price difference is closer to $90, but the price of a HBA seems to be close to $400 here, unless I'm looking at the wrong thing.
I was thinking more like a IBM M1015 or a Dell H220 or Dell H310 and used they are available on eBay for around $40-$60 all day long. You might even find a seller shipping to Sweden. Are you averse to used hardware?
Don't fret too much on your boot SSD. A basic one should suffice. You need to really think about performance SSD when you plan on using them for SLOG or L2ARC or the like.
I'm sorry but I have no idea what this means. Is there a resource you could point me towards?
NUT stands for Network UPS Tools check your exact model for compatibility.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
1. For the SAS Controller built in to the X11SSL-CF, would this be an appropriate cable?
I am not familiar with that brand, but I think it should work. Someone else with experience with that cable might chime in. However, remember that if you get a rack mount chassis then it will also depends on the type of backplane in the chassis.
2. In order to use the Kaby Lake CPU, the BIOS will probably need to be flashed. I could probably solve that by borrowing a CPU from a friend, but another option is to buy a Skylake Celeron or Pentium for flashing the BIOS. This second CPU could then be repurposed for a machine to host backups of important data, i.e. no transcoding or other fancy features. If I go this route, would a Celeron, Pentium or i3 be the best choice?
Best bet is to borrow a CPU if you need it. You might get lucky and your board might be pre flashed with the latest BIOS.

If you really must have a backup FreeNAS then I would go with the Pentium.
 

Stux

MVP
Joined
Jun 2, 2016
Messages
4,419
I am not familiar with that brand, but I think it should work. Someone else with experience with that cable might chime in. However, remember that if you get a rack mount chassis then it will also depends on the type of backplane in the chassis.
.

I can’t read Swedish, but it needs to be a forward and not reverse breakout cable.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
The description says m-SAS SFF-8087 to 4x SATA ...so I assume that is a forward breakout cable and not a reverse breakout cable.
 

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626
Last edited:

Kallman

Cadet
Joined
Jun 15, 2018
Messages
8
Are you averse to used hardware?

Not really. I'm more skeptical of used hardware when it comes to ESD-sensitive components like motherboards, RAM or hard drives. A HBA would count as well. I've been taught about factory SSD precautions by someone who used to work in one, and also learned that most PC builders tend to underestimate the danger of ESD. Statements like "I've been building computers for X years now and I've never had an issue" seem to be everywhere. On the other hand, people who deal in server tech are probably more knowledgeable than most consumers.

I've decided to go ahead with the Define R6. While a proper rackmount case would be nice to have, the amount of effort it would take to quiet it down doesn't really seem justified when the Fractal case would probably be even quieter and is more than enough for my present storage needs.

As for the motherboard, going with the X11SSM-F along with and HBA is not only cheaper, but I won't actually need to buy the HBA until I decide to expand, as I will be starting out with with 2 SSDs and 6 hard drives, which will exactly fill out the 8 SATA ports on the X11SSM-F. I am however considering getting the X11SSH-F instead, as that would give me an M.2 slot instead of a PCI-e slot that I'm most likely not going to use anyway.

My remaining questions are therefore:
  1. Is there any reason to go with the X11SSM-F over the X11SSH-F, or, is there any reason why I would want the extra PCI-e slot?
  2. Does FreeNAS currently support NVMe SSDs (for booting or storage)?
  3. If not, given that AHCI PCI-e SSDs aren't really a thing anymore, would a SATA SSD in a PCI-e M.2 slot work at all?
  4. Finding a good quality 2.5" SSD at under 250 GB is difficult enough, are there any 120 GB or less M.2 SSDs from reputable brands that aren't WD Green, Kingston or other "lower quality" products?
After this I should be just about ready to start ordering and putting stuff together!
 

pro lamer

Guru
Joined
Feb 16, 2018
Messages
626
SATA SSD in a PCI-e M.2 slot work at all?
The X11SSH-F m.2 port seems to be PCI-e only (I can't download the manual now so I am basing on the product page) so you need a PCI-e version of an SSD.

EDIT: if it was PCI-e/SATA then it might reuse one of the SATA connectors (making it occupied thus the total SATA disks count would not increase)
 
Last edited:

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
Is there any reason to go with the X11SSM-F over the X11SSH-F, or, is there any reason why I would want the extra PCI-e slot?
An extra PCI slot is always better for expansion. Maybe in the future, your needs grow and you need more than 11 HDDs or possibly even more. Having that extra PCI means you can plug in an extra HBA. Other than that, you might think of using a hypervisor to handle some of the drives. So you can have 1 HBA handling FreeNAS on a separate pool. and the other PCI for a RAID card handling drives for a VM server. Possibilities is what I am getting at.
Does FreeNAS currently support NVMe SSDs (for booting or storage)?
It does.
If not, given that AHCI PCI-e SSDs aren't really a thing any more, would a SATA SSD in a PCI-e M.2 slot work at all?
I wouldn't go through all that complication which is probably going to be more expensive than simply using a Crucial MX series 120GB SSD as my boot.
Finding a good quality 2.5" SSD at under 250 GB is difficult enough, are there any 120 GB or less M.2 SSDs from reputable brands that aren't WD Green, Kingston or other "lower quality" products?
120GB SATA SSDs are easy enough to find. Also this question is moot, if you opt for the X11SSM-F for that extra PCI slot.
 

Kallman

Cadet
Joined
Jun 15, 2018
Messages
8
I wouldn't go through all that complication which is probably going to be more expensive than simply using a Crucial MX series 120GB SSD as my boot.

The only 120 GB SSD available is the BX300. I could go for that one or the Intel 545s which appears to be on sale right now at the same store where I'll be buying some other parts.
 

Inxsible

Guru
Joined
Aug 14, 2017
Messages
1,123
The only 120 GB SSD available is the BX300. I could go for that one or the Intel 545s which appears to be on sale right now at the same store where I'll be buying some other parts.
Swedish market must be different then.

Lots of options under $30 here in the US. Have you checked for AData? Those seem to hold up as good as the Crucials.
https://www.newegg.com/Product/Prod...cription=120GB+SSD&ignorear=0&N=-1&isNodeId=1

https://www.amazon.com/s/ref=nb_sb_noss?url=search-alias=aps&field-keywords=120GB+SSD
 
Top