SOLVED First FreeNAS build for: PLEX, Media storage and more

[zdesigns]

Hi I am new to the FreeNAS scene and would like to get some feedback on what I plan to purchase.

Things to take into cosideration when reading:
1. I am on kind of a budget, so there will be some compromises
2. Prices on some products are much higher in Denmark where i live (example. a e5-2680v3 costs about 2.230 USD)
3. There is not that good of a used market in Denmark and importing from US or other countries outside of EU is expensive.
4. It has to be low power cost
5. Has to be somewhat quiet

This is the build I am planning:
CPU: Intel Xeon E3-1240 V6
MB: Supermicro X11SSM-F
RAM: 1 x Samsung DDR4-2400 REG/ECC SC 16GB
Boot Drive: Intel SSD D3-S4510 240GB
Storage: 6 x WD Red NAS 4TB
PSU: Seasonic Focus Plus 550 Gold
Case: Fractal Design Define R6 or Fractal Design Node 804 (Have not decided yet)

I still need a good CPU cooler/heatsink, any recommendations?
I also see that many people buy LSI controllers, will this be benefitial for my setup/usecase?
Regarding the storage, i am currently not sure what raid to go with, so if anyone has recommendation or advice please let me know. (The storage will be expanded in the future)
If you have recommendations for other hardware, please let me know

Will the described build be able to run:
- PLEX server (2-3 streams at max, mostly not transcoding)
- Media Storage (Photos and 4K Video i shoot with my sony A7III)
- Some VM- or Docker enviroment, that will be used for small webserver and other projects.

 

[anmnz]

Seems an OK build for the described usage.

Assuming the CPU comes with a cooler, the stock one will work fine.

The motherboard has 8 SATA ports. Seems enough for what you are planning. You can upgrade to an LSI HBA later if you need more.

Your boot drive is bigger than it needs to be.

Your most obvious choice is between 6 disks in RAIDZ-2 (more space-efficient) and 3 pairs of mirrors (more performant for hosting your VMs, easier to upgrade in smaller steps later). Personally I'd go for RAIDZ-2 and add an SSD for VMs. (Per my sig.)

I think you'll want more RAM if you start making heavier use of VMs -- can be added later (the only drawback is that if you buy more RAM later you should test it which will mean downtime if you don't have another machine to test it in).

But this:

1. I am on kind of a budget, so there will be some compromises
2. Prices on some products are much higher in Denmark where i live (example. a e5-2680v3 costs about 2.230 USD)
3. There is not that good of a used market in Denmark and importing from US or other countries outside of EU is expensive.

The used market elsewhere in the EU is surely worth looking at?

Here is an E5-2680 v3 in NL, listed on ebay.de, for one tenth of that price you mentioned:
https://www.ebay.de/itm/173933157518

Here is a reasonably large seller of used gear in the UK (still in the EU...) that seems to make an effort to be competitive for European markets (they sell a lot on eBay too): https://www.bargainhardware.co.uk/

These are just a couple of random examples; there are lots more -- but the point is I think the the market for used server gear in the EU might be a lot better than you think...

 

[Constantin]

If storage is your primary intent, also consider the https://www.supermicro.com/en/products/motherboard/X10SDV-2C-7TP4F

Lots of SATA ports, 25w TDP consumption, SFP+, etc. all in one board. It’s pretty future-proof. Over here, this embedded board is about $450.

 

[zdesigns]

If storage is your primary intent, also consider the https://www.supermicro.com/en/products/motherboard/X10SDV-2C-7TP4F

Lots of SATA ports, 25w TDP consumption, SFP+, etc. all in one board. It’s pretty future-proof. Over here, this embedded board is about $450.

I don't think that the processor is powerfull enough for the other tasks

 

[zdesigns]

Assuming the CPU comes with a cooler, the stock one will work fine.

The CPU i am currently looking to buy, does not come with a cooler... Any suggestions?

Your boot drive is bigger than it needs to be.

I think this was one of the cheapest SSDs from a decent brand (it is also compatible with the MB, if you check their compatibility list)

Your most obvious choice is between 6 disks in RAIDZ-2 (more space-efficient) and 3 pairs of mirrors (more performant for hosting your VMs, easier to upgrade in smaller steps later). Personally I'd go for RAIDZ-2 and add an SSD for VMs. (Per my sig.)

An option is that i use the 240GB SSD for VMs and then get another SSD for the boot drive

I think you'll want more RAM if you start making heavier use of VMs -- can be added later (the only drawback is that if you buy more RAM later you should test it which will mean downtime if you don't have another machine to test it in).

Depending on the price, i just get the 32GB ram from the beginning. Is the RAM speed fine or should i get faster RAM?

But this:

The used market elsewhere in the EU is surely worth looking at?

Here is an E5-2680 v3 in NL, listed on ebay.de, for one tenth of that price you mentioned:
https://www.ebay.de/itm/173933157518

Here is a reasonably large seller of used gear in the UK (still in the EU...) that seems to make an effort to be competitive for European markets (they sell a lot on eBay too): https://www.bargainhardware.co.uk/

These are just a couple of random examples; there are lots more -- but the point is I think the the market for used server gear in the EU might be a lot better than you think...

Buying used, is only interesting to me if there is a significant price cut. The E5-2680 v3 is only a bit more expensive than the e3-1240 v6, but I would need another MB that is more expensive as well, so not sure about it. I would go from about 10k to 18k in passmark, but the power consumption and tdp would mean that it cost more to run and that I would need a better cooler to keep down temps and that might add noise. At least that what I gather from it. Also when buying used, there is no warranty and what guarantee is there that i get the product and its intact? how many hours has it been run for and so on.

 

[Constantin]

I grant you that the transcoding might be an issue. I can’t speak to VM use either.

That said, I get the impression from my reading here that VMs greatly benefit from a dedicated SSD storage pool, not great CPU capacities. But I’ll let others opine on that aspect.

I’m a firm believer in transcoding locally, not at the server level. The op needs to figure out what goal is more important to him/her: transcoding at the server level or energy efficiency.

For my application, local transcoding is a better approach simply because the (substantial) power needs are limited to the time when someone is actually watching.

I’d rather invest in a long term server and then upgrade the “head units” as needs change and better hardware comes along. Usually, head units benefit from application-optimized hardware, ie custom chips that do a better job at lower cost than general-purpose CPUs.

To some extent it also depends on the network protocol that the OP wants to use. For example, SMB is a single-thread process and the faster turbo mode on the D-1508 vs. the D-1537 in my rig could theoretically result in a faster throughput from the -2c-

 

[xaintly]

Hi there, I built almost this exact system last week!
I have the Node 804, 8x WD Red 10tb drives, Seasonic FOCUS Plus SSR-750FX and 32 GB of ECC NEMIX RAM.
My boot drive is a Sandisk Cruzer 32gb flash drive plugged into the USB slot on the motherboard (internal).
The rest is the same as your proposed build.

I installed PLEX and it worked great to stream to my TV and mobile devices.
I'm only transcoding to 1080 though, I don't have any 4k devices.
The Xeon E3-1240 v6 has a Passmark score of 10458 and a great cache.
I have seen memory spike up to 20gb in use sometimes, so if you plan to PLEX I'd suggest at least 32gb of ram.
I didn't think the motherboard supported ram faster than 2400.

With 8 drives in the 804, drive temperatures were going above 42 degrees with just the single fan the Node 804 comes with for the drive compartment. The drive compartment fan is an exhaust fan and doesn't blow across the drives (you can flip it around, but there's no filter there so you'll be pulling in dirt as well).
I bought a Noctua NF-F12 iPPC-3000 fan to supplement the drive cooling and that worked great. I added it to the drive bay opposite the exhaust fan as an intake fan that brings in air and blows across the drives.
I had to adjust the speed manually though; the Noctua Fan has a 4-pin connector and the variable speed case fan controller in the 804 has annoying plugs that only accept 3-pin connectors. Plug the Noctua into the 'FANA' connector on the X11SSM board and use IPMI to set the fan speed to 40%.
If you set the fan at 100% it will be very noisy, at 40% + the fan that came with the 804, all drives are in the 34-38 degree range (some drop down to 32 sometimes, but I don't see any going over 40 any more).

My boxed CPU from Newegg came with a cooler, but I replaced it with this one: Noctua NH-L9i low-profile CPU cooler. The low profile wasn't necessary, there is plenty of room in the 804.

Here's my google docs from when I was comparing prices of enclosures, drives, etc in case it helps you:
https://docs.google.com/spreadsheets/d/1y9c8EKRQlVcJD_KFiNw73oPGnJRDxZfwSCsIvagQxrA/edit?usp=sharing

 

[Constantin]

The case and its fans make a big difference re: drive temperature. I have a dedicated 5.25” bay based fan controller for the hard drives. They basically stay at 31*C normally, maybe 33*C under scrub conditions. But I also have three Noctua 120mm fans dedicated to the drives alone.

Keeping the CPU and especially the HBA Chip cool was a lot harder. They never exceed 50*C now, but only because I put dedicated blowers on them plus additional blowers that sweep the rest of the motherboard to keep the L2ARC, RAM, and the SLOG cool.

Xaintly, what is your power consumption during boot, idle, and normal file serving?

 

[zdesigns]

My boot drive is a Sandisk Cruzer 32gb flash drive plugged into the USB slot on the motherboard (internal).
The rest is the same as your proposed build.

I see a lot of people saying that they just use an USB for boot device. Is that for cost saving or just because its waste to do anything else?

I installed PLEX and it worked great to stream to my TV and mobile devices.
I'm only transcoding to 1080 though, I don't have any 4k devices.

As i understand it, transcoding is only happening, if you dont play it as is with Direct Play or Direct Stream.
My Media files are also mostly in H.264 .MP4 and .MKV files. In some occasions i will

I have seen memory spike up to 20gb in use sometimes, so if you plan to PLEX I'd suggest at least 32gb of ram.

You mention 20gb in use at some times, what is all that ram being used for?

I didn't think the motherboard supported ram faster than 2400.

Its a bit weird, because on the product page, the max speed mentioned is 2400, but if you check for compatible memory, it does say it supports 2666 speed as well or it might just be some memory that is capable of 2666 but runs at a lower speed.

With 8 drives in the 804, drive temperatures were going above 42 degrees with just the single fan the Node 804 comes with for the drive compartment. The drive compartment fan is an exhaust fan and doesn't blow across the drives (you can flip it around, but there's no filter there so you'll be pulling in dirt as well).
I bought a Noctua NF-F12 iPPC-3000 fan to supplement the drive cooling and that worked great. I added it to the drive bay opposite the exhaust fan as an intake fan that brings in air and blows across the drives.
I had to adjust the speed manually though; the Noctua Fan has a 4-pin connector and the variable speed case fan controller in the 804 has annoying plugs that only accept 3-pin connectors. Plug the Noctua into the 'FANA' connector on the X11SSM board and use IPMI to set the fan speed to 40%.
If you set the fan at 100% it will be very noisy, at 40% + the fan that came with the 804, all drives are in the 34-38 degree range (some drop down to 32 sometimes, but I don't see any going over 40 any more).

I will keep that in mind, but currently I think that I will go with the larger case, since it will be more future proof, regarding expansions. That should help some with the HDD temp.

My boxed CPU from Newegg came with a cooler, but I replaced it with this one: Noctua NH-L9i low-profile CPU cooler. The low profile wasn't necessary, there is plenty of room in the 804.

I will check out the Noctua cooler and look at some reviews


Also, what power consumption do you have on your build? Boot, Idle and heavy use

 

[xaintly]

Xaintly, what is your power consumption during boot, idle, and normal file serving?

That's a good question! I'm not sure how to tell. The IPMI sensors report voltage but not watts. Do you plug your device into something like a 'Kill a Watt' to find out?

I see a lot of people saying that they just use an USB for boot device. Is that for cost saving or just because its waste to do anything else?

In this case, I used all 8 of the SATA connectors on the board & PSU connectors and I didn't want to install another drive controller, & molex splitter cables. The motherboard has an internal USB 3.0 slot you can plug flash drives into. From reading the forums, it didn't sound like FreeNAS makes much use of storage beyond 16gb.

As i understand it, transcoding is only happening, if you don't play it as is with Direct Play or Direct Stream.

I wasn't sure how much I'd actually be transcoding. PLEX recommends a passmark score of 12-17k for doing 4k transcoding (more if your server is doing other stuff). A passmark score of 2000 is recommended for 1080p, so I chose a processor with a good price point above 10k.

I can't speak to other people's results, however from some other forums, the Atom C3858 worked just fine with PLEX, and the Synology Diskstations supposedly run PLEX ok with the Atom C3538 / Celeron J3455, which have passmark scores barely above 2000. The Supermicro A2SDI-12C-HLN4F motherboard has 12 SATA3 connectors and a passmark of 4852. I was really tempted by that one.

You mention 20gb in use at some times, what is all that ram being used for?

I only noticed it went up to 20gb when I zoomed out on the graph and went back over the history of a few days. I'll have to watch it closely to see what makes it spike. I was purposefully trying to push the system (copying lots of files onto it while streaming PLEX). I don't have de-duplication turned on.

 

[Constantin]

Hi Xaintly,

Yes, I used a kill-a-Watt. I know it’s not as accurate as the yokogawa’s at work but it usually does a pretty good job. I simply took a video during boot to capture the maximum power draw (HDDs spinning up) as well as power draw during ops and idle.

While some folk have used USB sticks for a while, the forum is full of stories re USB stick corruption and so on. If you go the USB route, at the very least go for a mirrored boot drive. Given a choice, I’d always go SSD or SATADOM. My current system has a mirrored 64GB SATADOM set so I don’t have to worry about my boot volumes filling too much between system upgrades.

Regarding passmark, the D-1508 has a passmark of about 3800. My D-1537 is allegedly twice as powerful (~7700) but the D-1508 runs at 2.2GHz vs. 1.7GHz for the D-1537. Maximum theoretical performance for protocols like SMB is hence likely to be better on the D-1508.

 

[zdesigns]

While some folk have used USB sticks for a while, the forum is full of stories re USB stick corruption and so on. If you go the USB route, at the very least go for a mirrored boot drive. Given a choice, I’d always go SSD or SATADOM. My current system has a mirrored 64GB SATADOM set so I don’t have to worry about my boot volumes filling too much between system upgrades.

What is the advantage of SATADOMs over SSDs?

 

[Constantin]

SATADOMs plug directly into Supermicro and similar server motherboards and get data and power through a non-Standard SATA port (no other cabling needed.) If your motherboard only has regular SATA ports, then a small IDE power plug adapter can be used to get 5V to the SATADOM.

“Traditional” SATA-interfaced SSDs need to be fed data and power separately but are also a lot less expensive. SATADOMs are a very compact solution and you pay for the form factor and alleged server-grade durability accordingly.

Thus, either of the above options will “eat” a SATA port for each SSD or SATADOM you attach.

Another SSD option is mSATA or m.2-based sticks. They will use either an available port on your motherboard for that purpose or can be installed on applicable PCIe “riser” cards. These interfaces provide both power and data. Faster SSD interfaces like m.2 PCIe 3.0x4 should be reserved for SLOG use, should you contemplate adding home. (I use a mSATA Interface on my motherboard for L2ARC, the m.2 for the SLOG).

Any of these interfaces is fast enough (including USB) to keep the OS happy. However, based on my reading here, SSDs and SATADOMs typically last a lot longer than USB sticks. Mirroring is a good idea also, especially if you’re running an encrypted boot pool.

 

[Vivan]

I would actually suggest getting a CPU cooler. The stock one isn't able to handle it when Plex decides to take up 100% of CPU for an extended period of time during a transcode. I have a similar build and get the motherboard beep warning occasionally because the stock cooler is not good enough. It may also just be that the automatic fan control on the supermicro board isn't good enough but I haven't managed to really find a better controller.

 

[xaintly]

Yes, I used a kill-a-Watt. I know it’s not as accurate as the yokogawa’s at work but it usually does a pretty good job. I simply took a video during boot to capture the maximum power draw (HDDs spinning up) as well as power draw during ops and idle.

They seem useful to have, I bought one and I'll report the wattage of this build after it comes in.

Thanks for the advice on USB stick corruption! I took a backup of the config, and I'll order a PCI-E SATA card & some SATADOMs soon to use as the boot device instead.

 

[xaintly]

I recorded video of boot + 3 minutes, and then another minute while transcoding with PLEX.
Powered off: 3W
While booting: 80W-190W
Normal operation (transcoding didn't change the wattage): 70W (spikes up to 90W sometimes)

 

[Constantin]

That’s awesome!

 

[zdesigns]

I recorded video of boot + 3 minutes, and then another minute while transcoding with PLEX.
Powered off: 3W
While booting: 80W-190W
Normal operation (transcoding didn't change the wattage): 70W (spikes up to 90W sometimes)

Seems to be a good choice to go with the E3-1240V6 :)

 

[adrianwi]

Hardware choices look good. Not sure about your choice of Media Player :p

I'd have a look at emby :)

 

[zdesigns]

Hardware choices look good. Not sure about your choice of Media Player :p

I'd have a look at emby :)

What are some of the reasons to go with emby over plex ?