New Attempt at TrueNAS Hardware

[SeaWolfX]

I am making a new attempt at building a Free/TrueNAS server ad could really use some input on my hardware selection. Primarily to be used for file sharing and media storage / streaming which will be shared with a couple of other family members outside my household who will only be remote users. The current HW selection is at the high end of my budget.

Functions
Files Server (Documents / Photos / Configs / Other)

• Local and remote access for upload / download.
• Automated backup to remote location.

Media Server (Movies / Music / Books)

• Media download (seedbox).
• Media streaming.

Requirements

• 24/7 availability.
• 1-2 (max 3) concurrent transcoding streams.
• No buffering when streaming on local network.
• Possible to easily expand HD pools in the future.
• RAID / Mirroring / Redundancy (for storage and system drives).
• High data integrity (for critical files documents / photos).
• Ability to control everything remotely (SSH / Web GUI services).
• Health monitoring/status and reports on hard drives.
• Configurations / applications that gets f**** does not impact the whole system and are easy to replace / revert.
• Reverse proxy in front of any web based services (i.e. re-direct each web service / container port to separate sub-domain on FQDN).
• Easy to update applications and services w/little or no downtime.

Environment

• Internet Connection: 500 / 500
• LAN: 1Gb Ethernet
• Gateway: pfSense w/pfBlockerNG and OpenVPN
• Remote Access: FQDN
• Network: IoT Devices (like Chromecast, Google Speakers) on Separate Subnet

Hardware

• Motherboard: Gigabyte X570 Aorus Pro
• CPU: AMD Ryzen 5 Socket AM4 5600X 3.7Ghz
• Memory: Crucial DDR4 2666MHz ECC Reg 32GB
• Network: HP 1Gb Quad Port PCI-E Ethernet Adapter
• System Drives: 1 x 256GB SSD SATA
• Virtual Machine Drives: 2 x NVMe M.2 SSD (Samsung 970 Pro 512GB)
• Storage Pool: 4 x 8TB SATA Drives (Seagate Exos)
• Drive Cabinets / Backplanes 3.5": StarTech 4 Bay

Can anyone confirm that the motherboard / CPU / memory combination will run ECC? Gigabyte states that they are supporting ECC memory on the board, not sure if that means that I will actually have ECC running.

 

[jayecin]

That CPU will support a max of 2 4k transcode streams at a time according to Plex recommendations. You dont have a GPU, Ryzen desktops dont have an integrated GPU so you will need one for at least the initial setup. I also wouldnt use NVMEs for virtual machine drives, you wont get any sort of performance increase over traditional SATA SSDs, you will just spend more money for less space. 500GB (assuming you are mirroring the drives) isnt a whole lot of space for virtual machines, i currently use 2 X 500GB SATA SSDs for that same purpose, its not a whole lot of space, especially once you start installing applications and storing data on them and using jails. I have 32GB of ram as well and while its enough, again its not a lot of extra room. You need to be careful allocating RAM to VMs as its not shared, it becomes dedicated. So 2GB to a windows 10 VM, means the rest of the system only has 30GB, even if the windows server only uses 500mb.

 

[Herr_Merlin]

setting up requierments for enterprise class and going for cheap desktop hardware, which is not really capable of ECC... go for used server hardware and 128+ GB of memory and much more CPU performance.

 

[ChrisRJ]

• Possible to easily expand HD pools in the future.

Given that this an area where ZFS has its limits (relative to home-use, which it was not designed for), you will need to be more specific. If your reference point is a typical NAS, you might need to make some compromises.

 

[SeaWolfX]

That CPU will support a max of 2 4k transcode streams at a time according to Plex recommendations.

Thanks for your input.

I think the CPU will still be fine. I do not anticipate more than 2 concurrent stream and most of my movies are just HD anyway :)

You dont have a GPU, Ryzen desktops dont have an integrated GPU so you will need one for at least the initial setup.

That's right, I will get a cheap GPU for the setup. I might even have one lying around.

I also wouldnt use NVMEs for virtual machine drives, you wont get any sort of performance increase over traditional SATA SSDs, you will just spend more money for less space. 500GB (assuming you are mirroring the drives) isnt a whole lot of space for virtual machines, i currently use 2 X 500GB SATA SSDs for that same purpose, its not a whole lot of space, especially once you start installing applications and storing data on them and using jails.

That is good point. The main reason I want to use NVMe is because I already have one available and though I might as well use the M.2 slot so that I have more available SATA ports.

I have 32GB of ram as well and while its enough, again its not a lot of extra room. You need to be careful allocating RAM to VMs as its not shared, it becomes dedicated. So 2GB to a windows 10 VM, means the rest of the system only has 30GB, even if the windows server only uses 500mb.

Noted. I do not think I will be running many VMs anyway. I might just spin up one now and again for some testing. I am not 100% sure about the ECC compatibility though.

 

[SeaWolfX]

setting up requierments for enterprise class and going for cheap desktop hardware, which is not really capable of ECC... go for used server hardware and 128+ GB of memory and much more CPU performance.

Would you say my requirements Enterprise class? It's just a home file / media server with some extra focus on reliability and data integrity which I hope can be achieved by ECC and ZFS. I would not call $2500 cheap, but I guess it depends on your perspective.

Are you certain that the configuration is not ECC capable?

 

[Herr_Merlin]

It is ECC "capable" but error handling and reporting to the OS and no IMPI to report to. Even Ryzen plattforms with IPMI have issues to report to it in regards of ECC errors. Thus ECC is not much of use with AMD desktop line.

There was a long thread from someone here somewhere, who did took the time and tested a ton of AMD desktop / prosumer boards for Ryzen and the way ECC error handling was implemented with those. The final result was a pure catastrophe if I recall correctly. Even tested the Ryzen "server boards" from asrock rack with not much better results. That person was even in contact with the manufactors and AMD...

If you want AMD go Epyc. TR is not an option either. Or go Intel Xeon.

 

[SeaWolfX]

Given that this an area where ZFS has its limits (relative to home-use, which it was not designed for), you will need to be more specific. If your reference point is a typical NAS, you might need to make some compromises.

I am thinking of setting the 4 x 8TB disks up in a 2 x 2 mirror vdev pool. That way, if I want to extend it I can add 2 x 8TB disks at a time. I have read many suggesting this as good setup in smaller home servers.

 

[jayecin]

Would you say my requirements Enterprise class? It's just a home file / media server with some extra focus on reliability and data integrity which I hope can be achieved by ECC and ZFS. I would not call $2500 cheap, but I guess it depends on your perspective.

Are you certain that the configuration is not ECC capable?

This community has a massive hard on for ECC without actually understanding when and why its useful. Even the creator of the ZFS file system says there is nothing inherit about it that makes it require ECC more than any other file system. You are far more likely to lose data to other sources than you are not fixing a single bit flipped in memory due to cosmic rays or massive EMF sources. ECC wont fix a bad memory sector, it wont prevent other memory errors and its not a guarantee of anything other than a single pixel in a photo from flipping from grey to white if that pixel happens to flip while the picture is loaded into memory and actively rewritten.

 

[Chris Moore]

• Motherboard: Gigabyte X570 Aorus Pro
• CPU: AMD Ryzen 5 Socket AM4 5600X 3.7Ghz

This is just asking for problems

HP 1Gb Quad Port PCI-E Ethernet Adapter

HP adapters have a history of not working well in FreeNAS.

This community has a massive hard on for ECC without actually understanding when and why its useful.

@jgreco is that what this community does? I thought ECC was a good thing and here we are being schooled about how stupid we are?

 

[jayecin]

This is just asking for problems

@jgreco is that what this community does? I thought ECC was a good thing and here we are being schooled about how stupid we are?

Dont be so hard on yourself.

 

[SeaWolfX]

This is just asking for problems

Thanks for the feedback. I won't get far without knowing why though :)

Is it the AMD platform together with FreeNAS that is the issue? If so is there an Intel alternative with the same performance in the same price range?

HP adapters have a history of not working well in FreeNAS.

Ok. It was just a nice to have anyway.

 

[Newfoundland.Republic]

If you don't already have the motherboard, you could look at replacing the with an AsRock Rack X470D4U. Cheaper too...

 

[Newfoundland.Republic]

@jgreco is that what this community does? I thought ECC was a good thing and here we are being schooled about how stupid we are?

Hmmmm... Linus Torvalds has an opinion (and, yes, he does have more than one strong opinion ) on ECC RAM. See Linus Torvalds Blasts Intel For Strangling the ECC Memory Market

 

[microserf]

This community has a massive hard on for ECC without actually understanding when and why its useful. Even the creator of the ZFS file system says there is nothing inherit about it that makes it require ECC more than any other file system. You are far more likely to lose data to other sources than you are not fixing a single bit flipped in memory due to cosmic rays or massive EMF sources. ECC wont fix a bad memory sector, it wont prevent other memory errors and its not a guarantee of anything other than a single pixel in a photo from flipping from grey to white if that pixel happens to flip while the picture is loaded into memory and actively rewritten.

I've had a few drives fail. One sudden death, the rest slowly. S.M.A.R.T. reporting caught all of the slowly failing drive indicators and replacement was a breeze. RAIDZ2 made the sudden death a relatively painless and low stress event.

I've also had an NFS share along with an iSCSI target holding boot images, encrypted backups, and block devices intermittently corrupted. Not the underlying pool but some data within. Depending on the day, logs pointed at pretty much everything as a possible source of the problem. It was a faulty UDIMM. Since then, I've had two other memory modules fail. The systems they were in reported which RDIMM was the problem. One of those systems stopped using the faulting RDIMM and put some reserved RAM into use. Both made it easy to find and replace the hardware.

 

[microserf]

I think the CPU will still be fine. I do not anticipate more than 2 concurrent stream and most of my movies are just HD anyway :)
...
That's right, I will get a cheap GPU for the setup. I might even have one lying around.

Transcoding on a storage server? Leave the CPU out of it and go big or go home: https://www.reddit.com/r/nvidia/comments/g4fq6a/3x_tesla_t4_upgrade_for_my_plex_server/

 

[Etorix]

Thanks for the feedback. I won't get far without knowing why though :)

AMD has lesser support from TrueNAS than Intel at this point.
You don't need the latest and greatest chipset and CPU to serve files, so both the X570 and the 5600X indicate that you're thinking about your server from the viewpoint of a consumer desktop.

As you want to transcode, a Xeon E or a Core i3-8xxx/9xxx (ECC capable!) with an iGPU in a server-grade motherboard (e.g. Supermicro X11SSx or X11SCx) would be a more traditional choice—and save you some money for more RAM or more drives.

 

[SeaWolfX]

AMD has lesser support from TrueNAS than Intel at this point.
You don't need the latest and greatest chipset and CPU to serve files, so both the X570 and the 5600X indicate that you're thinking about your server from the viewpoint of a consumer desktop.

As you want to transcode, a Xeon E or a Core i3-8xxx/9xxx (ECC capable!) with an iGPU in a server-grade motherboard (e.g. Supermicro X11SSx or X11SCx) would be a more traditional choice—and save you some money for more RAM or more drives.

I could go for something like this:

Supermicro X11SSH-F
Intel Xeon E3-1230v6 3.5GHz Socket 1151

The thing is it would cost me almost the same where I live (I guess maybe it's a lover volume market for server grade hw). I know the AMD HW is more desktop oriented, but besides less support from TrueNAS, what disadvantages does it have compared with Supermicron / Intel? Less reliable?

 

[SeaWolfX]

If you don't already have the motherboard, you could look at replacing the with an AsRock Rack X470D4U. Cheaper too...

Thanks for the tip! The least expensive one I found here was $50 more than the Gigabyte X570 Aorus Pro though

 

[Newfoundland.Republic]

Supermicro X11 is a good choice. I like the X11SSH-F - if I was replacing my X9 board, that one would be in the running. It gives you IPMI and Intel i210-AT NICs that work well.

Intel i3-8xxx/9xxx would be a good choice if you do not need the cores. Note that the "F" suffix means no video on the CPU.