NAS Hardware Help -- 60TB

[StoreMore]

I have been reading about FreeNAS for quite some time now and am finally ready to start building my new NAS. I previously used another off the shelf device so all data will be moved from this device once its setup, tested, burned-in. My existing 5 bay NAS is almost filled at about 8TB.

I would love some oversight from more experience FreeNAS users to make sure this setup makes sense. I have no specific budget but I do want to be cost efficient. I do value long term serviceability, reliability, ease of use, low noise.

USES & GOALS:
1. High performance 10G file storage for all existing data and future expansion (A few desktops / laptops & several streamers)

2. Some lite VM/Jail use (maybe 1-2, not needing to be on all the time)

3. Access remotely (owncloud or similar)

4. Additional Online Backblaze B2 for all critical data

5. Enabling snapshots for added restore functionality

6. Plex (??? - See #1 under questions)


PROPOSED HARDWARE:
Case - NSC-810A $220 (like the shape /form factor, similar size to off the shelf smaller nases, also considered the storinator av15 but its a bit pricey and would be the largest i want to go, also considered the Fractal 804 Node, anything in the realm of these size wise would definitely be a possibility)

PSU - 1U Flex Power Supply 350W $90 (made for case)

Boot Drives - Samsung MUF-32AB/AM FIT Plus 32GB USB Flash - $10 X 2 (planning on running 2 mirrored)

Storage Drives - 8 x 10TB White/Red drives $180 X 10 (purchased, ready to be shucked, plan on raidz2)

Network Adapter - NetApp/Chelsio 110-1114-30 111-00603+A0 10GbE SPF+ Dual-Port PCI-E Network Adapter $40 (purchased)

Cache Drive - Intel 900P 280GB 3D XPoint SSD AIC $270 (already have in my possession)

Motherboard - Supermicro X11SCH-LN4F Motherboard $300

CPU - Intel Xeon E-2126G 6 Core $300 (supports quicksync)

RAM - 128GB (4X32GB) Samsung DDR4-2666 PC4-21300 Registered ECC Server Memory $1,000 (ebay used)


QUESTIONS:
1. Definitely thought this whole time I wanted to run plex off the main unit here but earlier today almost decided against it since I can take my existing older htpc (i5-3570 - not used at tv anymore) and turn it into a plex box with a Quadro P2000 or similar that can hardware transcode. Based on what I have read in the threads it looks like a setup like this should easily exceed my new of maybe a 1-3 streams at anytime

2. Would using the Intel 900P SSD as a read/write cache improve performance over just having plenty of memory?

3. The X11SCH-LN4F supports dual M.2 drives. Would it make sense to have a pool of mirrored 970 EVOs for standard file access or making this a storage pool for regularly accessed documents? Some other config that makes more sense?

4. Based on my expected use and goals does this CPU/Motherboard/RAM/Case combo make sense? If theres an older generation cpu/mobo combo that uses ddr3 based ram i'd be happy to go to something like that as long as its powerful enough and supports the neccesary amount of the ram. The case chosen above supports a max of micro-atx.

5. Do I actually need that much ram (128GB) for the system right now or will 64GB be sufficient until I actually use more of the storage space?

Anything else I should consider? Thanks so much in advance!

 

[Chris Moore]

Definitely thought this whole time I wanted to run plex off the main unit

Your proposed hardware is more than adequate to run Plex, so there is no reason to run a second system for Plex.

2. Would using the Intel 900P SSD as a read/write cache improve performance over just having plenty of memory?

For your described use plan, it might help, but only if you have some files that are being read repetitively. Here is a video that discusses using a cache drive:
https://www.youtube.com/watch?v=oDbGj4YJXDw&t

3. The X11SCH-LN4F supports dual M.2 drives. Would it make sense to have a pool of mirrored 970 EVOs for standard file access or making this a storage pool for regularly accessed documents?

No. I don't see how that would be worth the trouble.

4. Based on my expected use and goals does this CPU/Motherboard/RAM/Case combo make sense?

You should have about 50TB of usable storage with those drives in RAIDz2, and you said that you had 8TB of data, so you should have at least 40TB of free-space and the compute capacity of the system should be more than needed for your proposed use. CPU mark of 11869 and Plex only calls for 2000 per each 1080p stream.
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E-2126G+@+3.30GHz&id=3360

https://support.plex.tv/articles/201774043-what-kind-of-cpu-do-i-need-for-my-server/

5. Do I actually need that much ram (128GB) for the system right now or will 64GB be sufficient until I actually use more of the storage space?

What makes you think you need that much RAM? The 'rule of thumb' about a GB per TB was to keep people from trying to run a system with 4GB of memory. Once you are over 16GB of RAM, you are probably good as long as you are only doing file sharing, but having the extra memory will make performance a bit better, particularly with running some virtualization, which you said you wanted to be able to do.

Anything else I should consider? Thanks so much in advance!

There is some really good reading in the "Useful Links" button in my signature. Please avail yourself. There are some scripts in there that are particularly useful.

 

[Chris Moore]

PS: I wouldn't bother with 4k transcoding, here is why:
https://www.cnet.com/news/why-ultra-hd-4k-tvs-are-still-stupid/

 

[StoreMore]

Your proposed hardware is more than adequate to run Plex, so there is no reason to run a second system for Plex.

That will be nice to consolidate it all into one!

For your described use plan, it might help, but only if you have some files that are being read repetitively. Here is a video that discusses using a cache drive:
https://www.youtube.com/watch?v=oDbGj4YJXDw&t

Based on your video and the other few I watched I'll just plan to use the spinning disks and system memory for now. The work load I have is not views of the same data so the L2ARC cache isn't relevant. I might end up doing something the the ZIL/SLOG but I think I am fine for now.

You should have about 50TB of usable storage with those drives in RAIDz2, and you said that you had 8TB of data, so you should have at least 40TB of free-space and the compute capacity of the system should be more than needed for your proposed use. CPU mark of 11869 and Plex only calls for 2000 per each 1080p stream.
https://www.cpubenchmark.net/cpu.php?cpu=Intel+Xeon+E-2126G+@+3.30GHz&id=3360

https://support.plex.tv/articles/201774043-what-kind-of-cpu-do-i-need-for-my-server/

I should be all set just need to figure out how to tell the system not to try and transcode if I click on a 4K title. I wonder if there's a way for plex to store multiple versions of the same movie (4K + 1080P) of each so if it transcodes it just uses the 1080P version?

What makes you think you need that much RAM? The 'rule of thumb' about a GB per TB was to keep people from trying to run a system with 4GB of memory. Once you are over 16GB of RAM, you are probably good as long as you are only doing file sharing, but having the extra memory will make performance a bit better, particularly with running some virtualization, which you said you wanted to be able to do.

I was just using the 1GB RAM per 1TB rule. So, in this case I'll probably drop down to 32GB or so and will add more later if necessary.

There is some really good reading in the "Useful Links" button in my signature. Please avail yourself. There are some scripts in there that are particularly useful.

Lots of goodies in there!

Thanks for all the help

 

[Chris Moore]

I wonder if there's a way for plex to store multiple versions of the same movie (4K + 1080P) of each so if it transcodes it just uses the 1080P version?

Yes, you just have to file them correctly. This post on their site gives a brief description:

https://support.plex.tv/articles/200381043-multi-version-movies/

 

[Chris Moore]

Storage Drives - 8 x 10TB White/Red drives $180 X 10

https://wintelguy.com/zfs-calc.pl

I thought you might want to look, if you have not already:

The minimum free space line relates to the best practice of not filling the pool above 80% due to the copy on write nature of the file system.

 

[StoreMore]

https://wintelguy.com/zfs-calc.pl

I thought you might want to look, if you have not already:

View attachment 28084

The minimum free space line relates to the best practice of not filling the pool above 80% due to the copy on write nature of the file system.

That's pretty close to where I thought I'd be, thanks!

Also, noted the plex article for saving to different formats of the same film.

Thanks again!