Build with i3 and 64GB ram limitation?

[jquib]

Hello community,

Currently I use a Qnap 6 drive array in RAID6. It's largest job is as a Plex Media server for 1080p and 4K rips. Unfortunately, after 4 yrs and 25TBs it is full and rather than buying another Qnap I've gone back to seriously considering Truenas as I feel ZFS is superior to Ext4.

I already have the hard drives purchased. I have 8 14TB WD ultrastar DC HC350s....model # WUH7214ALE6L4. Honestly when I purchased them my mind was on a 8 bay Qnap device but I really would like something with ZFS. I need a minute financially so that is giving me time to reconsider the rest of the hardware.

I'm thinking about the Supermicro X11SCH-F and an Intel Core i3-9300. I would like to use the 8 drives for a RAIDZ2 (I believe that is ZFS talk for RAID6?) Those drives in a RAID6 would yield 84TB usable space. -- I assume RAIDZ2 would be similar yield
My question is this: If I follow the guideline of 1GB RAM for every 1TB storage space that would be 84GB of RAM but in this build I would do 64GB of ECC RAM because the intel Core i3 can only use a max of 64BG RAM according to ark.intel.com

How does the community feel about an 80TB-ish RAIDZ2 array with only 64GB of RAM?

I appreciate the feedback.
-jason

 

[jgreco]

The memory thing is fine. Not fantastic, but fine. There used to be tremendous problems with people desperately trying to build 40TB filers with 8GB of RAM. That won't work, and could even be hazardous to your pool, even today. However, as you get out there in RAM, it is more forgiving, and you primarily pay a performance tax. If you didn't care about performance too much, 64GB should easily run 100TB or maybe even 150-200TB of space, but it wouldn't be as zippy as a properly sized filer. Note that overhead for jails or VM's reduces the amount of RAM available to the ARC, and when we say "64GB" we mean "around 64GB that the ARC and the base system (with its several GB of overhead) require", not "64GB of RAM occupied by a 20GB Plex VM and a bunch of other VM's eating 30GB more, leaving only 14GB for the ARC/base system".

I'm the person who initially established the 8GB minimum RAM requirement for FreeNAS due to system disasters, and I've been carefully keeping an eye on this for many years. In my opinion, you can be comfortable that I have not seen problems with such a slight mismatch as what you suggest. It would be slightly dicier with only 32GB of RAM, and I would strongly advise against 16GB RAM. Slight bending is no problem.

RAIDZ2 is not RAID6; RAIDZ2 is a data storage method that allows a loss of redundancy that is similar to RAID6 but operates in a very different manner. We are very particular about the terminology because it is not an array-level redundancy strategy, but rather a per-block redundancy strategy. RAIDZ2 also does not give a guaranteed amount of space, because the amount of space actually used for parity is variable, but this is mostly a concern for small data blocks and metadata. If you're storing large files, you get "approximately" the numbers you are expecting.

 

[mjt5282]

i run a TN core server that is primarily a Plex Media server, with 2x 8x8 TiB and 64 GB of ram. My primary suggestion is that your media datasets are set to use recordsize=1M . I think you'll be pleased with the performance, as long as your plex client can direct play your MKV files. Lack of support for Plex GPU Transcoding is the one gotcha for TN Core, IMHO. You are spending a lot of time thinking about your pool layout and less time on your CPU requirements, which would be admittedly small if you can direct play through your plex client. ZFS many years ago greatly exceeded my expectations for a media server (coming from underpowered ReadyNas 4 bay), I think you'll be pleased.

 

[Etorix]

8*14 TB in Z2 would give 6*14 TB of raw space, minus some ZFS overhead for metadata and keeping in mind that you should not fill more than say 80% of that, and preferably less. So you're looking at filling about 60 TB of data before having to consider an upgrade.
As you start with 25 TB and good components, you're in the clear. Do think about a backup though.

If the 64 GB limit of a Core i3 is of concern, the alternative would be to look for a refurbished Xeon E5 server (Supermicro X9) and lots of now cheap DDR3 RDIMM.

 

[jquib]

i run a TN core server that is primarily a Plex Media server, with 2x 8x8 TiB and 64 GB of ram. My primary suggestion is that your media datasets are set to use recordsize=1M . I think you'll be pleased with the performance, as long as your plex client can direct play your MKV files. Lack of support for Plex GPU Transcoding is the one gotcha for TN Core, IMHO. You are spending a lot of time thinking about your pool layout and less time on your CPU requirements, which would be admittedly small if you can direct play through your plex client. ZFS many years ago greatly exceeded my expectations for a media server (coming from underpowered ReadyNas 4 bay), I think you'll be pleased.

Thanks for the feedback.
My primary Plex client is an Nvidia Shield Pro so yes it can direct play. I focus on 1080p and 4K MKVs with all of the HDR and surround sound (Atmos and DTS:X) intact. So high bitrate data, and IMO the Nvidia Shield is a must. Plus the Shield is the only Client I know of that can support Dolby Vision and Atmos. The build will have transcode duty when a Roku, phone, Chromebook, etc,...is asking for a file. My current Qnap handles that just fine and even the i3-9300 I propose is more powerful than the i5 that is in my current Qnap. A few friends access my 1080p files and are allowed to stream but I don't let them go for my 4K rips as my 50 Mbps upload speed bottlenecks that beyond usability. Tough thing of course in laying out a build is trying to account for how it might be used in years to come and how might requirements change.

My primary attraction to ZFS is long term data integrity. I've spent a lot of time and money collecting 4K movies,(not to mention family photos) and I'd like to get rid of the original disks. It would be heartbreaking if 5 yrs from now I went to play a movie and the data was corrupted.

 

[jquib]

The memory thing is fine. Not fantastic, but fine. There used to be tremendous problems with people desperately trying to build 40TB filers with 8GB of RAM. That won't work, and could even be hazardous to your pool, even today. However, as you get out there in RAM, it is more forgiving, and you primarily pay a performance tax. If you didn't care about performance too much, 64GB should easily run 100TB or maybe even 150-200TB of space, but it wouldn't be as zippy as a properly sized filer. Note that overhead for jails or VM's reduces the amount of RAM available to the ARC, and when we say "64GB" we mean "around 64GB that the ARC and the base system (with its several GB of overhead) require", not "64GB of RAM occupied by a 20GB Plex VM and a bunch of other VM's eating 30GB more, leaving only 14GB for the ARC/base system".

I'm the person who initially established the 8GB minimum RAM requirement for FreeNAS due to system disasters, and I've been carefully keeping an eye on this for many years. In my opinion, you can be comfortable that I have not seen problems with such a slight mismatch as what you suggest. It would be slightly dicier with only 32GB of RAM, and I would strongly advise against 16GB RAM. Slight bending is no problem.

RAIDZ2 is not RAID6; RAIDZ2 is a data storage method that allows a loss of redundancy that is similar to RAID6 but operates in a very different manner. We are very particular about the terminology because it is not an array-level redundancy strategy, but rather a per-block redundancy strategy. RAIDZ2 also does not give a guaranteed amount of space, because the amount of space actually used for parity is variable, but this is mostly a concern for small data blocks and metadata. If you're storing large files, you get "approximately" the numbers you are expecting.

Thanks for the advice.
It looks like the same motherboard I have my eye on (Supermicro X11SCH-F) works with Xeon E-2100 and E-2200 family of CPUs. I could go for the Xeon E-2234 CPU. The E-2234 supports 128GB of RAM. By my math those two changes would cost about $700 more.
I'm not crazy about spending that much more money but it isn't out of the question. My intention of any build I undertake is long term performance and reliability and obviously if making the effort for this sort of thing, that applies. It's just that I'm typically building Windows PCs so I'm definitely out of my element here.

One of the things that pushed me to Qnap years ago and away from FreeNas(TrueNas) was that I appreciate there are things about ZFS I don't fully understand and I feared making some sort of irrevocable, catastrophic error. Anyway, thanks again for input!

 

[Etorix]

My primary attraction to ZFS is long term data integrity. I've spent a lot of time and money collecting 4K movies,(not to mention family photos) and I'd like to get rid of the original disks. It would be heartbreaking if 5 yrs from now I went to play a movie and the data was corrupted.

DANGER HERE! ZFS guards against corruption but is not a backup, and you still need at least one external backup, preferably off-site and/or off-line to guard against loss of data.

Going for a Xeon E-2000, and the higher pricing, is mostly worth it if you need more cores and go higher in the stack. Otherwise, the extra $700 could buy you a refurbished server…

 

[jquib]

DANGER HERE! ZFS guards against corruption but is not a backup, and you still need at least one external backup, preferably off-site and/or off-line to guard against loss of data.

Going for a Xeon E-2000, and the higher pricing, is mostly worth it if you need more cores and go higher in the stack. Otherwise, the extra $700 could buy you a refurbished server…

Thanks, and I get that ZFS is not a backup. I was thinking I might repurpose my current 6 bay Qnap as backup storage.

 

[mjt5282]

you can use cold storage ZFS drives and replicate data to them via rsync or syncoid . Another offsite option is to create a jail and install rclone, uploading critical non-replaceable data to s3 cloud storage such as B2 or Wasabi or Amazon AWS. I guess TN has rclone built in these days. I use it through a jail.

 

[NugentS]

Which QNAP do you have? Its possible to run TNC on some QNAPS (I do - and use it as a destination for replicated snapshots only - See QNAS in my sig)
Given you live in the US - get a second hand Xeon off Ebay, preferably from a system dismantler rather than a random person who has (may have) abused the chip. You can get an awful lot of Xeon (with more PCIe Lanes and more memory support) for not a lot of money

 

[jquib]

Which QNAP do you have? Its possible to run TNC on some QNAPS (I do - and use it as a destination for replicated snapshots only - See QNAS in my sig)
Given you live in the US - get a second hand Xeon off Ebay, preferably from a system dismantler rather than a random person who has (may have) abused the chip. You can get an awful lot of Xeon (with more PCIe Lanes and more memory support) for not a lot of money

I have the TVS-671
It has a i5-4590S and 8GB non-ecc ram. Not the kind of setup I would think you'd want to run TNC on but I can consider options.

 

[NugentS]

Hmmm, that's got an HDMI on the back, and an Intel CPU. Based on my limited testing that would probably work.
You will need to plug in a keyboard (mouse not required) and a monitor
I would also suggest opening up the box and removing the eUSB that the QNAP is (probably) booting from (that way you can add it again later and return the QNAP to being a QNAP. Its also probably not big enough for TN if its like mine)

Use a USB flash drive as a installer media
Use a USB flash drive for the initial TN Boot Pool - if it works then switch out later for a USB to M.2 adapter

And Try it. If it works consider doubling the memory to 16GB (or better if it can support more)

I personally wouldn't use this as a main NAS. But I would (and do) use this as a snapshot replication target. People do use lesser powered devices as a main NAS though running TN

You will need to boot into BIOS and change the boot device to your USB devices and then just install TN. Note that the device display won't work afterwards (mine just says booting) (but there are possible solutions for that (I haven't tried though) - assuming that TN works at all)

 

[NugentS]

I am assuming you have a backup of the data elsewhere as doing the above is going to mean formatting the disks obviously