RAM requirements for 1M files

[kenster]

the FreeNAS documentation recommends 1GB / TB of disk space. I'm working to spec out a FreeNAS build. The current NTFS file system stores 1M files across 7TB.

How do I calculate RAM requirements for 1M files using 7TB of disk space with dedupe on?

 

[Chris Moore]

Deduplication uses a lot of system resources. You should plan for around 20GB of system memory for each TB of pool data.

Have you used ZFS / FreeNAS before or is this the first you are looking into this?

 

[Chris Moore]

the FreeNAS documentation recommends 1GB / TB of disk space. I'm working to spec out a FreeNAS build. The current NTFS file system stores 1M files across 7TB.

How do I calculate RAM requirements for 1M files using 7TB of disk space with dedupe on?

PS. How much storage are you looking to impleent? Just doing 7TB doesn't appear to be very much.

 

[kenster]

We're currently using 7 of 10TB on a Synology and I'd like to replace it with Freenas / ZFS because of the snapshotting.
It's my first build, so sorting weather we should sue deduplication.

We're coming from a file system that is 7TB 1M+ files and growing. I'd like to build 30 t- 40 TB of space at least to start with room to expand down the road by adding DAS.

 

[Chris Moore]

For that much data, you would need a lot of RAM look at this article, even though it is old, the reasoning is sound:

ZFS: To Dedupe or Not to Dedupe…

…that is the question. Ever since the introduction of deduplication into ZFS, users have been divided into two camps: One side enthusiastically adopted deduplication as a way to save storage space, while the other remained skeptical, pointing out that dedupe has a cost, and that it may not be...

constantin.glez.de

I would suggest you do some reading about FreeNAS and ZFS. Here are some intersting items to look at...

The 'Hidden' Cost of Using ZFS for Your Home NAS
https://louwrentius.com/the-hidden-cost-of-using-zfs-for-your-home-nas.html

Slideshow explaining VDev, zpool, ZIL and L2ARC
https://www.ixsystems.com/community...ning-vdev-zpool-zil-and-l2arc-for-noobs.7775/

Overview of ZFS Pools in FreeNAS from the iXsystems blog:
https://www.ixsystems.com/blog/zfs-pools-in-freenas/

Terminology and Abbreviations Primer
https://www.ixsystems.com/community/threads/terminology-and-abbreviations-primer.28174/

Why not to use RAID-5 or RAIDz1
https://www.zdnet.com/article/why-raid-5-stops-working-in-2009/

FreeNAS® Quick Hardware Guide
https://www.ixsystems.com/community/resources/freenas®-quick-hardware-guide.7/

Hardware Recommendations Guide Rev. 1e) 2017-05-06
https://www.ixsystems.com/community/resources/hardware-recommendations-guide.12/

Hardware Recommendations by @cyberjock - from 26 Aug 2014 - and still valid
https://www.ixsystems.com/community/threads/hardware-recommendations-read-this-first.23069/

 

[kenster]

awesome, thanks I'll dive into that