How much storage 64GB RAM Supports?

[mzarrugh]

I did my research, I know the rule of thumb (1GB RAM per TB storage). But I also looked at this:

http://www.freenas.org/blog/a-compl...are-design-part-i-purpose-and-best-practices/

8GB of RAM will get you through the 24TB range. Beyond that 16GB is a safer minimum, and once you get past 100TB of storage, 32GB is recommended. However, that’s just to satisfy the stability side of things

I'm building a system with 12 drive bays. Right now, I will put six 10TB drives in it in RAID-Z2. In the future, I will add another 6 drives and make another RAID-Z2.

I'm not sure if an FCLGA1151 motherboard, which accepts maximum of 64GB RAM, is enough or do I need to look at LGA2011?

How much raw storage can I go with 64GB of RAM?

 

[tfran1990]

1Gig of ram per TB of storage is recommended for max performance.

 

[mzarrugh]

1Gig of ram per TB of storage is recommended for max performance.

I know, but that's a soft rule though. I can get 2TB SSD L2ARC for less than the cost of 64GB RAM more.

 

[tfran1990]

This may help you answer your question.
https://forums.freenas.org/index.php?threads/l2arc-or-enough-memory.42708/#post-278149

 

[jgreco]

I know, but that's a soft rule though. I can get 2TB SSD L2ARC for less than the cost of 64GB RAM more.

Yes, but you need a *lot* of RAM to support that much L2ARC. The usual guidance is about 5:1 L2ARC:ARC. L2ARC is not a substitute for RAM. L2ARC basically works with *more* RAM in order to greatly expand caching capabilities. L2ARC should probably be around 256GB SSD for 64GB RAM.

If you don't care about performance, you should be able to do ~100-120TB on 64GB of RAM. This assumes you're not using L2ARC. If you're using L2ARC, that creates additional pressure on the ARC and you may actually run into problems, because then you've got TWO things that are stressing the ARC.

Oh I see @tfran1990 has already quoted another series of answers on the issue.

Anyways, your real question was E3 or E5. The 64GB limit on E3 probably limits you if you're planning to make a massive pool. If you're looking at 120TB of raw space, you can definitely do it on E3 but if you get fragmentation as the pool fills (you didn't state a use case?) then you have the potential to lose performance. It's certainly possible that you could be able to do L2ARC but I feel like it's one of those "look at the stats once the thing is running and full of data" dealies to know if it is worth it. You can always remove an L2ARC if it is causing distress, but the E3 at 64GB means you cannot add RAM to fix performance issues.

If you go E5, you can start at 64GB and add a 256GB L2ARC right away, knowing that you can address performance issues with 32GB or 64GB more RAM.

 

[tfran1990]

I may be wrong but i thought FreeNAS does not only ARC with ram but "other things" like preventing data rot by refreshing and checking for flipped bits. working with compression by modifying data blocks and checking parity.

 

[jgreco]

I'm sorry but I simply cannot parse your point or meaning out of that. What are you asking?

 

[tfran1990]

I'm sorry but I simply cannot parse your point or meaning out of that. What are you asking?

correct me if im wrong but i thought FreeNAS does ARC with ram and "other things" like preventing data rot by refreshing and checking for flipped bits and working with compression by modifying data blocks and checking parity.

 

[danb35]

FreeNAS does lots of things in RAM (indeed, it does pretty much everything in RAM, as does every other operating system), but the primary reason that the recommend amount of RAM scales (roughly) with the storage capacity is ARC. Checksums, compression, etc., happen as data is read and written, but don't consume RAM on a long-term basis.

 

[mzarrugh]

If you don't care about performance, you should be able to do ~100-120TB on 64GB of RAM

My performance needs aren't insane. The system will have no more than 6 ports of Gigabit Ethernet in LACP.

Potentially 80 people will have some of their data on it. But up to 20 will access it at once. 20 is the extreme case here.

I believe that the majority of users will access it over ac wifi rather than wired.

you didn't state a use case

The use case is samba shares. I don't expect actual work being done live on the shared file. Mostly copy and paste.

The file types are not known exactly. But nothing out of this world.

Also, some use of plex.

 

[jgreco]

Six ports of gigE is heavy usage. Poor performance could be a tenth of that (600Mbps), or even less. If you're expecting multigigabit performance for 80 users, you're solidly in "small departmental fileserver" territory, and you should expect that sooner or later you are likely to want to need 128GB of RAM for a 120GB pool, depending on the actual workload, especially after a year has passed and there's fragmentation.

There isn't any HARM in getting 64GB and seeing how it goes, but you probably don't want to lock yourself into a platform that limits you to that.