ECC Ram memory size versus performance and cost

[Ed Clarke]

I understand that terrible things can happen if you run low on memory while using ZFS. Is there any way to tell exactly when I am in danger of running out of space? Right now I'm using 16GB of ECC ram and a seven drive ZFS pool using four data drives and three parity drives. It is possible to upgrade from 16GB to 32GB for about $150. It's also possible to upgrade to 64GB... for $900.

I'd prefer to add another seven drive vdev to the pool if possible rather than spend $900 on ram. When my economic situation improves I'll upgrade to 32GB if that makes sense. What performance tools are on FreeNAS to let me catch problems before they catch me?

 

[cyberjock]

Ed,

The danger isn't with ZFS running low, at least not exactly. The problem is that ZFS needs a certain amount of RAM to function, even at an atrocious performance level. Well, the services you are running as well as the other kernel code needs space too. FreeNAS also uses something like 1.5GB of RAM for a ramdisk and loads the OS into the ramdisk on bootup. Add all of those up and you end up with something that's <8GB of RAM (nobody knows the exact number). Once you go below that number things get dangerous because you aren't meeting minimum needs for the system.

Once you've got more than that ZFS should free up RAM used for ARC as needed for other services the system uses. So once you hit that 8GB of RAM you are pretty much 'safe'. After that more RAM is only to improve performance of the pool.

So if your pool is already performing acceptably for you then you don't need more RAM. But, if you got cash burning a hole in your pocket and you'd like to max it out at 32GB since you can I definitely say go for it.

From my personal experience one of two things happens when you don't have enough RAM for your pool size:

1. Pools slowly get slower and slower and you get kind of disappointed.
2. Your pool goes off a cliff in the performance arena. You got 100MB/sec yesterday and today you can't even stream a single SD movie from your server without it skipping.

In either case your data really isn't at risk from insufficient RAM so long as you got that 8GB minimum. It might be useless because of its performance, but your data will be safe.

Hope this all helps!