Upgrading RAM down the line

[KempelofDoom]

So I'm rebuilding my server from the ground up and since I'm going with a server board and ECC memory, I can upgrade to 256GB of memory. I only have 8 slots and I can only get the 16GB sticks for now. So I figured that I would start with 64GB to start and add as I go. I usually just add memory as I need it but this is a different beast and I wanted to see if anyone had experienced issues or if it's allowed at all.

They make 32GB sticks but they cost $1000 a stick when the 16GB sticks run $180 each. Though if I used registered memory then I can expand to 512GB of memory when 64GB registered sticks come to market. Ultimately I want to expand to 100TB maybe 150TB but that won't be for another year or two.

 

[jgreco]

Get 4x16GB and then worry about it later iff it becomes a problem, which it probably won't be for 100-150TB, but if it is a problem, 128GB is an easy jump.

 

[KempelofDoom]

So FreeNAS won't give me a hard time if I were to upgrade the memory in intervals? One of the things I can think of is that in Linux, your swap is usually 2 times as large as your amount of memory. Is that the case with FreeNAS?

 

[jgreco]

Yeah, no, it's no problem at all. If you are not unhappy with the prospect of buying 4x16GB then we're having a very pleasant conversation. The problem becomes one of what makes the most sense financially. Most of the trouble around here is where people want to put 4GB on their 2005 vintage Pentium 4. FreeNAS requires minimum 8GB for any system, to be stable, a little more (maybe 16-32GB) to remain stable on large pools. The rule of thumb is 1GB per TB but once you get out past 32GB, this really becomes a whole lot more open.

So here's the stuff you Need To Know:

1) The extremes: If you are running a busy departmental fileserver (or backups or other frequent data access), 64GB is probably not enough. If you are running an archival system to store ISO's online, you are unlikely to ever need more than 64GB until you're up in the hundreds-of-TB range. For everything else, start with 64GB and feel it out.

2) If you're having a problem with IOPS for read speed, you are at a healthy size to consider a SSD for L2ARC. But only if you have a demonstrated need; the L2ARC will change some of the performance characteristics of the system negatively. If this turns out to be the case, identify the size of your working set - the amount of data read from pool on a regular basis - and see if it fits in maybe a 256GB SSD. That's a reasonable L2ARC. This is a HELL of a lot cheaper than RAM.

3) Adding another 64GB is not a problem at all. Drop it in, run memory tests for awhile, and have a go.

4) As your pool fills, you may start to feel a performance falloff. This is more-or-less expected as a pool gets more full. You can choose to suffer it or you can throw RAM at it.

As for swap, the swap on a FreeNAS system is there for crisis, not for normal use. The developers have seen fit to estimate that a 2GB-per-disk swap partition will roughly scale your typical FreeNAS box appropriately. The problems are mainly that the system can start to require a lot of memory when doing a pool import, and if you run out of physical RAM and need to swap, if there's no swap, there's no pool import possible. I don't really advocate tinkering with that, but it should be safe to increase the per-disk swap a bit (like to 4GB) if that makes you comfortable.