How much RAM should I use? (supermicro config)

[DangerIsGo]

FreeNAS-9.10.2-U6
My current chassis/MB is this: https://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1CR36N.cfm
http://www.supermicro.com/products/motherboard/Xeon/C600/X9DRi-LN4F_.cfm
My RAM: http://www.compsource.com/pn/DTM64400B/Dataram-127/8gb--240pin-2rx4-Registered/
I have 12x8GB sticks of ECC Registered filled up for the CPU 1.
CPU 1 is for the front backplane.
CPU2 is not installed as it's only for the rear backplane. (24 HDDs in the front, 12 in the back)
I have 18 4TB HDDs (in 3x6HDD ZRAID6 config) installed and am in the planning phases of installing 6 more 4TB HDDs for another ZRAID6 dev.

Currently, I am using 96GB of RAM for the main pool. When that is filled, I want to fill the next CPU/RAM/12HDDs.

I have a few questions:
1) Because of how the MB is configured (12 RAM slots for CPU1, 12 for CPU2), do I have to use the same RAM for CPU2 or can I get away with a different manufacturer? (of course as long as its ECC Registered as the current stuff is)
2) My supplier only was able to locate 6 sticks which would put me at 48GB (for that CPU, 144GB total). Would that be enough for the rear drives (totaling 48TB of HDD space when filled)? I also have several jails with NZBGet, Emby, Sonarr, CouchPotato, Transmission running.
3) Is there any way to find out how much of the RAM is being utilized over a period of time? Maybe 96GB is overkill and I would be able to leverage some RAM from there if need be?
Thanks!

 

[Ericloewe]

1) Because of how the MB is configured (12 RAM slots for CPU1, 12 for CPU2), do I have to use the same RAM for CPU2 or can I get away with a different manufacturer? (of course as long as its ECC Registered as the current stuff is)

I'd never heard that question, but logic dictates that you can use different RAM on the second CPU (same specs are probably a good idea, due to BIOS limitations). Even on a single CPU it's going to be fine 99% of the time.

2) My supplier only was able to locate 6 sticks which would put me at 48GB (for that CPU, 144GB total). Would that be enough for the rear drives (totaling 48TB of HDD space when filled)? I also have several jails with NZBGet, Emby, Sonarr, CouchPotato, Transmission running.

144GB is a crapton of RAM for most applications. Even the single CPU is probably enough.
L2ARC might make more sense than blindly adding more RAM.

3) Is there any way to find out how much of the RAM is being utilized over a period of time? Maybe 96GB is overkill and I would be able to leverage some RAM from there if need be?
Thanks!

Yes, of course, it's in the reporting section.

 

[DangerIsGo]

I'd never heard that question, but logic dictates that you can use different RAM on the second CPU (same specs are probably a good idea, due to BIOS limitations). Even on a single CPU it's going to be fine 99% of the time.

Is there *any* possibility of data corruption or pool failure using different manufacturers (but still ECC/Registered RAM)? I just came across the X10 motherboard RAM recommendations and was wondering if anything like that existed for the X9 (mine)?

144GB is a crapton of RAM for most applications. Even the single CPU is probably enough.

I'd love to keep 1 CPU but with the way the MB is designed, I need the 2nd CPU in order to access the rear backplane (and additional 12 memory slots).

 

[Ericloewe]

Is there *any* possibility of data corruption or pool failure using different manufacturers (but still ECC/Registered RAM)? I just came across the X10 motherboard RAM recommendations and was wondering if anything like that existed for the X9 (mine)?

It's ECC. It either works, works and warns you something was corrected, doesn't work and the system immediately panics or works so badly the system doesn't get far with all the memory corruption going on.

 

[Pezo]

the way the MB is designed, I need the 2nd CPU in order to access the rear backplane

That doesn't sound right. Can't you just connect the HBA where the rear backplane is connected to a PCIe slot from the first CPU?

 

[Ericloewe]

That doesn't sound right. Can't you just connect the HBA where the rear backplane is connected to a PCIe slot from the first CPU?

Yeah, there's some significant confusion in this thread.

 

[DaveY]

From Supermicro's website:

* (Both CPUs need to be installed for full access to PCI-E slots and onboard controllers. See manual block diagram for details.)

For memory, it should be OK to use different brand/model, but you need to keep speed and ECC the same. If you had a Dell PowerEdge server, then you do need to have same brand/model/speed/capacity for both CPUs across the DIMM slots. And they need to be in multiples of 4 or you'll get memory errors on POST.

144GB total is plenty for your setup. Just because CPU2 is needed to access the backplane controller doesn't mean only 48GB of RAM is available to those drives. Unless Supermicro does something funky with the RAM, FreeNAS will use the entire 144GB of RAM to manage all the drives it sees.

 

[DangerIsGo]

That doesn't sound right. Can't you just connect the HBA where the rear backplane is connected to a PCIe slot from the first CPU?

You know what, you clarified something for me. (Thanks) While you are correct, I was confusing it with the need to access the second set of 12 RAM slots to expand my RAM capacity. Over the years, I must have been interchanging both thinking I needed the second CPU to operate the rear backplane.

Yeah, there's some significant confusion in this thread.

All has been clarified from my previous statement! (I hope)

144GB total is plenty for your setup. Just because CPU2 is needed to access the backplane controller doesn't mean only 48GB of RAM is available to those drives. Unless Supermicro does something funky with the RAM, FreeNAS will use the entire 144GB of RAM to manage all the drives it sees.

With my current setup (18x4TB drives), in the near future the front will be at capacity (24x4TB = 96TB, current RAM = 96GB) and will need to populate the rear drives. With the jails I have and currently at 80% capacity (hence the need to expand), is 96GB RAM enough for 144TB (or more if I ever go 6/8TB HDDs) or should I look into expanding my RAM?

With that said, for the Supermicro X9, are there any recommendations? Nemix/black diamond is pretty inexpensive I saw but its PC3.
Can you run PC3L and PC3? (My stuff is PC3L which runs $100+ on NE)

Thanks

EDIT: Just saw this on amazon: https://www.amazon.com/8G-Memory-DD...24PY/ref=sr_1_7?ie=UTF8&qid=1505011119&sr=8-7

 

[DaveY]

From Supermicro's website, it does say the board supports both PC3 and PC3L, but recommend you don't mix them for obvious reasons. Supermicro has a list of tested memories in the link below. I would suggest sticking to their list for stability. Hynix comes up a lot in their tested memories. I've used their Registered ECC before and they work great in Dell servers. Don't know about Sueprmicro boards though so you may have to do a bit more research. Here's the link to Supermicro's recommended memory list.

https://www.supermicro.com/products/system/4U/6047/SSG-6047R-E1CR36N.cfm