Quick question: How much RAM for VMs usable?

[IceBoosteR]

Hi all,

I am planning around with my server gear. For my FreeNAS system I have 40TB of storage and 24GB of ECC memory.
Currently I have two VMs running with 1GB and 3GB of memory, so I have 20GB for the host-OS and ZFS and 4GB for VMs only. But I would like to add more VMs, waht would you think? How much RAM can I give to the VMs at all?
16GB+8GB?
There is no plan for hardware upgrades atm. Just a quick question ;)
Basically my system has not much to do, most of the time it is idling and when I need data they are "mostly" not in ARC ...
Any suggestions? Thanks.

 

[MrToddsFriends]

But I would like to add more VMs, waht would you think? How much RAM can I give to the VMs at all?
16GB+8GB?

If I was in your place I would definitely do my own experiments with the existing two and maybe a third "dummy" VM. Depending on network share usage you may want to see from your own experience if less memory available for ARC (also holding metadata) will lead to a noticeable impact on performance or not.

Are you seeing very low numbers in Reporting -> ZFS -> ARC Size in the FreeNAS GUI in your current situation (with 4GB reserved for VMs)? Any peculiarities in Reporting -> Memory?

 

[IceBoosteR]

Thanks for your answer.
Normally i do really not care about the ARC. Not because I am to stupid to understand what the benefits are, it's because even if I need data which are not in ARC, I got easily my 110-120 megs per second through ethernet. My VMs and jails are not regulary requesting data. Nevertheless I have attached a screenshot from my actual ARC statistics, just from normal usage from today. I have activated 2 VMs and 2 jails.
But I will try to find out if I rach a bottleneck if I go with more VMs... maybe it is just a number in my head which I want ZFS/FreeNAS to use on their own :=)

 

[MrToddsFriends]

Reporting -> ZFS -> ARC Size doesn't show a bottleneck. If Reporting -> Memory is inconspicuous, too, do not hesitate to reserve another 4 GB for an additional VM (or for whatever purpose you like).

The "1 GB of RAM per terabyte of disk capacity" guideline is old and somewhat overstated as already pointed out in the manual.
http://doc.freenas.org/11/intro.html#ram

Also monitor the mentioned Reporting graphs over longer time spans (daily/weekly/monthly views accessible via the "-" magnifying glass button).

 

[Chris Moore]

FreeNAS needs to keep that 8GB minimum available for some underlying functionality of ZFS that is not part of ARC and if you have less than 8GB available, it can cause instability. It might be a good idea for you to bump the RAM up to 32GB. It is only money, right?

 

[IceBoosteR]

Thanks for your answers guys.
@Chris Moore yes, and no. The actual reason for not upgrading is the fact, that I may need to think about a dedicated ESXi host for VMs. Powerful, but not consuming a lot of power. I am not sure at this point if I need ECC there or not ;) (propably I don't need it, but its the same pricetag).
On the other hand, I am using DDR3 memory. It would be better to have a system with DDR4, so I do not want to invest in a dying system, as DDR4 is the standard now. I think thats the major point here.
Nevertheless, I could use cloud services on the other hand, too...
And I do not see any reason for an upgrade to a new platform, as my E3-1225v3 has some CPU-cycles left to use^^

Edit: Pricetag for upgrading would be 160€, which is maybe ok. But then I have 2x4GB sticks laying around, which is not economic at all XD

 

[Chris Moore]

Edit: Pricetag for upgrading would be 160€, which is maybe ok. But then I have 2x4GB sticks laying around, which is not economic at all XD

I know that US prices don't always translate to other countries, but you might take a look at something like this:

$109.00
Intel Xeon E5-2650 V2 2.6GHz 8 Core
https://www.ebay.com/itm/SR1A8-Inte...8-Core-CM8063501375101-Processor/192524131095

$199.99
SuperMicro X9SRL-F Motherboard
https://www.ebay.com/itm/SuperMicro-X9SRL-F-Motherboard/153025744510

$140.00
32GB (4X8GB) ECC Memory, Registered
https://www.ebay.com/itm/32GB-4X8GB-MEMORY-RAM-4-Supermicro-X9SRW-F-X9SRW-3F-X9SRL-F-X9SRL-B103/282323196602

That system board can go up to 256GB of RAM if I recall correctly.

 

[IceBoosteR]

I know that US prices don't always translate to other countries, but you might take a look at something like this:

$109.00
Intel Xeon E5-2650 V2 2.6GHz 8 Core
https://www.ebay.com/itm/SR1A8-Inte...8-Core-CM8063501375101-Processor/192524131095

$199.99
SuperMicro X9SRL-F Motherboard
https://www.ebay.com/itm/SuperMicro-X9SRL-F-Motherboard/153025744510

$140.00
32GB (4X8GB) ECC Memory, Registered
https://www.ebay.com/itm/32GB-4X8GB...RW-F-X9SRW-3F-X9SRL-F-X9SRL-B103/282323196602

That system board can go up to 256GB of RAM if I recall correctly.

Love these prices ;)
So What probalby would be an issue is the board. It is not very common to get a SuperMicro in Germany, and if you see one, pricetags are hilarious... (350 ish)
RAM and CPU should be fine. But you forget that I will need a PSU (+80€) and a proper case (+75€) and I am not 100% convinced of the idea to be honest.

 

[MrToddsFriends]

But I would like to add more VMs, waht would you think? How much RAM can I give to the VMs at all?
16GB+8GB?

Back to your original question: How does the "16GB+8GB" memory distribution model work for you? Which means 16GB for FreeNAS, including ARC for plenty of storage (40TB), and 8GB for VMs?

Did you see any bottlenecks or pitfalls so far, when incrementing the memory reserved for VMs from your formerly used "20GB+4GB" memory distribution model?

 

[IceBoosteR]

Back to your original question: How does the "16GB+8GB" memory distribution model work for you? Which means 16GB for FreeNAS, including ARC for plenty of storage (40TB), and 8GB for VMs?

Did you see any bottlenecks or pitfalls so far, when incrementing the memory reserved for VMs from your formerly used "20GB+4GB" memory distribution model?

Unfortunately I haven't had the time to test it, or do even something on my server(s). Other stuff is blocking my time atm :/

 

[IceBoosteR]

So I had some time now to see how my system is handling the situation. I am running now a jail with Urbackup and two Ubuntu-Server VMs with 3GB memory each. I do see about 500-1000MB of Swap usage, so I assume that I should increase RAM.
@Chris Moore
As I have said, prices for motherboards and CPUs are higher here in Germany but I found a combination of a
Intel Xeon E5-1650v2 (6x3,5Ghz, but 130W TDP) = 140€
a new Supermicro MBD-X9SRA-B for 220€
and I get 16GB of DDR3 reg RAM for 80€

Is that a good idea for a upgrade? The most concern I do have with the idle power consumption. My system is running 24/7, should be stable and use less power. Right now my system is using around 70W in idle, around 30W is the system itself without any peripherals (HDDs and stuff).
Or do I go and buy a Ryzen setup? I suggest you would say no, because its consumer graded hardware.
Threadripper is a different story, would fit to the purpose but power consumption is high.
On the other hand 8 Core used Xeons often has low clock speed, whichis maybe a bottleneck for 10GB(e) with Ssamba in the near future.

I gt the idea to get the new Coffee-Lake Xeons (former E3-1230v7) are now named to E-2416G. But that would cost about 1000ish for a new board, CPU and 32GB of RAM and I do not know if this is worth it. I do like the idea of a system which is running all my stuff, no need for a second system, but if the powerconsumption of two systems is low enough (energy pricec per KWh are incredible....) I can also do that.

In the future I would like to add an NVR system with 4-6 cameras and Graphite and Grafana, the systems should be capable of that. I also do not mind to switch to a ESXi solution.

Many thanks for any help & ideas ;)

(PS. DDR4 reg RAM cost 200 bucks per 16GB)

 

[Chris Moore]

a new Supermicro MBD-X9SRA-B for 220€

I wouldn't get the X9SRA-B because it doesn't include IPMI. It has Audio instead.

The one you want to get, if you can find it, is the X9SRL-F (the -F means it has IPMI)

The X9SRL-F also has a couple extra PCIe slots, but the big thing is the IPMI.

Or do I go and buy a Ryzen setup? I suggest you would say no, because its consumer graded hardware.

The trouble with Ryzen is that there are still some incompatibilities, based on the comments I have read in the forum from people that are already trying to use them. I am sure they will eventually be fully supported, but at this time there are still some problems.

On the other hand 8 Core used Xeons often has low clock speed, whichis maybe a bottleneck for 10GB(e) with Ssamba in the near future.

I have the board above with a 8 core at 2.6 GHz and I was able to copy 9TB of data from one pool to another in about 2 hours and 15 minutes.
Here is an illustration of the CPU utilization during that:

As you can see, the CPU load isn't an issue. The CPU is mostly tasked by virtual machines and things like Plex transcoding. File operations are pretty light weight.

In the future I would like to add an NVR system with 4-6 cameras and Graphite and Grafana, the systems should be capable of that. I also do not mind to switch to a ESXi solution.

I have one of the X9SRL-F boards fitted with a 10core Xeon @ 2.8 GHz that I use for my ESXi host and this one fitted with the 8 core Xeon for my FreeNAS and I have not had a problem doing all I wanted to do. There are many options, but I chose this hardware because of the low cost and it is still (despite age) very capable.

 

[IceBoosteR]

Hi @Chris Moore ,
thank you for your answer.

I had another look on Ebay. Unfortunately there is no X9SRL-F available and I understand the advantages of the IMPI. But as this option is not available I would not mind using the X9SRA-B instead - and it's new. Don't know if the integrated Intel NICs are good, but I do like the external USB 3.0 as I now must have an external USB 3.0 drive for daily backup (due to lost data).

I would agree to your Ryzen opinion, as Supermicro does not have an AM4 board available.

Can I ask how much watts your system is using? I see CPU isn't an issue - but think about Samba. It's single threaded IIRC and this is then bound to one CPU cure. Copy on the local machine may ot be affected by this.

I am likely to use used hardware, I have two rackserver (HP ones) and they are running fine. I have even swapped the CPUs of one to 12 cores (2x6) which is great for transcoding.
But the most issue is idle power consumption, as my system is using right now as much power (70w) that I could yearly buy one 4TB and a 450W PSU for that ;)

Cheers
Ice

 

[IceBoosteR]

Another important question: Can Supermicro handle all kinds of registered DDR3 RAM? I mean, should I use Samsung, HP or Dell as a brand?

On the other hand, for around 300€ I could get an E5-2690v2 (10 cores 3Ghz)...
But looking at the passmark score, the newly Xeons (v7) should deliver similar results and I guess with lower power consumption. Therefore I need to wait for it. Hard decision^^

 

[Chris Moore]

But looking at the passmark score, the newly Xeons (v7) should deliver similar results and I guess with lower power consumption. Therefore I need to wait for it. Hard decision^^

The new CPUs are so expensive that you would need to run it for 20 years to make up the difference in cost on the power savings.

Another important question: Can Supermicro handle all kinds of registered DDR3 RAM? I mean, should I use Samsung, HP or Dell as a brand?

I use Samsung memory with my system but there is plenty of good memory out there. If it is Registered ECC and the right clock speed, it should be good...

 

[IceBoosteR]

The new CPUs are so expensive that you would need to run it for 20 years to make up the difference in cost on the power savings.

I use Samsung memory with my system but there is plenty of good memory out there. If it is Registered ECC and the right clock speed, it should be good...

For sure?
Damn, you are making me spending money on more server hardware:p
Btw: is this a downgrade to go from a E3-1225v3 to E5-2690v2 ?
Now I need to think about my T20 rig ...

Thnaks for the answer :)
Is there anything else (downside) of the MBD-X9SRA-B apart of hte IPMI?

 

[Chris Moore]

Btw: is this a downgrade to go from a E3-1225v3 to E5-2690v2 ?

Technically, it might be, but I don't think so because a Xeon E5 is a significantly better processor than a Xeon E3, with more PCIe lanes and ability to control more RAM.

 

[Chris Moore]

Thnaks for the answer :)
Is there anything else (downside) of the MBD-X9SRA-B apart of hte IPMI?

The X9SRA-B has that silly old 32bit PCI slot (the white one) that isn't good for much of anything and I don't think the board has any video capability built-in, so you would need a video card. It really is a workstation system board for a high-end desktop. Not saying anything bad about it, that is the model board I have in my wife's desktop with a Xeon 1650 v2 CPU @ 3.5GHz. It works like a champ for regular desktop computer chores, but it really isn't optimized for use as a server. It is like using a Ferrari to haul furniture. It might be able to do it, but it really isn't made for it.

 

[Chris Moore]

I had another look on Ebay. Unfortunately there is no X9SRL-F available and I understand the advantages of the IMPI.

https://www.ebay.com/itm/Supermicro...-8C-2-0GHz-w-Heatsink-I-O-Shield/352401605107

Can I ask how much watts your system is using? I see CPU isn't an issue - but think about Samba. It's single threaded IIRC and this is then bound to one CPU cure. Copy on the local machine may ot be affected by this.

My power consumption is a little high because of the number of drives I have. I only have 12 drives in the storage pool, but I have 4 more in a backup pool and 16 more in an iSCSI pool and at the moment I have another 12 in the server to make a copy of the main storage pool, AND I have 4 SAS SSDs in there too. It is a 48 bay chassis and I have it completely full right now. Still, the power consumption is about 380 watts, but it was more like 120 to 150 when I only had the data pool and backup pool and most of that was the drives. In general, you can figure 7 to 10 watts per drive.
I have not had any trouble with SMB. I don't have any clients that are fast enough to continuously max out my 10Gb network, but I can spike to full performance of the NIC and the limitation appears to be the client system. I will try to run some testing to see if I can document that.

But the most issue is idle power consumption, as my system is using right now as much power (70w) that I could yearly buy one 4TB and a 450W PSU for that

Sorry, I get my electricity for free, so I have a hard time understanding the cost involved. When I look at my consumption, the only concern I have is if I am overloading the circuit. The servers and desktops I have running in my home office draw around 1000 watts (120 volt) and my biggest concern is with the air conditioning to keep the room cool.

 

[IceBoosteR]

https://www.ebay.com/itm/Supermicro...-8C-2-0GHz-w-Heatsink-I-O-Shield/352401605107

My power consumption is a little high because of the number of drives I have. I only have 12 drives in the storage pool, but I have 4 more in a backup pool and 16 more in an iSCSI pool and at the moment I have another 12 in the server to make a copy of the main storage pool, AND I have 4 SAS SSDs in there too. It is a 48 bay chassis and I have it completely full right now. Still, the power consumption is about 380 watts, but it was more like 120 to 150 when I only had the data pool and backup pool and most of that was the drives. In general, you can figure 7 to 10 watts per drive.
I have not had any trouble with SMB. I don't have any clients that are fast enough to continuously max out my 10Gb network, but I can spike to full performance of the NIC and the limitation appears to be the client system. I will try to run some testing to see if I can document that.

Sorry, I get my electricity for free, so I have a hard time understanding the cost involved. When I look at my consumption, the only concern I have is if I am overloading the circuit. The servers and desktops I have running in my home office draw around 1000 watts (120 volt) and my biggest concern is with the air conditioning to keep the room cool.

Thank you for your answer and sharing of the information. I will have a look if shipping and taxes of this offer is suitable.
Anyway - I really smiled a bit when I've read that you don't need to care about electricity. I would like to have (or not to have?) to think about that :D :D
But your servers really use a lot of power^^