VM Virtual CPU/Cores/Threads jargon

[Apollo]

Hi there,

I am trying to run a VM on TrueNAS, and I am unable to make sense of the information required to setup the VM, nor do I understand the math behind the error message.
How am I to make sense of the data?
What is the reasoning and relationship between Virtual CPU, Cores and Threads?
What am I to use?

 

[Samuel Tai]

This is for HyperThreading support. Think of it this way: a vCPU can consist of a number of cores; each core can support some number of simultaneous threads (usually 1-2). The product of all 3 numbers gives the total number of threads in flight. The error message indicates bhyve has a hard-coded maximum of 16 threads.

This mirrors the physical topology of real CPUs. For example, my Xeon E3-1230L v3 has 4 cores, each of which can support 2 threads. To mirror this in the VM config, you'd set 1 vCPU, 4 cores, 2 threads.

 

[jasonsansone]

Does anyone know if you can define package or thread affinity for NUMA? I have two CPU’s of 12 cores with hyper threading for a total of 48 available threads. Running a VM with 16 threads will logically allocate across both packages and memory controllers. I would prefer to lock the VM to one CPU.

 

[Apollo]

This is for HyperThreading support. Think of it this way: a vCPU can consist of a number of cores; each core can support some number of simultaneous threads (usually 1-2). The product of all 3 numbers gives the total number of threads in flight. The error message indicates bhyve has a hard-coded maximum of 16 threads.

This mirrors the physical topology of real CPUs. For example, my Xeon E3-1230L v3 has 4 cores, each of which can support 2 threads. To mirror this in the VM config, you'd set 1 vCPU, 4 cores, 2 threads.

Hi Samuel,

Thank you for the clarification. One thing I don't have a grasp about yet, is the benefit or drawback for using 1 or more vCPU's to acheive the maximum 16 count.

Let's say I have 8 cores with 2 threads per core.
What should I expect in term of performance and whatnot between the following configurations. Are there any non valid configurations?

1) 1 vCPU * 8 CPU * 2 Threads = 16
2) 2 vCPU * 4 CPU * 2 Threads =16
3) 2 vCPU * 8 CPU * 1 Thread =16
4) 8 vCPU * 1 CPU * 2 Threads = 16

 

[jasonsansone]

Hi Samuel,

Thank you for the clarification. One thing I don't have a grasp about yet, is the benefit or drawback for using 1 or more vCPU's to acheive the maximum 16 count.

Let's say I have 8 cores with 2 threads per core.
What should I expect in term of performance and whatnot between the following configurations. Are there any non valid configurations?

1) 1 vCPU * 8 CPU * 2 Threads = 16
2) 2 vCPU * 4 CPU * 2 Threads =16
3) 2 vCPU * 8 CPU * 1 Thread =16
4) 8 vCPU * 1 CPU * 2 Threads = 16

Licensing. Windows has limits on CPU’s and ties licensing to the number of processors.

 

[Apollo]

Licensing. Windows has limits on CPU’s and ties licensing to the number of processors.

@jasonsansone, this doesn't answer the question.
Windows has some limits but easily supports 16core/32threads.
Regardless, I intend to run Ubuntu but it could be anything like Debian...
Understanding the limits and know how the OS interact with the vCPU's will allow making the right config.

 

[jasonsansone]

@jasonsansone, this doesn't answer the question.
Windows has some limits but easily supports 16core/32threads.
Regardless, I intend to run Ubuntu but it could be anything like Debian...
Understanding the limits and know how the OS interact with the vCPU's will allow making the right config.

Windows supports multiple threads, not necessarily multiple processors. As a test, use more than two vCPU's in a Windows 10 VM. Only two will appear as available. Prior to Windows Server 2019, licensing was based on per-cpu, not per-core. Microsoft realized that due to virtualization, you could assign many physical cpu's to a VM that appeared like a single vCPU. Hence why you can adjust the difference between vCPU's and vThreads.

 

[Apollo]

Windows supports multiple threads, not necessarily multiple processors. As a test, use more than two vCPU's in a Windows 10 VM. Only two will appear as available. Prior to Windows Server 2019, licensing was based on per-cpu, not per-core. Microsoft realized that due to virtualization, you could assign many physical cpu's to a VM that appeared like a single vCPU. Hence why you can adjust the difference between vCPU's and vThreads.

Thank you for the clarification. I wasn't aware of the CPU limit from that point of view.
Any ideas about Linux, would it make sense to use 1vCPU or more?

 

[Patrick M. Hausen]

@jasonsansone Understood and I already knew this. But my takeaway of @Apollo's question - that I cannot answer either and am just as curious about - is:

What is better/recommended?

Say I have those 8 cores with 2 threads each in my system and I want to use 4 threads for a particular application should I set

- 1 package, 4 cores, 1 t/c
- 1 package, 2 cores, 2 t/c
- 2 packages, 2 cores, 1 t/c
- ...

What are the observable effects on e.g. a Linux VM and the system as a whole? Does one of those settings impose additional overhead either in the guest scheduler or in bhyve? Should I match the real CPU topology? ...

Thanks,
Patrick

Edit: for example I can imagine on a machine with more than one package you want to make as probable as possible that all threads of a VM are scheduled on the same package during their lifetime. Because NUMA and stuff ...

 

[Samuel Tai]

Does anyone know if you can define package or thread affinity for NUMA?

This isn't currently doable in bhyve, sorry.

 

[Apollo]

@jasonsansone Understood and I already knew this. But my takeaway of @Apollo's question - that I cannot answer either and am just as curious about - is:

What is better/recommended?

Say I have those 8 cores with 2 threads each in my system and I want to use 4 threads for a particular application should I set

- 1 package, 4 cores, 1 t/c
- 1 package, 2 cores, 2 t/c
- 2 packages, 2 cores, 1 t/c
- ...

What are the observable effects on e.g. a Linux VM and the system as a whole? Does one of those settings impose additional overhead either in the guest scheduler or in bhyve? Should I match the real CPU topology? ...

Thanks,
Patrick

@Patrick M. Hausen, this was exactly my concern and thank you for rephrasing it.

 

[Patrick M. Hausen]

This isn't currently doable in bhyve, sorry.

After reading @jasonsansone's post ...

If you match the topolgy of your host i.e. in his example configure the VM with 2 packages, 4 cores, 2 threads - will those resources presented to the guest be tied to the matching host threads?
I.e. will the guest have 4 cores on one of the physical CPUs and 4 on the the other one? Will the memory the guest allocates be connected to one or the other package if the guest is NUMA aware?

Kind regards,
Patrick

 

[jasonsansone]

After reading @jasonsansone's post ...

If you match the topolgy of your host i.e. in his example configure the VM with 2 packages, 4 cores, 2 threads - will those resources presented to the guest be tied to the matching host threads?
I.e. will the guest have 4 cores on one of the physical CPUs and 4 on the the other one? Will the memory the guest allocates be connected to one or the other package if the guest is NUMA aware?

Kind regards,
Patrick

I can't claim to have conducted extensive testing, but I can say that my limited testing shows there is no correlation between the virtual and physical topology. Assigning one or more virtual packages doesn't alter that all load gets evenly distributed across all available physical cores. There is no maintained thread, core, or processor affinity (hence my prior question).

For a test, create a VM for transcoding (h264 or h265). Slowly increase the thread count in relation to your physical threads. Regardless of the vCores, you will see the load balanced across all physical cores.