Multi Processor NUMA question

[catherineva11e]

Hi I have a dual processor system and I'm trying to figure out how to access the two seperate NUMA-Domains. Basically i want to run a specific process on a single domain. Noob here but I'd like to think I know a little. I've read through man numa. At least I think I'm piecing things together.

My BIOS has a setting named "NUMA support" with these options "disable, enable, NUMA for SLES 11". Someone also said I need to change node interleaving but it's not in my BIOS. All 3 "NUMA support" settings don't seem to change a thing.

I've seen elsewhere in this forum where others systems report CPU cores on separate numa-domains.

Code:

...
cpu38: <ACPI CPU> numa-domain 1 on acpi0
cpu39: <ACPI CPU> numa-domain 1 on acpi0
cpu40: <ACPI CPU> numa-domain 2 on acpi0
cpu41: <ACPI CPU> numa-domain 2 on acpi0
...

Below is what my system reports.

Code:

# sysctl hw.machine hw.model hw.ncpu vm.ndomains vm.default_policy vm.phys_locality vm.phys_segs
hw.machine: amd64
hw.model: Intel(R) Xeon(R) CPU           X5690  @ 3.47GHz
hw.ncpu: 24
vm.ndomains: 1
vm.default_policy: rr
sysctl: unknown oid 'vm.phys_locality'
vm.phys_segs:
SEGMENT 0:

start:     0x10000
end:       0x99000
domain:    0
free list: 0xffffffff82170080

SEGMENT 1:
.
.
.
SEGMENT 5:

start:     0x100000000
end:       0x17a5ad9000
domain:    0
free list: 0xffffffff8216fba0


# dmesg | grep CPU
CPU: Intel(R) Xeon(R) CPU           X5690  @ 3.47GHz (3466.86-MHz K8-class CPU)
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
...
SMP: AP CPU #22 Launched!
SMP: AP CPU #23 Launched!
cpu0: <ACPI CPU> on acpi0
cpu1: <ACPI CPU> on acpi0
...
cpu22: <ACPI CPU> on acpi0
cpu23: <ACPI CPU> on acpi0
coretemp0: <CPU On-Die Thermal Sensors> on cpu0
coretemp1: <CPU On-Die Thermal Sensors> on cpu1
...
coretemp22: <CPU On-Die Thermal Sensors> on cpu22
coretemp23: <CPU On-Die Thermal Sensors> on cpu23

# dmesg | grep SMP
FreeBSD/SMP: Multiprocessor System Detected: 24 CPUs
FreeBSD/SMP: 2 package(s) x 6 core(s) x 2 hardware threads
SMP: AP CPU #1 Launched!
SMP: AP CPU #2 Launched!
...
SMP: AP CPU #22 Launched!
SMP: AP CPU #23 Launched!

Am I correct in thinking that if NUMA was configured correctly in the BIOS the system should be reporting both numa-domain 1 & 2?

So far I've used cpuset to configure cpu afinity (for a specific process) but I'm not sure if that is correctly dividing the memory.

 

[Heracles]

Hey Catherine,

Usually, it is better to let the operating system balances the process freely between the different NUMA. What is it that you try to achieve by forcing processes to a NUMA or another ? When I tried to play with numa.Affinity in my ESXi server, I confirmed that I got it right but at the end, I was not even close to outperform ESXi in its hardware management.

Some settings will use NUMA for redundancy, one NUMA mirroring the other so the system will survive problems in CPU and RAM. In this case, you will see only half of your hardware resources. Is it what you try to achieve ?

 

[catherineva11e]

I'm encoding video. When I run handbrake it loads the CPU from 50-90% fluctuating constantly. I can run two instances encoding different videos and saturate the CPU. My thought is if I restrict each instance to a separate numa node it would increase efficiency.

 

[catherineva11e]

So would y'all say ditch numa and cpuset and just run one (or two?) instance/encode on all cores and let the OS handle everything?

 

[Heracles]

Hi again Catherine,

Indeed, I would let the operating system handles the NUMA by itself...