Node Interleaving and FreeNAS

[SubnetMask]

I recently learned something on my Blue Iris machine regarding Node Interleaving on multi socket systems. I had found that because Windows recognizes NUMA Nodes and Node Interleaving is disabled by default (At least on the Dell Poweredge machines), Blue Iris, was only running on one socket, so the other socket was sitting pretty much idle. After someone suggesting that I look for Node Interleaving, I did and enabled it, after which there were no longer NUMA nodes and the Blue Iris load was spread across both sockets. Of course this was on Windows.

That being said, how does having Node Interleaving either enabled or disabled impact BSD/FreeNAS? Is there a preference for better overall performance, or is there really no significant difference?

 

[dlavigne]

Did you find the answer to your question?

 

[kdragon75]

I would with my limited knowledge, speculate that due to the number of single theaded services on FreeNAS, you would see a performance hit with anything that constantly migrated processes for one CPU to the other. Not to mention cross node memory access would still be slower.

 

[SubnetMask]

No, never found an answer. I did do some testing unrelated to this with the AES performance test, and regardless of if node interleaving was enabled or disabled, the performance was essentially the same. Not sure how much of a test that is in terms of how node interleaving affects overall performance with FreeNAS though. Now power saving, on the other hand, had a HUGE effect on the performance. With power saving enabled, it was probably about 60% of what the X5670's yielded, but with it disabled, it was about 50-60% faster than the X5670's.

While yes, CPU1 accessing memory on CPU2's controller would be slower than accessing memory addressed by itself, just how much slower, really? Would it be enough to really see any difference? On the other hand, in some cases, such as the Blue Iris example I 'discovered', having node interleaving disabled, thus presenting two NUMA nodes to the OS, had a SIGNIFICANT effect on Blue Iris's performance and constrained it to one CPU socket - Enabling node interleaving allowed it to spread its work out over both sockets / all 24 logical cores, thus significantly helping its performance and reducing overall system load.

 

[rvassar]

A little light reading:

https://www.usenix.org/legacy/event/bsdcon03/tech/full_papers/roberson/roberson.pdf

This paper is quite old, and Jeff Roberson has been working on NUMA as recently as 2018... But it gives you an overview of the scoring mechanisms in play. The kernel scheduler is going to do it's best to gather up the pages and herd the threads to minimize the cross socket transfers.

I'm not sure if FreeBSD has anything like Solaris pbind(1M), which allows you to bind processes to CPU's in creative ways...

 

[catherineva11e]

I'm really interested in this as well. Aside from file serving I also run plex and I've setup a jail to do video encoding. When encoding a video it doesn't saturate the processors, it sits around 75% but constantly varies greatly +/- 15%. I think if I restricted an encode to one NUMA node and run separate encodes on each node it would increase performance. I used cpuset to run two encodes on separate cpus and it definitely improved as the total CPU utilization sits at a relatively constant 90% +/- 5%.

Code:

cpuset -l 1-11 HandBrakeCLI --queue-import-file one.json
cpuset -l 13-23 HandBrakeCLI --queue-import-file two.json

That's great and everything but how do I enable NUMA nodes and assign a process to one?

 

[SubnetMask]

I'll answer the part of this that I can - If you have a system with multiple populated sockets, odds are pretty good each socket is being presented as a NUMA node as node interleaving is usually disabled by default. Only when Node Interleaving is enabled will the two NUMA nodes (in the case of a dual socket system) be presented to the OS as one NUMA node. I'm not sure of if there are any CPUs out there now that would have groups of cores presented as different NUMA nodes in the same processor package though.

 

[catherineva11e]

How do I enable/disable node interleaving?
How do I verify it's disabled?
How do I assign processes to specific nodes?

 

[SubnetMask]

1 How do I enable/disable node interleaving?
2 How do I verify it's disabled?
3 How do I assign processes to specific nodes?

1 It may vary a little from manufacturer to manufacturer and generation to generation, but on a Dell Poweredge R720, it's in the BIOS > Memory Settings > Node Interleaving.

2 Does it say 'Enabled' or Disabled' in the BIOS? Enabled = all NUMA nodes presented to the OS as a single NUMA node. Disabled = All NUMA nodes are presented to the OS individually.

3 Can't help you with that one.