Increase Bhyve vCore Limit in TrueNAS 12?

[jasonsansone]

At present, "the product of vCPUs, cores, and threads must not exceed 16" for Bhyve virtual machines. A Bhyve VM in 11.3-U1 or in a 12 Nightly cannot have more then 12 vCores, regardless of tunables like hw.vmm.topology.cores_per_package or hw.vmm.topology.threads_per_core. I have three NAS which all have 24 cores / 48 threads. The 16 vCore limit means that a single VM can use only 1/3 of my total compute. I have seen multiple posts on this forum repeat that the 16 vCore limit is a hard Bhyve limit, but that isn't true. Is there any chance the vCore limit will be raised in TrueNAS 12? Thanks!

 

[Kris Moore]

Looks like in upstream FreeBSD this is still hard-set to 16:

freebsd-src/sys/amd64/include/vmm.h at stable/12 · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src

github.com

No idea what changing that does, I would have to assume there is some negative side-effect if nobody has bothered to raise it or make it a sysctl yet...

The post you reference does say beyond 24 it gets pretty rough, and its a known issue that bhyve needs work to scale with more vCores. We'd be hesitant to locally raise that limit here if there is a possible bad impact on the system as well.

 

[jasonsansone]

Looks like in upstream FreeBSD this is still hard-set to 16:

freebsd-src/sys/amd64/include/vmm.h at stable/12 · freebsd/freebsd-src

The FreeBSD src tree publish-only repository. Experimenting with 'simple' pull requests.... - freebsd/freebsd-src

github.com

No idea what changing that does, I would have to assume there is some negative side-effect if nobody has bothered to raise it or make it a sysctl yet...

The post you reference does say beyond 24 it gets pretty rough, and its a known issue that bhyve needs work to scale with more vCores. We'd be hesitant to locally raise that limit here if there is a possible bad impact on the system as well.

Was unaware of the upstream hard code. Thank you. All hypervisors I am familiar with scale poorly as thread count increases (VMWare, HyperV, Proxmox, etc). I wasn’t looking for insanity, just maybe 24 max. Appreciate the response and looking in to it!

 

[jasonsansone]

I have been testing TrueNAS SCALE and noticed the vCore limit from bhyve has been carried over to KVM. I created a suggestion in Jira to hopefully increase the artificial limit that doesnt effect KVM like it does bhyve.

 

[Kris Moore]

I have been testing TrueNAS SCALE and noticed the vCore limit from bhyve has been carried over to KVM. I created a suggestion in Jira to hopefully increase the artificial limit that doesnt effect KVM like it does bhyve.

Seems easy enough to fix. I suspect you'll see that land in SCALE nightly images soon.

 

[jasonsansone]

Should be easy to change, but figured I would try to aid the development process with the ticket. You can't know what users are wanting if no one speaks up!