Register for the iXsystems Community to get an ad-free experience and exclusive discounts in our eBay Store.

SOLVED High CPU Usage on Ryzen 1700

Western Digital Drives - The Preferred Drives of FreeNAS and TrueNAS CORE

eljojo

Neophyte
Joined
Mar 1, 2019
Messages
6
Hello everyone, this is my first post in the forum, please let me know if I'm posting in the incorrect format.

I am new to Freenas. So far been running pretty nice, but I noticed my machine has a very high cpu usage coming from the `middlewared` process.
Looking at its logs, I see that my guest os (nixos - recent linux kernel) seems to be sending a specific instruction which is making middlewared log like crazy.
It's essentially using 100% of a single core at all times, even if the guest is idle. Something weird I noticed is that the guest is also using an extra full core of the host, even if within the guest there's basically no load.

Here's the output of middlewared's logs when shutting down the guest:

Code:
[2019/03/02 02:25:50] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 1
[2019/03/02 02:25:50] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 1
[2019/03/02 02:25:51] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 1
[2019/03/02 02:25:51] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 1
[2019/03/02 02:25:52] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 1
[2019/03/02 02:25:52] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 1
[2019/03/02 02:25:53] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 1
[2019/03/02 02:25:53] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 1
[2019/03/02 02:25:54] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 1
[2019/03/02 02:25:54] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 1
[2019/03/02 02:25:55] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 1
[2019/03/02 02:25:55] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 1
[2019/03/02 02:25:56] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 9
[2019/03/02 02:25:56] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 9
[2019/03/02 02:25:56] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0x400) on vcpu 9
[2019/03/02 02:25:56] (DEBUG) VMService.run():287 - nixos: wrmsr to register 0xc0011020(0) on vcpu 9
[2019/03/02 02:27:56] (DEBUG) VMService.run():287 - nixos: Unhandled ps2 keyboard command 0xf6
[2019/03/02 02:27:56] (DEBUG) VMService.run():287 - nixos: fbuf frame buffer base: 0xa42e00000 [sz 16777216]
[2019/03/02 02:27:56] (DEBUG) VMService.run():287 - nixos: Waiting for vnc client...
[2019/03/02 02:27:56] (INFO) VMService.run():304 - ===> Powered off VM: nixos ID: 5 BHYVE_CODE: 1
[2019/03/02 02:27:56] (ERROR) VMService.running():438 - ===> VMM nixos is running without bhyve process.
[2019/03/02 02:27:56] (DEBUG) VMService.__teardown_guest_vmemory():339 - ===> Give back guest memory to ARC: 8589934592
[2019/03/02 02:27:56] (WARNING) VMService.destroy_vm():319 - ===> Destroying VM: nixos ID: 5 BHYVE_CODE: 1
[2019/03/02 02:27:56] (DEBUG) VMService.kill_bhyve_web():406 - ==> Killing WEBVNC: 4369


If I scroll higher down the file, I can tell that `nixos: wrmsr to register 0xc0011020(0)` is definitely happening VERY often, seems to reach the 10M limit of logrotation within seconds:
Code:
ls -lh /var/log/
-rw-r--r--  1 root     wheel     3.5M Mar  1 21:27 middlewared.log
-rw-r--r--  1 root     wheel      10M Mar  1 21:24 middlewared.log.1
-rw-r--r--  1 root     wheel      10M Mar  1 21:24 middlewared.log.2
-rw-r--r--  1 root     wheel      10M Mar  1 21:23 middlewared.log.3
-rw-r--r--  1 root     wheel      10M Mar  1 21:23 middlewared.log.4
-rw-r--r--  1 root     wheel      10M Mar  1 21:23 middlewared.log.5


This is not happening when the guest is off.
I am running FreeNAS-11.2-U2.1, but this also happened with FreeNAS-11.2-U2

Any ideas?

Edit: I've opened a patch on freebsd for bhyve, maybe this will help: https://github.com/freebsd/freebsd/pull/392
 
Last edited:
D

dlavigne

Guest
Please also create a report at redmine.ixsystems.com that references your patch and post the issue number here.
 
Top