SOLVED How to Control bhyve I/O Sync?

[yottabit]

Has anyone figured out the default I/O sync behavior for a bhyve VM created in FreeNAS 11?

I'm having a problem where the VM (Windows 10 in this case) is under reasonable disk I/O load, the entire server becomes fairly unresponsive when any disk I/O is involved.

Host config:

• 5 disks are in a long-time RAID-Z2, no other problems with jails, SMB, etc.
• 16/32 Xeon cores/threads
• 64 GB RAM
• No host swap being used

Guest config:

• Windows 10 x64
• Fresh install, and updated
• 80 GB zvol for disk, on the Z2 vol mentioned above
• 12 GB RAM allocated

Where is the bhyve I/O sync/async parameter enumerated in the FreeNAS configs? Anyone know the default?

While the host becomes unresponsive every once in a while, the guest VM is so unresponsive I can't get VNC to show any activity, and can't get RDP to connect. I usually use RDP and it will actually terminate the session citing the host has stopped responding. :-/

And during these periods, the host is showing bhyve at 800%+ utilization, mainly with state kqread.

 

[yottabit]

Here are the VM stats if that helps.

root@nas1:~ # bhyvectl --vm=lytro1 --get-stats
vcpu0 stats:
number of times wrmsr was intercepted 12
number of monitor trap exits 0
number of times pause was intercepted 5121472
vm exits due to interrupt window opening 51976074
vm exits due to nmi window opening 0
number of times in/out was intercepted 2039653
number of times cpuid was intercepted 1914
vm exits due to nested page fault 127201
vm exits for instruction emulation 61160238
number of vm exits for unknown reason 0
number of times astpending at exit 24466
number of times idle requested at exit 0
number of vm exits handled in userspace 68520446
number of times rendezvous pending at exit 9
number of vm exits due to exceptions 0
number of NMIs delivered to vcpu 0
number of ExtINTs delivered to vcpu 0
Resident memory 25448448
Wired memory 0
vcpu total runtime 1485622999684
EOI without any in-service interrupt 0
error interrupts generated by vlapic 0
timer interrupts generated by vlapic 0
corrected machine check interrupts generated by vlapic 0
lvts triggered[0] 0
lvts triggered[1] 0
lvts triggered[2] 0
lvts triggered[3] 0
lvts triggered[4] 0
lvts triggered[5] 0
lvts triggered[6] 0
ipis sent to vcpu[0] 1349295
ipis sent to vcpu[1] 1375940
ipis sent to vcpu[2] 1371392
ipis sent to vcpu[3] 1369840
ipis sent to vcpu[4] 1372610
ipis sent to vcpu[5] 1372062
ipis sent to vcpu[6] 1370431
ipis sent to vcpu[7] 1370624
ipis sent to vcpu[8] 0
ipis sent to vcpu[9] 0
ipis sent to vcpu[10] 0
ipis sent to vcpu[11] 0
ipis sent to vcpu[12] 0
ipis sent to vcpu[13] 0
ipis sent to vcpu[14] 0
ipis sent to vcpu[15] 0
number of ticks vcpu was idle 49789
vcpu migration across host cpus 2389
total number of vm exits 361132122
vm exits due to external interrupt 75242704
Number of vpid invalidations saved 494
Number of vpid invalidations done 1895
number of times hlt was intercepted 71860
number of times %cr access was intercepted 165390058
number of times rdmsr was intercepted 931


 

[yottabit]

I'm pretty sure it was related to the e1000 network driver. I've switched the virtual network over to virtio and installed the virtio network driver from Red Hat, and so far so good. It has been under 1600% load now for over 15 minutes, which is a record. The host and the guest are responding as expected.

My clue with in the Windows event log. There were a couple entries about the network driver failing and being re-enabled before there were no more entries (system hang). I assume the vCPUs stayed at nearly 100% once this happened because the guest kernel couldn't issue a pause instruction as it had crashed.