Possibly bug in VM performance found, I need someone to reproduce

JerRatt

Dabbler
Joined
May 17, 2022
Messages
14
I've been dealing with a strange issue and I think I've pinpointed at least a large part of the cause.

Running TrueNAS Scale 22.02.2, any Windows 10 VM that I load is extremely slow with basic things, such as opening File Explorer or Computer Management. File Explorer can take 5+ seconds to load and even longer to fill in the window. Computer Management can take 30+ seconds to appear. Moving around in anything just seems slow as can be, almost like a mixture of a slow disk speed and delayed video.

This is with Windows 10 using VirtIO drivers, with the disk and nic assigned to the VM's using VirtIO. During this, the VM's show that it is hardly using any CPU, memory, or disk activity at all.

But I was able to speed up the VM entirely, by 400% or more, by simply setting TrueNAS's local apps to use a pool. I had reinstalled TrueNAS and hadn't yet assigned the local apps to a pool at all, and the VM's I loaded were slow. The moment I add the local apps to use a pool, any pool at all, the VM's speed up exponentially. I don't even have to reboot the VM's after I set the pool on the local apps, they just speed up right away. Same thing happens in reverse, if I unset the local apps from using a pool, the VM's go back to being extremely slow/laggy.

Can anyone give this a try and see if they can reproduce it? Take a new or existing Windows 10 Pro VM that's working fine, and unset your TrueNAS local apps from using a pool at all. It should slow down the VM's, test it by doing things like clicking the start button, opening File Explorer, opening Computer Management, etc. It should feel laggy or like it's just waiting for the disk to respond. Time it between the clicks, then re-set the local apps to use a pool, and try again to see that it's much faster.

Specs:

  • AMD Threadripper 1920X
  • ASRock X399 Taichi
  • 128GB (8x16GB) Crucial CT8G4WFD824A Unbuffered ECC
  • AVAGO/LSI 9400i-8i SAS3408 12Gbps HBA Adapter
  • Supermicro BPN-SAS3-743A 8-Port SAS3/SAS2/SATA 12Gbps Backplane
  • 8 x Seagate Exos X18 18TB HDD ST18000NM004J SAS 12Gbps 512e/4Kn (RAIDZ2 Array, Avago/LSI Controller)
  • 2 x Crucial 120GB SSD (Boot Mirror, Onboard Ports)
  • 2 x Crucial 1TB SSD (VM Mirror 1, Onboard Ports)
  • 2 x Western Digital 960GB NVME (VM Mirror 2, Onboard Ports)
  • Supermicro 4U case w/2000watt Redundant Power Supply, on a large APC data center battery system and conditioner
 

brando56894

Wizard
Joined
Feb 15, 2014
Messages
1,537
File Explorer can take 5+ seconds to load and even longer to fill in the window. Computer Management can take 30+ seconds to appear.
Not that this really helps you, but that's been my experience for years on and off with Windows running natively on an NVME drive, and it always confuses the hell out of me.
 

JerRatt

Dabbler
Joined
May 17, 2022
Messages
14
Not that this really helps you, but that's been my experience for years on and off with Windows running natively on an NVME drive, and it always confuses the hell out of me.

I actually know that issue you're speaking about, but it's not that one specifically going on here.

This one happens whether VMs are on NVME, SATA SSD, or even a 12 disk pool of HDDs.

It's something with k3d/k3s service freaking out if the apps aren't set to a pool, somehow affecting VM speeds only within the VM itself.
 
Top