Performance issues / Bugs/ slow VM on Scale

[axrusar]

I have setup Scale on this machine:

I have only one VM running Ubuntu server 20.04 on it with a nextcloud docker installation (AIO).

I took a snapshot of the vanilla Ubuntu server. Pre-nextcloud installation:

The VM is configured with 4 threads for CPU, 4Gb of ram.

Now...

i was setting up Nextcloud and had a few issues, so i reverted the aforementioned snapshot quite a few times before i finally got the Nextcloud instance running due to some weird networking issues..
Right after reverting to the snapshot, i got errors like this when booting the VM server again:

File system errors, with very long wait times to get the server running again

I rebooted TrueNAS and the VM was able to start again with no errors.
Restored the snapshot once again... and now the VM would not even start anymore:

But .. i restarted TrueNAS AGAIN, and the VM is able to start once more.

I proceed to do some configuration inside the VM, and ops!!

The VNC connection gets lost, and it wont connect again..
So... Lets restart TrueNAS yet another time... (i already needed to restart 4-5 times to the the VM in a somewhat working state.

Ok... i better use SSH to work on the VM.
I can ping the server... but it wont connect... i open the VNC option again from the TrueNAS GUI, and see that the VM server got stuck again trying to boot up with any other random error.

Ok... restart the VM again...
Now it works.

I am finished installing the Nextcloud Instance on it, great.
Now i go configure and test the Nextcloud installation. I click on the different sections, and things get random now.
I can click on the Photos section, and have to wait 15 seconds for the page to open, looking at the spinning circle.
Or.. all of a sudden, things move kind of fluently (considering i am running on 2 spinner drives), but 2 or 3 clicks away, the server gets stuck again
so in other words.. it is absolutely UNUSABLE for anything.
I am monitoring the VM resources while the instance is taking forever to load anything.. this is how it looks:

Definitely not "loaded" or screaming "I need more resources to run please!".

My question here is....
are you guys really using this as a hypervisor for your VMs?
May be TrueNAS Scale in beta stage? or am i so unlucky i get all these random issues trying to run a simple VM?

The VM has a vanilla Ubuntu server installed from the official ISO.

I will wipe the whole TrueNAS install and use ProxMox instead.
But i am curious how you guys manage to get things working here, in one week of experimenting with it, i can write a full book of bugs, errors and issues i had trying to run the thing..

 

[Samuel Tai]

You experienced this horrible performance because you didn't make your pool large enough. ZFS file system performance drops off drastically above 80% because of fragmentation.

The path to success for block storage

It seems like I haven't written a sticky for awhile, but just in the last week I've had to cover this topic several times. ZFS does two different things very well. One is storage of large sequentially-written files, such as archives, logs, or data files, where the file does not have the middle...

www.truenas.com

Also, TrueNAS is not intended to be a general-purpose hypervisor, like ProxMox. It's optimized for storage and does a little bit of KVM VM action on the side. So, you're using it a bit beyond its intended purpose.

 

[axrusar]

Thank you Samuel.
Very informative and useful post by the grinch.
I am running a small VM as a backup server in another proxmox setup with yet more limited hardware specs than the little PC i showed before, and although the "performance" is not there.. i did not have so many glitches and what not.
Also, i took the time to run some memory tests on the system i was playing with TrueNAS, and found a bad ram stick, so i will definitely be trying again using TrueNAS in the future.

I will leave ZFS alone for now as i understand i will eventually run into performance issues even running a single VM where i will need to store multiple backup snapshots and run over 50% of usage sooner than later, so the graph posted by grinch is a warning telling me i better use something else until i get access to better/bigger hardware.

So is there a somewhat "general" rule of how much extra beef is needed on a system to keep just a couple VMs running in my example, a nextcloud VM with very little traffic (2-3 users), and a secondary backup server taking periodic snapshots of a 100gb shared data folder?
I do not have access to xeons, triple digit ram memory amounts, or 60-70+ % free space hard disks now.
Although the minimum system requirements in the TrueNAS manual is very friendly, how far can one actually go on regular desktop computer hardware?

 

[Samuel Tai]

Most desktops aren't able to run more than 1-2 VMs on an occasional basis, using something like VirtualBox as a Type 2 hypervisor. They don't really have the hardware support for Type 1 hypervisors. Even if they do have the required BIOS bits (VT-d, VT-x, etc.), these features aren't as fully debugged as their server equivalents.

With respect to NextCloud, you may not need to run it in a VM, as this is available as a Kubernetes application on Scale. Also, if you have the space to move the shared data folder to Scale, snapshots aren't full size, as they only consist of pointers to changed blocks. That is, if you're willing to use TrueNAS to its strengths (as a storage platform) as opposed to its weaknesses (as a hypervisor on the hardware you have available), and to alter your use cases, you may find yourself enjoying a much better experience that what you've had so far.

 

[axrusar]

Ok understood. Thank you.
The reason i am running nextcloud in a VM rather than the available plugin in TrueNAS is the other setup options already include the collabora, talk, redis, and other server extensions not available in the vanilla NC installed by the plugin.

It is funny you mentioned VirtualBox. I have truenas core setup on a virtualbox VM in my laptop and it actually runs pretty well, including the nextcloud plugin, much better than the actual baremetal setup i was testing yesterday.