Broadcom Sucks - Moving Away from ESXi - CORE or SCALE?

[Scharbag]

So, Broadcom sucks - that is clear.

With the cancelation of the free version of ESXi, I am looking at options to migrate back to TrueNAS alone. I am very lucky to have a bunch of spare HPE Gen9 DL380s at my disposal so hardware is not an issue for me.

I have provisions a Windows VM and a few Ubuntu VMs on my existing bare metal CORE system (Dual Xeon, 512GB ECC RAM). I run all my schtuff on Docker and it seems to be working well.

I also have another system (Dual Xeon, 320GB ECC RAM) that I have spun up as my first SCALE system.

My question is - should I stick with CORE or make the move to SCALE for the long term? Bhyve is OK but seems more limited than KVM that comes with SCALE. Are there any stability issues to consider?

Also, I see that SCALE has a lot more Plugins/Apps than CORE does. Given I use Docker on Ubuntu, I am not sure if this is something to worry about but perhaps it is something that I need to consider.

Finally - are there any issues with importing CORE pools onto SCALE - from what I have read, this "should" not be an issue.

Anyway, just wondering which direction I should go given Broadcom is an Evil, Loathsome group of sociopaths... Also, I am kinda done with the complicated setups - I do enough of that at work now that a single box solution does have some attraction for me.

Thanks for all the help.

Cheers,

 

[Scharbag]

Update - Bhyve crashed on me...

May need to rethink this plan o'mine.

 

[danb35]

KVM (which SCALE uses) is reported to be a more stable hypervisor than bhyve. It's certainly more flexible, and forms the basis of at least Proxmox and possibly other hypervisor distros. And really, barring major changes, CORE will be completely EOL within a few years, so you really should be planning on moving to SCALE, if you're going to stick with TrueNAS at all.

Personally, though, if you have significant virtualization needs, I'd recommend a dedicated hypervisor. I've been very happy with Proxmox, but xcp-ng also seems like a good choice. Maybe you'd want to run TrueNAS as a VM under one of those.

 

[Scharbag]

I will likely take a look at Promox. Bhyve has not impressed me at all.

Cheers,

 

[Jailer]

I don't have much experience with hypervisors but I'll second the proxmox recommendation. My experience with it so far has been very positive.

 

[Samuel Tai]

Bhyve is no where near ready for prime time. I only run very light duty loads on it. It's not bad for running Windows desktops or a development container environment.

 

[Scharbag]

Yeah, my 3 days with Bhyve were not great. Flakey and unstable. Even the mount -a results were inconsistent where it would only mount about a third or the directories in each mount point. Then CPU panics, OS crashes etc. Very lame.

I will take a look at Proxmox. Just really sucks as we use VMware at work so I am well versed in it. But we may be moving away from VMware there too as the Broadcom pricing is up 200% and we are nearing a hardware refresh cycle. Anyway, thank you for the input.

Also, I will take a look at SCALE too. Everything seems similar to CORE and likely SCALE will be the long term product as BSD is a PITA compared to Linux. But I have wasted enough time screwing around this weekend!

Cheers,

 

[Samuel Tai]

I do get better bhyve stability if I only use VirtIO devices, instead of emulated devices. For the disk devices, I hard code the sector size to 512. I also find a swap partition within a VM is somewhat mandatory, to deal with bhyve's less than stellar memory management, as the VirtIO memory balloon driver is astonishingly absent.

 

[Scharbag]

I do get better bhyve stability if I only use VirtIO devices, instead of emulated devices. For the disk devices, I hard code the sector size to 512. I also find a swap partition within a VM is somewhat mandatory, to deal with bhyve's less than stellar memory management, as the VirtIO memory balloon driver is astonishingly absent.

That all boils down to "Bhyve is lame" in my mind. :)

 

[sretalla]

That all boils down to "Bhyve is lame" in my mind. :)

I'm sure that @Patrick M. Hausen would disagree with you there... he repeatedly mentions that his non-trivial production environment runs on it with rock-solid reliability... so perhaps it's just a question of how you're set up with it... maybe missing some driver updates or something in the VMs?

 

[Patrick M. Hausen]

Bhyve as in TN CORE 13.0 today is also quite outdated. The nightlies for 13.1 look very promising. I built a TrueNAS CORE/OPNsense hybrid with 2 of 4 network interfaces passed through to the VM guest.

I would be really interested why the VMs are running stably for me but not for everybody else. Linux - Debian and Ubuntu - "just work". Of course one uses VirtIO for all devices.

Windows needs a bit of a "dance" to get the VirtIO drivers into the guest, but like Linux: VirtIO for network and virtual disk and things work. Additionally set the blocksize for the virtual disk device to 512.

TPM emulation and pass through for Windows 11 available now - I have to check if it will make its way into FreeBSD 13.3, though. Topic for the next production users call on Thursday.

Kind regards,
Patrick

 

[Etorix]

Have a look at whether jailmaker on SCALE could replace some or all of your Linux (Ubuntu) VM:

Linux Jails - Experimental Script

"systemd-nspawn may be used to run a command or OS in a light-weight container. In many ways it is similar to chroot, but more powerful since it uses namespaces to fully virtualise the the process tree, IPC, hostname, domain name and, optionally, networking and user databases. It is similar to...

www.truenas.com

 

[ChrisRJ]

I would be really interested why the VMs are running stably for me but not for everybody else. Linux - Debian and Ubuntu - "just work". Of course one uses VirtIO for all devices.

You probably do some small things the right way, perhaps without even realizing it.

As to my personal experience (although on 12.0U8.1) this has been just as abysmal as for many others. Sometimes the virtual console wouldn't work, then a VM suddenly stopped after weeks or wouldn't boot anymore). In contrast my Syncthing jail has been working flawlessly for years.

For VMs I am running a separate XCP-ng machine. ESXi seems to be a bit more mature and stable (esp. XOA still has some rough edges), but overall that is ok.

 

[Samuel Tai]

Yes, avoid the VNC device, and just run a native VNC server in the VM. If forced to use the VNC device (i.e., during installation), hard-code the device port and set it to wait for boot. Don't use the web UI VNC client, but a real VNC client, like TigerVNC. Remove the VNC device afterwards.

Finally, make sure the device IDs increment contiguously from 1000, with the boot disk device using 1000, and any VNC device being last.

The VM UI in Core has various quirks documented only in the Python middleware code. I found the transitional CLI methods for iohyve in FreeNAS 9.x much more straightforward.

 

[RadolBR]

For VMs I am running a separate XCP-ng machine. ESXi seems to be a bit more mature and stable (esp. XOA still has some rough edges), but overall that is ok.

We use ProxMox, but... I don't like it. It works, but i dont like...

I want to move to XCP-ng soons as possible. Because its Xen based, i remember in the college i work for IT there, and we use XenServer for VM, works great.

And i prefer use XenCenter/XCP-ng Center, for me works better than XOA. And for me looks more "enterprise" than ProxMox

 

[Scharbag]

Bhyve as in TN CORE 13.0 today is also quite outdated. The nightlies for 13.1 look very promising. I built a TrueNAS CORE/OPNsense hybrid with 2 of 4 network interfaces passed through to the VM guest.

I would be really interested why the VMs are running stably for me but not for everybody else. Linux - Debian and Ubuntu - "just work". Of course one uses VirtIO for all devices.

Windows needs a bit of a "dance" to get the VirtIO drivers into the guest, but like Linux: VirtIO for network and virtual disk and things work. Additionally set the blocksize for the virtual disk device to 512.

TPM emulation and pass through for Windows 11 available now - I have to check if it will make its way into FreeBSD 13.3, though. Topic for the next production users call on Thursday.

Kind regards,
Patrick

I do not like to have to “dance” to get things working.

All I had running was Ubuntu and docker. No drivers should be required. Using cifs, mount did not complete properly and only some directories in a folder got mounted. Perhaps this was because mount and docker compose were firing at the same time. But if that is the case, I call LAME!! Walk and chew gum damnit.

Coming from an enterprise VMware shop, Bhyve is bush league at best.

Looking forward to trying Proxmox or Xen to get an idea of what is out there.

It would help my power bill to use a single box. But alas, I am back to VMware and Core for now. Here is hoping Broadcom does not murder my system on me…

 

[Patrick M. Hausen]

All I had running was Ubuntu and docker. No drivers should be required.

Ubuntu as a guest runs perfectly fine for me in CORE 13.0-U6.1. VirtIO disk and network, no VNC, serial console only.

Again: I wonder why your experience might be different. Not to blame you for "holding it wrong" but to pinpoint problems.

 

[Samuel Tai]

Perhaps this was because mount and docker compose were firing at the same time. But if that is the case, I call LAME!! Walk and chew gum damnit.

What I do is to create a local Docker volume attached to a CIFS or NFS mount. (NFS is preferred, due to CIFS being single-threaded.) I can then reference the Docker volume in the compose.

From https://docs.docker.com/storage/volumes/:

Code:

docker volume create \
    --driver local \
    --opt type=cifs \
    --opt device=//uxxxxx.your-server.de/backup \
    --opt o=addr=uxxxxx.your-server.de,username=uxxxxxxx,password=*****,file_mode=0777,dir_mode=0777 \
    --name cif-volume

 

[Ericloewe]

Yeah, I find it weird how there are so many complaints about Ubuntu on bhyve (apart from the console resolution, I remember dealing with that one). There must be this one thing that breaks that I just don't use (I have it easy, it's just the one VM I want to keep in the NAS to deal with the docker registry).

 

[danb35]

Ubuntu as a guest runs perfectly fine for me in CORE 13.0-U6.1.

It works reasonably well for me too, aside from the fact that it won't reliably reboot--I have to hard power-off through the TrueNAS UI and then restart it, most of the time. And that the console doesn't work at all, but that's surely a TrueNAS problem rather than a bhyve problem. I don't reboot it very often, so it isn't a huge problem.