Which hypervisor does TrueNAS SCALE use?

[Akhanaton]

I'm not familiar with open source hypervisors on BSD (TrueNAS Core) except for VirtualBox on my mac, but I am very familiar with hypervisors on Linux (TrueNAS Scale). I've read a lot of posts about converting qcow2 (kvm) and vmdk (VMWare) virtual machine images for TrueNAS SCALE. I would like to know specifically which hypervisor is in TrueNAS SCALE. It pretty much has to be kvm or xen. There's no other choices I'm aware of. I know both SoftLayer (IBM Cloud) and AWS started with xen but later moved to their own versions of kvm and selected a new name for their images - AMI for AWS and FVD for IBM Cloud. They have the kvm source code so it's no problem to change the name of the image but it's pretty much a qcow2. I don't mind doing image conversions but I'd like to know the basis. Please let me know which hypervisor is included in TrueNAS SCALE.

Thanks,
Charles McCane CCIE#5163 AWS-SAA VCP-DC

 

[danb35]

kvm

 

[Akhanaton]

Thank you, sir. Most appreciated. So the qcow conversion mentioned in other threads is needed because iXsystems forked kvm and re-coded some parts of it. All good. I will perform some image conversions and follow other threads' instructions to build vm's from other platforms' virtual machine images. This is critical for migrations. To clear up any confusion, vm's are entirely contained within virtual machine images which can be copied as a file generally onto any storage or filesystem including zfs where they will reside as files and may grow in size as the vm requires more space for whatever OS/app is in the vm. I will assume kvm device driver mappings & capabilities however, any reference material on iXsystem's kvm fork insofar as it differs from kvm would be great.

 

[danb35]

I'm not aware of iX' having forked or altered kvm--I won't say it didn't happen, but I'm not aware of it. But AFAIK, SCALE uses zvols for VM virtual disks, not .qcow2 files (putting a CoW virtual system on top of a CoW real system would seem kind of silly, and possibly dangerous).

Keep in mind, though, that any flavor of TrueNAS expects you to do pretty much anything through the GUI. If you're planning on creating KVM VMs at the command line, you're likely to confuse the TrueNAS middleware, with consequences that may be harmful to your data. But beyond that, I probably can't give you much help, as (1) I barely use SCALE at all, and (2) I don't really use TrueNAS for virtualization--I have a Proxmox cluster for that.

 

[Akhanaton]

As a former Sun SE, I've been doing zfs since Solaris 10 in 2005. Super glad the open source community has adopted it for Linux, and maybe a little bit surprised Larry Ellison@Oracle allowed it. But it's just a file system and virtual machine images are just files. Of course you can create a relatively small virtual machine image to instantiate an OS and from that OS you can connect outbound to volumes outside the vm.

On a side note, I remain impressed by the dedication to IT shown by those who run large private labs. In the past, I've dedicated a room to lab that ran enough equipment to generate thousands of watts in continuous energy usage. Now down to 200 watts and the lab is in a closet with a bathroom fan blowing air into the attic. I miss my SPARC servers....*not*. For those who are into video content creation and need fast networking, take a look at the 4 port 10G switch from Microtik for $150. I still have too many 10G connected servers to retire my Arista switch (those servers remain off unless needed for proof-of-concepts (PoC) for work. Note that Microtik switches have horrible firmware so best to use them as an unmanaged switch with a single VLAN, no configuration needed. I have asked Microtik to move towards a white box switch onie-based boot loader but no luck so far.

 

[danb35]

But it's just a file system and virtual machine images are just files.

No, they aren't, except in the general Unix-y sense in which everything is a file. But if you've been doing zfs for 17 years, I don't need to tell you that a zvol is a virtual block device, not a file in the normal sense of the word.

 

[Akhanaton]

I am not here to argue with a Hall of Famer who has 13,139 messages on this forum. You are correct that zvol is a virtual block device. I am correct that a virtual machine image is a file. I didn't go into the "secret sauce" that iXsystems may have used with kvm to make zfs volumes available to a vm that was instantiated from a virtual machine image. The instantiation itself can be as simple as the following CLI:

kvm -hda cloudera-quickstart-vm-4.4.0-1-kvm.qcow2 -net nic -net user -m 512

From a couple of years of experience on RedHat OSP (OpenStack Platform) at Verizon, I can report that kvm instances are essentially instantiated in a managed form using a kvm CLI similar to the above but with a much longer string of options that are orchestrated by OpenStack Horizon. Externally it doesn't appear to be all that complex but once you get into instantiating and managing kvm vm's @scale, it's quite the scary monster.

 

[Akhanaton]

Also note that the concept of virtual block devices originated with a 3rd party add-on for Solaris called Veritas Volume Manager. I took the course, got the cert long ago....

 

[Patrick M. Hausen]

iXsystems may have used with kvm to make zfs volumes available to a vm that was instantiated from a virtual machine image

On all TrueNAS platforms - CORE/FreeBSD and SCALE/Debian - there are no virtual machine image files. The virtual disks are ZVOLs.

 

[danb35]

I am correct that a virtual machine image is a file.

If by "virtual machine image" you mean "the thing the running virtual machine sees as a disk", no, you are not--it's a zvol, not a file. There is no file in a filesystem that serves as a storage device for the virtual machine. There is no file in a filesystem from which such a storage device is generated. It's all zvols, and only zvols.

Also note that the concept of virtual block devices originated with a 3rd party add-on for Solaris called Veritas Volume Manager. I took the course, got the cert long ago....

Um, go you, I guess. I'm not sure how relevant that is to this discussion.

I don't doubt that you know a great deal more than I about Solaris, and possibly even ZFS--but I think your knowledge is leading you to make incorrect assumptions about important subjects here.

 

[Akhanaton]

I've migrated thousands of vm's that use files as volumes. In most cases it's a single file that represents a volume to esxi or kvm. All the block mapping is virtual. Once the virtual disks are unmounted, they can be copied as files, moved to another server, and re-mounted as virtual disks. It works the same in all cloud environments I'm aware of. You can alternatively choose keep the volumes mounted and copy everything out of them using the OS but nobody does that. The world has moved to virtual disk images. Containers are images as well. I'll play around with zfs on Ubuntu to see if I can create a zvol, un-mount it, copy the image file or files that make up the zvol to another server, and re-mount them there. Should be super simple. On my TrueNAS SCALE I can do a "zfs list" and get a long list of what appear to be volumes stored as files in directories. When I go to the directory and list the files there, it looks like there's a namespace mask that prevents me from seeing the files. Interesting.

 

[danb35]

copy the image file or files that make up the zvol to another server

For as much experience as you say you have with ZFS, you don't seem to know very much about it. For the nth time, zvols aren't files.

When I go to the directory and list the files there, it looks like there's a namespace mask that prevents me from seeing the files.

...because they aren't files. As I keep telling you.

 

[Akhanaton]

I have experience with zfs on Solaris. It's evolved since then. I don't think the concepts have changed but I could be wrong. Since you've positioned yourself as the ultimate zfs expert, why don't you tell me what I'm seeing when I do a zfs list and see a bunch of volume names in directories within the Linux file system.

 

[danb35]

why don't you tell me what I'm seeing when I do a zfs list and see a bunch of volume names in directories within the Linux file system.

Sorry, my crystal ball is out of service, so I don't know what you're seeing. But if what you're seeing looks like tank/dan_backup/pbs/zvol or tank/vms/pihole-kkhpg5 in the list below, they're exactly what I've been saying: zvols.

Code:

root@cbnas[~]# zfs list        
NAME                                                         USED  AVAIL     REFER  MOUNTPOINT
...
tank/dan_backup/pbs                                         4.09T  9.22T       96K  /mnt/tank/dan_backup/pbs
tank/dan_backup/pbs/zvol                                    4.09T  11.3T     1.88T  -
...
tank/vms                                                    10.2G  9.22T       96K  /mnt/tank/vms
tank/vms/pihole-kkhpg5                                      10.2G  9.23T     3.46G  -

 

[Patrick M. Hausen]

I do a zfs list and see a bunch of volume names

What you see are volumes. Not files. Block devices. They have been in ZFS from the beginning. The fact that ESXi uses files on its own VMFS file system does not change the fact that TrueNAS doesn't. TrueNAS uses block devices for VM virtual disks.

https://docs.oracle.com/cd/E19253-01/819-5461/gaypf/index.html

 

[freqlabs]

We haven't changed anything in KVM, and we haven't changed anything in libvirt for that matter either. Our UI only allows VMs to be backed by zvols, that's why you have to do a conversion from whatever file format you have to the raw disk format of a zvol. There's nothing special about it.