SOLVED Step by step: All-in-one Recovery Backup Appliance with FreeNAS-11.1u6, bhyve and Veeam

[blanchet]

Abstract
This is a step-by-step guide to create an all-in-one recovery backup appliance with FreeNAS-11.1U6, bhyve and Veeam.

Context

• My virtual machines are hosted on a VMware cluster.
  
• My virtual machines are protected by Veeam Backup & Replication 9.5.
  
• The backup files *.vbk are stored on an off-site FreeNAS server.
  
• For convenience and performance reasons, Veeam runs in a virtual machine within the same VMware cluster and datastore.
  
• In case of complete datastore destruction, Veeam dies. Therefore it must be reinstalled from scratch then it can restore the backup.
  
• Reinstalling Windows+Veeam from zero takes at minimum 1 hour, so it hurts your Recovery Time Objective (RTO)
• My workstation has only a small bandwidth to the datacenter, so it cannot host a Veeam recovery console.

Goal

• Create a virtual machine that runs directly on the off-site FreeNAS server.
  
• Install Veeam inside to have a 2nd instance of Veeam.
  
• Only Veeam Backup Free Edition is required, because it is just for restoring, not for scheduled backups.
• This vm is totally independent from VMware and its datastore.

Pitfalls

• the virtio storage driver for Windows crashes bhyve and FreeNAS11.1u6
• Veeam Backup needs Microsoft SQL Server Express 2012 that supports only
  • single socket with 4 cores.
    
  • 512 bytes sectors. (I have also tried 4K sectors but it fails)
• Use only IP address in Veeam to avoid DNS dependencies

Step-by-step guide

• To overcome license limits of SQL Server Express 2012, configure bhyve to show cores instead of sockets inside virtual machines:
  • in System | Tunables | Add Tunable
    
    Indicate the number of core your CPU have (in my case 10)
    Variable: hw.vmm.topology.cores_per_package
Value: 10
Type: Loader
Comment: For SQL Server Express in bhyve
Enabled: [x]
    
    If you have hyper-threading cores, specify also
    Variable: hw.vmm.topology.threads_per_core
Value: 2
Type: Loader
Comment: For SQL Server Express in bhyve
Enabled: [x]
    
    Reboot FreeNAS to apply the new settings.
  
  
• To be compatible with SQL Server Express 2012, create a virtual disk with 512-bytes sectors:
  • zfs create -V 80G -o volblocksize=512 tank1/veeam_backup_disk0
    SQL Server Express fails with any other setup (for example volblocksize=4096 or raw file instead of zvol)

• Create a bhyve virtual machines
  • In VMs | Add VM
VM Type: Virtual Machine
Name: veeam_backup
Description: Veeam recovery console
Virtual CPUs: 4
Memory Size (MiB): 8192
Boot Method: UEFI
Autostart: [x]
  
  
• Add devices
  1. NIC:
     In my case the physical inteface is cxl0, and I have chosen a very easy-to-remember MAC address
     VM: veeam_backup
Type: Network Interface
Adapter Type: VirtIO
NIC to attach: cxl0
MAC Address: aa:aa:aa:aa:aa:aa
     
     
  2. VNC:
     For the very beginning, we select no password
     VM: veeam_backup
Type: VNC
Resolution: 1280x1024
VNC port: 5901
Bind to: YOUR_IP_ADDRESS
Wait to boot: [ ]
Password:
VNC Web: [x]
     
     
  3. DISK:
     You must specify a zvol with 512-byte sector in AHCI.
     It fails with VirtIO, raw file or 4K sectors.
     VM: veeam_backup
Type: Disk
ZVol : tank1/veeam_backup_disk0
Mode: AHCI
Disk sectorsize: 512
     
     
  4. CDROM
     VM: veeam_backup
Type: CD-ROM
CD-ROM (ISO) : /mnt/tank1/iso/windows_server_2012r2.iso
     
     
• Power-On the virtual machine
• connect to the VNC Web console
• Install Windows as usual
  
  
• Install the virtio driver for the network
  • shutdown the VM, because the CD-ROM cannot be reconfigured when the vm is running
  • Download the virtio-win iso
  • Connect ISO to the VM
  • Power on the VM
  • In Windows Device Manager,
    • select the unrecognized device
    • right-click Update Driver Software
      
    • browse the CD-ROM to D:\NetKVM\2k12R2\amd64
    • install the driver
    • Now you have a network connection
    • Configure the network to use a static IP address.
      
    • Specify Google DNS 8.8.8.8 to not depend on your DNS.
      
• Install Veeam Backup
  • shutdown the VM, because the CD-ROM cannot be reconfigured when the vm is running
  • Connect VeeamBackup&Replication_9.5.0.1922.Update3a.iso to the CD-ROM.
    
  • Install Veeam with default settings (click Next everytime)
  • Veeam FREE Edition is sufficient, because it is used only for restoration
  • Setup Veeam
    • Switch to Advanced view
    • Configure Backup Repository with IP address only
    • Connect ESXi hosts with IP address only
    • Do not connect to vCenter, because vCenter may be down when you need this recovery console
    • Check that that you can restore a vm.
  • 
    
• Finalize the installation
  Setup a password for VNC
  VM : veeam_backup
Type: VNC
Resolution: 1280x1024
VNC port: 5901
Bind to: YOUR_IP_ADDRESS
Wait to boot: [ ]
Password: ********
VNC Web: [x]
  
  
• Power off the VM and snapshot it
  zfs snap tank1/veeam_backup_disk0@veeam_setup_end

Now, in case of problem, you can restore easily any virtual machines, even Veeam and vCenter.

 

[kdragon75]

Thank you for the FANTASTIC write up!
I didn't think you needed a Veeam license per instance, only per vmware cluster total sockets.
If you would be willing to go one step further and place feature requests for core/socket/hyper threading options and if it's not already an option, the ability to set zvol block size in the VM creation wizard.

 

[blanchet]

For a recovery console, it is safer to depend on the minimal number of components, therefore I prefer using only Veeam Backup & Replication Free Edition to avoid any license file.
Now, I am waiting for the next version of FreeNAS to check if virtio-storage works finally with Windows guest, because it is quite slow in AHCI mode.

 

[blanchet]

You can easily extend this guide to build a Full Backup Appliance instead of a simple Recovery Backup Appliance, because Veeam Backup & Replication Free Edition can also backup an unlimited number of VM.

Limitations of the free version:

• no incremental backup, only full backup (VeeamZIP).
  
• no scheduled backup from the GUI, only from a PowerShell script

Despite this limitations, VBR Free is a viable and decent solution for small clusters. Nevertheless, for larger cluster, the Standard or Enteprise version is preferable.

A last word for home lab users:

• Veeam does not support VMware ESXi Free because the API is disabled by VMware. To workaround this limitation, you can switch to Microsoft Hyper-V Server Free, or install Veeam Endpoint Backup FREE in each guest. It is less convenient but it work with any hypervisor, including Xen and bhyve.

 

[blanchet]

Since January 2019, there is a new Community Edition of Veeam Backup
This version is basically the standard version but with a limit of 10 VMs.

It is a great improvement compare to the previous free version of Veeam
which had neither scheduler nor incremental backup.

Detailed article about the CE version