Need help with Virtualization topology (VM? docker? Jail? something else?)

[Stolen_Walnut]

I currently have a setup that I'm unhappy with and would like to migrate to a better setup, but am struggling to get the virtualization right.

Current situation	Try to migrate to
Old NAS Pool of HDDs dataset for misc dataset for Gitlab Main Server NFS mounts from misc Gitlab NFS mounts from Old NAS New NAS Pool of NVMes	Old NAS Pool of HDDs Repurposed for replications Main Server NFS mounts from misc New NAS Pool of NVMes dataset for misc dataset for Gitlab Gitlab Using Gitlab dataset directly (not over the network) for OS & data

The thought here was that it felt wrong for Gitlab to be sending so much traffic over the network for it's general operations. It seems like there should be a straight forward way to virtualize Gitlab & point it at its dataset on the NVMe pool, but that's alluding me.

Fumbling my way through some of the setup, I found myself in this situation:

• New NAS OS: TrueNAS
  • Pool of NVMes
    • dataset for misc
    • dataset for Gitlab
      • VM image
  • VM: Gitlab
    • Point to VM image for OS & data

Which scratches the itch of getting Gitlab to use local storage, but seems to have come with a few cons:

• VM image saved onto dataset as one big file -- all or nothing for data loss / replication (or is this a bad assumption?)
• Seems like I had to set a max file size for VM image which would need to be manually monitored and expanded as needed

A Bonus for me here as well would be to be able to grep through files straight on the dataset, which it looks like I loose with this approach
Am I over looking a way to just pass a dataset to a VM to put its files on directly / expand as needed?


I was originally thinking I'd be using jails, but it looks like that was removed from the GUI (never got around to actually trying that out, so not sure if that was what I wanted to begin with)

Thought about maybe run a Linux distro with Gitlab on it and throwing TrueNAS in docker maybe, but it looks like that's not supported.
A hypervisor running a VM of TrueNAS and a VM of Gitlab brings networking back into the mix, albeit virtualized.

Hrm.

Any help would be much appreciated, and please call out if I've made any bad assumptions or gotten things wrong.

Thanks!

Show : OS/HW

"New NAS"
OS Version: Currently TrueNAS-SCALE-22.12.0, but flexible if something makes more sense
Motherboard: MSI MEG Z690I UNIFY
CPU: Intel Core i5-12500
RAM: G.Skill Ripjaws S5 64GB (2 x 32GB) DDR5-5200
Drives

• Pools
  • boot-pool: SATA SSD - Samsung 870 EVO Series 2.5" 250GB
  • np:
    • mirror-0
      • NVMe: Inland Premium 2TB SSD M.2
      • NVMe: Crucial - P3 2TB Internal SSD
      • Note: I'm aware these are different sizes... One will be replaced with identical size before continuing

Hard disk controllers

• 3x integrated/on-board SSD/NVMe controllers:
  • lspci -nnn | grep -E "0106|0108"
    • 00:17.0 SATA controller [0106]: Intel Corporation Device [8086:7ae2] (rev 11)
    • 09:00.0 Non-Volatile memory controller [0108]: Micron/Crucial Technology Device [c0a9:540a] (rev 01)
    • 13:00.0 Non-Volatile memory controller [0108]: Phison Electronics Corporation PS5013 E13 NVMe Controller [1987:5013] (rev 01)

Network cards

• 2x:
  • lspci -nnn | grep 0200
    • 08:00.0 Ethernet controller [0200]: Aquantia Corp. AQC107 NBase-T/IEEE 802.3bz Ethernet Controller [AQtion] [1d6a:07b1] (rev 02)
    • 0b:00.0 Ethernet controller [0200]: Intel Corporation Ethernet Controller I225-V [8086:15f3] (rev 03)

 

[Whattteva]

I was originally thinking I'd be using jails, but it looks like that was removed from the GUI (never got around to actually trying that out, so not sure if that was what I wanted to begin with)

Jails are still very much in there. I feel like you're using TrueNAS SCALE instead of CORE if you don't see jails.

Thought about maybe run a Linux distro with Gitlab on it and throwing TrueNAS in docker maybe, but it looks like that's not supported.
A hypervisor running a VM of TrueNAS and a VM of Gitlab brings networking back into the mix, albeit virtualized.

I run TrueNAS both bare metal and virtualized (see signature). Virtualization is possible, but it needs to be done in a certain specialized "sauce" if you want to do it successfully. See this for more details.

 

[trueJack]

Just from looking through docs online, one of your options might be running Gitlab on Kubernetes:

https://about.gitlab.com/solutions/kubernetes/

Truenas Scale has Kubernetes and enables Kubernetes to connect to ZFS storage:

TrueNAS enables Container Storage and Kubernetes

The Democratic CSI driver integrates ZFS and TrueNAS into the Kubernetes environment and other container management platforms.

www.truenas.com

Mind you, I have no practical experience with this.

Also it looks like you want ZFS + virtualization or containers. Perhaps you are better off with Proxmox, in which case you could still use ZFS, but also Linux containers that could access ZFS. You could still use Truenas for your backup NAS.

 

[Ericloewe]

VM image saved onto dataset as one big file -- all or nothing for data loss / replication (or is this a bad assumption?)

You would probably want to use a zvol instead.

The thought here was that it felt wrong for Gitlab to be sending so much traffic over the network for it's general operations. It seems like there should be a straight forward way to virtualize Gitlab & point it at its dataset on the NVMe pool, but that's alluding eluding me.

Well, Gitlab can get ridiculously complicated. Let's assume this is along the lines of the oddly-named omnibus installation and everything ends up on local (to the Gitlab host) storage. So, all you really need is to set up a zvol, set the quota to whatever you need/want for Gitlab's data, set it up as a virtual disk and mount it in the VM.
If you're doing OS-level virtualization with Jails, Docker and the like, replace "zvol" with "normal dataset" and "virtual disk" with "storage mounted from the host".

Am I over looking a way to just pass a dataset to a VM to put its files on directly / expand as needed?

You can never do that without explicitly sharing it with something like SMB or NFS because that's how the abstractions work. Sure, there are some weird edge case solutions, but I wouldn't recommend them.
You can do that with OS-level virtualization, though. With the caveat that Docker makes it needlessly confusing and convoluted. Super simple on Jails. No idea about Kubernetes, but I expect it to be rather more complex than simple.

 

[Stolen_Walnut]

Thanks for the replies.

On the topic of zvols (ultimately went with this)...

• Static mounting for recovery
  • I originally stayed away from zvols because I thought you couldn't mount them from the host and explore the file contents, but realized later that was a bad assumption. It threw me off since it's not defaulted to allow you to do that but there's some option you can set to be able to expose the partitions in the zvol & then you can mount them.... don't have notes handy on what that was, but it's there & you can turn it on after the fact. Definitely wanted this in the case that my vm gets corrupt for w/e reason & I couldn't boot into the vm
• Dynamic mouning for grepping purposes
  • Unfortunately, you can't mount the zvols on the host and have them dynamically change as the vm is adjusting files, but I just pivoted to exploring the files straight from the vm which is snappy enough for me.

I ultimately landed in this configuration:

When i set this up I for sure wanted to separate swap out as a separate zvol, and just ended up splitting up boot & root as well... Looking back, if I did it again, I'd probably keep boot & root together mostly for the sake of being able to snapshot a single entity & not have to track which snapshot is paired with which. Small potatoes though.

Networking

• Since the share dataset is served via a 10G connection, I figured I'd macvtap that interfaces to give 10G capability to the VM as well..
• Ultimately just ended up giving the VM it's own 1G connection...
  • Don't have to worry about the setup/tear down of the tap affecting the 10G link
  • May re-evaluate if I'm finding myself abusing gitlab for more than just source code... lol Nothing like forcing myself to use my tools right by setting up minor annoyances if I use them wrong :)

Still need to set up replication tasks

So far pretty happy with it.

Critiques welcome.

Thanks