GitLab Agent and Auto-DevOps

[jct]

I think the final Tetris block just dropped, and fits satisfyingly well? Disappoint me if I'm wrong.

When SCALE cut off outside access to the K8s API, my GitLab Runners all ran into a wall. Then GitLab 14.5 brought their GitLab Agent down to the free tier, but only for push-based deployments (and with a ton of janky quick-pivot bugs and the documentation being slow to catch up). And GitLab 14.10 simplified/standardized GitLab Agent installation down to a dead-simple tiny Helm chart. All the while, you could see internal debate and lobbying going on for what just happened today:

Pull-based GitOps moving to GitLab Free tier​

If I understand this correctly, this now looks like something iX could actually support in short order and with fairly little effort. I offer my help, to the extent that it helps.

So… GitLab Agent is a lightweight little thing. You can and should run as many instances of it in one cluster as needed for separate RBACs.

I love what's been happening with SCALE's official app catalog lately, and while clearly it should always stay lean/mean/clean, I do humbly believe that an official GitLab Agent app would be an uncommonly superb fit and could model good behaviors and appeal to worthwhile markets.

Simply put: with such an app, a SCALE admin could deploy an instance of GitLab Agent in its own namespace. That Agent could safely add and remove revision-controlled Kubernetes manifests from a GitLab project. Those manifests would be kept to within the namespace. If additional restrictions are needed (e.g. use only the supplied storage or network classes) then those policies could be enforced by PSA.

One such app instance could be for a staging environment. One for review apps for Team A. One for review apps for Team B. Maybe even a production environment, depending on what production means in your team.

What say you: should this work? Is there anything I could be doing to help move this forward? Would it even be considered for the official catalog?

See also my NAS-114107. I'll be happy to work on this — unless that would slow down someone else for whom it's just better that they run point. It sure seemed like Docker Compose support went from plausible breakthrough to highlighted feature in magnificently short order. It seems to me that this one's actually much easier?

 

[morganL]

Can the gitlab agent run its own docker container?
Can it run on an external system.. server or laptop?

 

[jct]

Can the gitlab agent run its own docker container?
Can it run on an external system.. server or laptop?

Sorry; I'll try to set some context and hope that I'm adequately addressing your question.

Assume that you're starting with someone who already has a GitLab.com account, or who runs a self-hosted GitLab instance through other means. (i.e. this isn't the equivalent of the former GitLab plugin.)

That person can fire up a GitLab Agent container anywhere (GKE, homelab, TrueNAS SCALE) including behind NAT. As of last month: preferably through its Helm chart. No port forwarding, no special credentials. You just give it the URL to your GitLab instance and a pre-generated token.

The Agent then reaches out to the server, and maintains two-way eye contact in case either one needs the other's attention.

With the push-based method (made free months ago but swamped with major acknowledged but poorly-documented bugs on free or self-hosted systems) you could use the Agent's CI/CD tunnel to contact a GitLab Runner and process a workflow.

With the pull-based method (made free yesterday, presumably to be delivered Sunday) it simply reads manifests from your project repo and keeps its namespace in sync with those.

It might eventually be helpful to have a GitLab Runner app for builds too, but it's this repo->namespace GitOps link where I think TrueNAS SCALE adds specific value.

• Intro to GitLab Agent during its early reveal as a premium feature
• Official installation instructions still suffering some post-strategic-pivot baggage
• The Helm chart which is a breeze at the command line but could do more and could apply iX recommended practices

Thanks for responding!

 

[jct]

Can the gitlab agent run its own docker container?
Can it run on an external system.. server or laptop?

So… in short I'm not sure if you're really asking about GitLab Agent, vs. talking about GitLab Runner (which fires off containers to perform tasks).

When using a Helm-installed GitLab Agent as a conduit, each service can be run privately, with no public IP or port forwarding.

 

[morganL]

So… in short I'm not sure if you're really asking about GitLab Agent, vs. talking about GitLab Runner (which fires off containers to perform tasks).

When using a Helm-installed GitLab Agent as a conduit, each service can be run privately, with no public IP or port forwarding.

Apologies.. misread your post.

Will be interested to hear what issues you have in setting this up.

 

[jct]

Well, perhaps I gave myself a false sense of optimism by using a root shell to install the Helm chart.

My first take on an app catalog shows me that even if we supply the values to decline a new ServiceAccount and a ClusterRoleBinding, we're still getting stopped trying to smuggle in an apparently global ConfigMap. I can of course remove that, but I'm not sure whether it's necessary to further operation. (It appears to only be necessary in the case when a token is sealed with a caCert, but I don't know that for sure.)

(Brief aside: in all of these cases, rather than being told a resource is rejected by policy, I'm told that it "already exists"?!)

Show : Go figure

Code:

Error: Traceback (most recent call last):
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 412, in run
    await self.future
  File "/usr/lib/python3/dist-packages/middlewared/job.py", line 448, in __run_body
    rv = await self.method(*([self] + args))
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1143, in nf
    res = await f(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/schema.py", line 1275, in nf
    return await func(*args, **kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/chart_release.py", line 467, in do_create
    await self.middleware.call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1342, in call
    return await self._call(
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1302, in _call
    return await self.run_in_executor(prepared_call.executor, methodobj, *prepared_call.args)
  File "/usr/lib/python3/dist-packages/middlewared/main.py", line 1203, in run_in_executor
    return await loop.run_in_executor(pool, functools.partial(method, *args, **kwargs))
  File "/usr/lib/python3.9/concurrent/futures/thread.py", line 52, in run
    result = self.fn(*self.args, **self.kwargs)
  File "/usr/lib/python3/dist-packages/middlewared/plugins/chart_releases_linux/helm.py", line 44, in helm_action
    raise CallError(f'Failed to {tn_action} chart release: {stderr.decode()}')
middlewared.service_exception.CallError: [EFAULT] Failed to install chart release: Error: INSTALLATION FAILED: configmaps "teamwork-gitlab-agent" already exists

My experimental catalog is up on GitLab.com. If anyone has a moment, I'd love for us to figure this out.

https://gitlab.com/true-agent/catalog.git (use the "charts" train)

For those not familiar: first you register an agent within GitLab, then you install the agent somewhere using the token and URL it supplies.

But you can test whether it installs or not (and if not, how not) by just making up a token; a default URL is supplied.

 

[truecharts]

To be clear:
Helm Charts deployed outside of the Apps interface, are not officially guaranteed to keep working or covered by Apps backup API and such.

Which is actually also the case with flux, gitlab or other gitops solutions.
It might be something that @morganL missed here...

That being said:
In terms of "is it possible": yes.
We've ran experiments with multiple different gitops solutions in the past :)

 

[jct]

To be clear:
Helm Charts deployed outside of the Apps interface, are not officially guaranteed to keep working or covered by Apps backup API and such.

Yes, thank goodness! It’ll be a breath of fresh air for some of us, to have a clean option to more directly manage our own resources — in standard declarative style — and here with pull requests, staging and other dev-side tooling available.

I get the appeal of so-called “opinionated” or “batteries-included” frameworks, I do. But this other option — particularly for independent dev teams — currently feels underserved. I presume not intentionally, but yet unmistakably so.

If something like this gitlab-agent app can build off of your great work, letting us set aside and incubate any number of individual blank slate workspaces: well sir, that seems like different perspectives coming together and expanding viable use cases in a satisfying way.

In terms of "is it possible": yes.
We've ran experiments with multiple different gitops solutions in the past :)

Any with results you’d like me to consider? I recall that you’d warned off of externally creating any namespace with the reserved prefix “ix-“ (which if this app works right will be a non-issue).

And there was something about instantiating a separate instance of the storage driver? I lost track of that one. Maybe it was in your ArgoCD work.

Any other advice come to mind? I appreciate your checking my work!

 

[jct]

we're still getting stopped trying to smuggle in an apparently global ConfigMap. I can of course remove that, but I'm not sure whether it's necessary to further operation. (It appears to only be necessary in the case when a token is sealed with a caCert, but I don't know that for sure.)

Sure seems like removing that ConfigMap and its reference within the Deployment is all that's needed. This gentleman proves the concept by rendering the template and tweaking it by hand. (And then involving both Kustomize and envsubst.)

I think the solution is to insert four lines into the existing Helm chart to conditionalize those parts. But I'm not yet correctly deriving a new chart with those changes. I might still be referencing the original chart in the ecosystem?

 

[jct]

I think the solution is to insert four lines into the existing Helm chart

I've now taken that thought upstream.

 

[jct]

OK, boom. I've just landed my first successful agent check-in, straight from the catalog.

I'll need to double back and clean up after my sloppy experimentation. Then come up with a separate app for GitLab Runner — for builds.

FYI: GitLab 15.0.0 does not include pull-based integration at the free tier — only push-based. But a near future version should do so, as they've announced.

In order to make use of a push-based agent (i.e. GitLab's CI/CD Tunnel feature) one could manually insert a runner into the agent namespace, or possibly externally bind the two apps plus all requisite configuration. But none of that will be necessary once you have one or more Runners building images and one or more Agents consuming them, so I won't waste any more time on that.

 

[jct]

I'll put this up as a public catalog for now, and see where it leads.

https://gitlab.com/lab-scale/catalog/

 

[jct]

GitLab 15.0.2 doesn't include pull-based integration at the free tier either. Presumably that'll be a 15.1, on June 22.

But I've added an MVP app for GitLab Runner in addition to GitLab Agent.

 

[jct]

OK, for real now. They seem to be circling in on the formalities of a launch with 15.3, which is scheduled for August 22.

There's also now an unscheduled issue that would reduce my two-stage Runner app installation into a single sensible stage, which would be nice but isn't likely to land simultaneously with the new broader access to pull-based GitOps — straight from manifest repositories.

 

[jct]

Well, that's gratifying. Pull-based GitOps is now free, and my initial tests with my GitLab Agent app seem to work as intended.

I'm doubling back to understand recent changes, seeing what's needed in docs and such. GitLab's blog will likely see revised guidance over the next few days as well.

The GitLab Runner app is probably superfluous now, and should instead be managed with everything else through manifest git repositories.

 

[vampirebyte]

Thank you for your contribution, we are already using your chart successfully.

The only thing we still need and maybe we can help with some dev time would be to set the caching, ideally through local truenas S3-like API.

Currently we get this warning and the caching does not work... any idea where to begin?

 

[jct]

Currently we get this warning and the caching does not work... any idea where to begin?

Yikes! No, I'm afraid that's over my head at this point. Other than to reiterate the option of hosting runner manifests inside the GitLab Agent app.

 

[vampirebyte]

Just want to follow up here too for posterity: after playing around with @jct's catalog and agent, I've gave up and installed flux instead.

Obviously this is outside the scope of truenas support, but on the other hand, with the recent gitlab flux integration announcements + the ability to have kubernetes overview in gitlab + portainer installed as App for viewing/editing local k3s stuff + cert-manager & clusterissuer from @truecharts, flux installed with 3 commands on SCALE console, we have a very strong homelab / semi-professional setup.

One can learn then to do CI/CD from gitlab pipelines and run them on the dev truenas instance with full GitOps methodologies which is what any enterprise does or should do in their production workflows. You can also issue SSL certs automatically and even add traefik routing to your manifests = as close as it gets to any cloud deployment, we use it for hosting, testing and deploying our internally-developed apps.

Here is a very straightforward tutorial that also includes the gitlab agent install through flux-helm chart: https://docs.gitlab.com/ee/user/clusters/agent/gitops/flux_tutorial.html (works on free tier too)

BONUS: I've added my old debian VM with docker-ce + portainer-agent as a new environment to the Portainer App and now can also run docker-compose workflows with AutoGitOps as well.

Probably the only thing left is what happens in case of TrueNAS upgrade....

 

[jto]

Obviously this is outside the scope of truenas support, but on the other hand, with the recent gitlab flux integration announcements + the ability to have kubernetes overview in gitlab + portainer installed as App for viewing/editing local k3s stuff + cert-manager & clusterissuer from @truecharts, flux installed with 3 commands on SCALE console, we have a very strong homelab / semi-professional setup.

Would you be able to share the steps you went through to get flux setup on your scale. I've been trying today but I always end up failing at installing components in "flux-system" namespace. I'm assuming i'm missing step in the config somewhere to enable direct manipulation of kubernetes objects.

Code:

► installing components in "flux-system" namespace
E0904 11:29:02.437263  350486 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0904 11:29:02.438315  350486 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0904 11:29:02.439332  350486 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0904 11:29:02.440197  350486 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
E0904 11:29:02.442554  350486 memcache.go:265] couldn't get current server API group list: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused
✗ CustomResourceDefinition/alerts.notification.toolkit.fluxcd.io dry-run failed: Get "http://localhost:8080/api?timeout=32s": dial tcp [::1]:8080: connect: connection refused

 

[jct]

Just want to follow up here too for posterity: after playing around with @jct's catalog and agent, I've gave up and installed flux instead.

FWIW: me too. Probably should have said something back in May.

Flux seems to just do now for GitLab exactly what GitLab said flux would do for them and our CD.

Since that decision: I believe we've been enjoying our longest span of dependable improvements — without the usual monthly wild strategic curveballs and backtracking.

So far it's just… worked! Freeing me to return to my lane as a developer by nature, as opposed to a sysadmin by necessity.

I'd had so much of that development backlogged for so long that I've been on an extended wild-eyed sprint. So I haven't yet doubled back to reassess CI and Auto-DevOps and GitLab's ambitions in that area. My plan had been to make sure I grasp that before I contribute anything new and significant to the lab-scale wiki. Hopefully nobody's waiting for me to get there first.

Cheers!