Veeam TrueNas and S3 bucket with immutability

[infraerik]

The temptation is too big not to use it in prod aka to not compile an executable hehe – this feature just seems predestined to work with ZFS

How'd you do that? Can't seem to find any build scripts of MinIO for FreeBSD online;– not too familiar with Go's cross-platform dependencies. Just found the cross-compile.sh – let's see if I can somehow compile it successfully from FreeBSD

Just grab the file in the dropbox link

 

[xness]

Just grab the file in the dropbox link

I upgraded my CORE 13.0 instance to SCALE – which was its very own adventure – using the ISO via IMPI, since the plugin system will likely be deprecated in CORE as per @Kris Moore 's comment – he also outlined there might be some issues running Golang-based applications in FreeBSD

Sounds like future efforts might be directed more towards TrueNAS SCALE – though the MinIO setup on SCALE is a bit unintuitive...

 

[J0rgepereira]

I'm trying to setup the S3 repository for use with Veeam but I cannot seem to get the certificates to work. I'm using the Microsoft CA. Does anyone have any words of wisdom :) or can point me in the right direction.

Thanks, Jorge

 

[infraerik]

Yeah - there’s some work to be done clarifying the notions and dependencies of applications/plugins/containers/helm charts/K3S/K8S in TrueNAS SCALE. At least if you go the Docker route, you should be getting the latest version directly from the repository so you can go straight to object lock. But as soon as you add networking to containers things get complicated. At least with jails you could just spin up a test using DHCP and tinker before moving to fixed IPs

 

[xness]

Ha! Exactly what I'm struggling with right now too. Setting up a custom DNS record + assigning the container to a specific interface with VLAN tag
Will have to do some digging where the configuration is stored – documentation doesn't seem to mention it and it doesn't seem like docker-compose is used

 

[xness]

I'm trying to setup the S3 repository for use with Veeam but I cannot seem to get the certificates to work. I'm using the Microsoft CA. Does anyone have any words of wisdom :) or can point me in the right direction.

My words of wisdom for anyone trying to use AD CS: Don't use AD CS
The templates were created in Server 2003 and have not been updated ever since – ECDSA is still an issue for them and also this: From Misconfigured Certificate Template to Domain Admin

Also you basically need two dedicated hosts – one for Root CA the other for intermediate CA in DMZ, since CRLs need to be published via HTTP IIS aside from the default LDAP publishing endpoint if you want certificates to work properly,
The benefits AD CS adds (mainly auto-rollout of the CA + integrated "ACME" client for computer certificates) is outweighed by the complexity of running + maintaining the infrastructure as well as the security risks.

If you want a CA, I suggest trying django-ca

Aside from that a CA is not required to get MinIO to work. Simply self-sign a certificate and it's done – your MinIO is probably in a dedicated, low access management network anyways (right?)

 

[infraerik]

I second that. Minio has a great little tool for generating certificates (certgen) that conform to modern standards for internal use. Otherwise I put it behind an nginx proxy and use certbot.

 

[infraerik]

Ha! Exactly what I'm struggling with right now too. Setting up a custom DNS record + assigning the container to a specific interface with VLAN tag
Will have to do some digging where the configuration is stored – documentation doesn't seem to mention it and it doesn't seem like docker-compose is used

Don’t forget that there are two packaged options: the default Minio app as well as the BTCharts version. The BTCharts variant gives you a lot more control over the networking because it’s a full on K3S setup. At least there, you can select the interface to connect to, while I can’t find any networking options in the default app deployment wizard. While I appreciate the move towards this modern tooling, It adds a lot of complexity, overhead and non-obvious dependencies for simple applications.

OTOH, this will drag a new generation of sysadmins into the Kubernetes era (kicking and screaming).

Image title: deploying a single container app on Kubernetes

PS: if you use BTCharts, don’t forget to add 127.0.0.1 ix-truenas to /etc/hosts in the Network > Global Configuration

 

[xness]

Don’t forget that there are two packaged options: the default Minio app as well as the BTCharts version. The BTCharts variant gives you a lot more control over the networking because it’s a full on K3S setup. At least there, you can select the interface to connect to, while I can’t find any networking options in the default app deployment wizard. While I appreciate the move towards this modern tooling, It adds a lot of complexity, overhead and non-obvious dependencies for simple applications.

OTOH, this will drag a new generation of sysadmins into the Kubernetes era (kicking and screaming).

Image title: deploying a single container app on Kubernetes

PS: if you use BTCharts, don’t forget to add 127.0.0.1 ix-truenas to /etc/hosts in the Network > Global Configuration

Yeah I used the BTCharts Kubernetes version. Hahah whether you like it or not you'll have to work through it now indeed – wanted to look into Kubernetes by the end of year once Raspberry Pi Compute Modules were more readily available.

Anyways I think I'll scratch it for now – seems a bit overengineered to me and the setup GUI is just too limiting imo; MinIO cannot even loopback to its own DNS name within the container when setting one. When trying to access MinIO it via the webinterface it errors out with "Could not connect to <ipv4-at-dns-server>:9000" aka its own address

It's a single-node deployment anyways – so using docker-compose for now; allows using all the required features and an explicit configuration approach:

Code:

version: '3.7'

services:
  minio:
    image: quay.io/minio/minio:RELEASE.2022-07-04T21-02-54Z
    container_name: s3hyper
    restart: unless-stopped
    command: server /data --console-address ":9002"
    ports:
      - "9000:9000"
      - "9002:9002"
    volumes:
      - data:/data
      - certs:/certs
      - config:/root/.minio
    extra_hosts:
      - "hyper.s3.domain.tld:127.0.0.1"
    healthcheck:
      test: ["CMD", "curl", "-f", "http://localhost:9000/minio/health/live"]
      interval: 30s
      timeout: 20s
      retries: 3
    networks:
      - management
      - default  # storage

volumes:
  data:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: ${STORAGE_PATH}/data
  config:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: ${STORAGE_PATH}/config
  certs:
    driver: local
    driver_opts:
      type: none
      o: bind
      device: ${STORAGE_PATH}/certs


networks:
  default:
    driver: macvlan
    enable_ipv6: false
    driver_opts:
        parent: eno1.1150  # VLAN 1150
    ipam:
        driver: default
  management:
    driver: macvlan
    enable_ipv6: false
    driver_opts:
        parent: eno1
    ipam:
        driver: default

Code:

# .env
MINIO_ROOT_USER=s3-hyper
MINIO_ROOT_PASSWORD=hunter42
MINIO_SERVER_URL=https://hyper.s3.domain.tld
STORAGE_PATH=/mnt/Goliath/s3-hyper
TZ=Europe/Berlin

Let's see how it handles…