In the interest of self-flagellation and modernizing my self-hosting setup I’ve begun moving everything I have to a Kubernetes cluster.

My goal is to have a fairly robust GitOps-based k8s setup, where I no longer have to SSH to a server and perform a number of manual tasks to update and maintain my services. There are of course a lot of things to potentially add to such a setup, and this project will probably keep me busy for the next long while. But let’s start small.

For now, I’ve settled on running a single-node k3s cluster on a VM. For this I’ve chosen a Hetzner CX31 VPS.

My initial goal is as follows:

  • Get k3s running on my VM
  • Install Flux and have it monitor my home-ops repo.
  • Set up ingress and LetsEncrypt certificates via Traefik and cert-manager.
  • Install Weave GitOps, to have a neat GUI for managing Flux.
  • Have Weave be reachable through an external hostname over HTTPS.

This should serve as a good, solid PoC for running my own cluster with GitOps.

Installing k3s

The VM has been provisioned and I’m ready to install k3s. K3s deploys its own instance of Traefik by default, so we’ll have to disable that as we want to control that part ourselves. The entire install process can be accomplished with this scary curl/sh oneliner:

$ curl -sfL | INSTALL_K3S_EXEC="server --disable traefik" sh -

Once the install is complete, a kubeconfig file will have been generated for us, that allows us to reach and manage the cluster from the outside, located at /etc/rancher/k3s/k3s.yaml. We just need to scp this back home to our own machine and change the IP to point to the external IP of our VM.

$ kubectl get pods -A
NAMESPACE      NAME                                       READY   STATUS    RESTARTS   AGE
kube-system    local-path-provisioner-957fdf8bc-knfmv     1/1     Running   0          25h
kube-system    coredns-77ccd57875-pvpbb                   1/1     Running   0          25h
kube-system    metrics-server-648b5df564-lnhws            1/1     Running   0          25h

It works. 🎈

Fow now (until we get around to setting up some monitoring of the VM) we shouldn’t need to access the VM directly again.

Installing Flux

We’re using Flux for GitOps. Flux will monitor a git repository and reconcile whatever is defined there with what is running in the cluster. For this we’ll naturally need a git repository. I’ve decided to just start from scratch with this, taking inspiration from the Flux recommended ways of structuring a repo.

Flux is able to manage itself and its own configuration, but this presents us with a chicken-and-egg problem. Before this runs automatically there are a few manual steps we need to perform. Enter flux bootstrap.

Flux has a CLI that can be used to manage Flux in a cluster, and perform initial bootstrapping. The following command will install the necessary controllers and CRDs in the cluster, commit the corresponding manifests to git, set up Flux to monitor the repository and then finally wait for Flux to reconcile its own state.

$ flux bootstrap gitlab \
        --owner=cmoesgaard \
        --repository=home-ops \
        --branch=main \
        --path=./kubernetes/flux \

Checking flux stats will show us that everything looks to be in order.

$ flux stats
GitRepository        	1      	0      	0        	40.9 KiB
OCIRepository        	0      	0      	0        	-
HelmRepository       	0      	0      	0        	-
HelmChart            	0      	0      	0        	-
Bucket               	0      	0      	0        	-
Kustomization        	1      	0      	0        	-
HelmRelease          	0      	0      	0        	-
Alert                	0      	0      	0        	-
Provider             	0      	0      	0        	-
Receiver             	0      	0      	0        	-
ImageUpdateAutomation	0      	0      	0        	-
ImagePolicy          	0      	0      	0        	-
ImageRepository      	0      	0      	0        	-

From now on, everything merged to the main branch in the repo will now be reconciled by Flux. The kubernetes directory in my repo roughly has the following structure:

├── apps
├── charts
├── flux
│   └── flux-system
└── infrastructure
    ├── configs
    └── controllers

In order of reconciliation:

  • flux containing the Flux resources and CRDs and Kustomization manifests for the remaining components allowing Flux to track them too
  • charts containing HelmRepository manifests for the various apps I intend to install
  • infrastructure containing general infrastructure that needs to be in place for the cluster to work
    • configs containing things such as general cluster secrets and configuration
    • controllers containing things such as Traefik, Cert Manager and Weave GitOps
  • And finally apps containing the actual apps I want to self-host

Secret management

With GitOps we get a lovely paper trail regarding changes to our cluster, with every update or change corresponding to a git commit. We’d like to also have a way to store the various secrets we need as part of this, but without exposing every password and API token to the world. For this, we’ll use SOPS. SOPS allows us to store our secrets as encrypted files, so that they can be decrypted and read by us and by Flux in the cluster, but not by third parties snooping around my repository. Flux supports SOPS for decrypting secrets out of the box.

To make this work, we’ll need to perform another manual step. We’ll need to generate a set of encryption keys using age:

$ age-keygen -o age.agekey
Public key: age1vt4lmr873png75lskfhz9ymh29wvnr3gydgzea6w8r7wp4al54lsdhw08a

And then store the private key as a secret in the cluster.

$ cat age.agekey |
kubectl create secret generic sops-age \
--namespace=flux-system \

To allow SOPS to decrypt the files, the age.agekey must be stored in ~/.config/sops/age as keys.txt (or added to this file if it exists)

We only want SOPS to encrypt specific values inside our files, so we add a .sops.yaml file to the root of the repo containing the following, substituting the appropriate public key:

  - path_regex: kubernetes/.*\.yaml
    encrypted_regex: "^(data|stringData)$"
    age: age1vt4lmr873png75lskfhz9ymh29wvnr3gydgzea6w8r7wp4al54lsdhw08a

SOPS can now be used in the following way to encrypt secrets in-place:

$ sops -i -e cluster-secrets.yaml

Turning a secret like this:

apiVersion: v1
kind: Secret
  name: cluster-secrets
  namespace: flux-system
  PASSWORD: hunter2

Into this:

apiVersion: v1
kind: Secret
  name: cluster-secrets
  namespace: flux-system
  PASSWORD: ENC[AES256_GCM,data:VYLeI8Ft,iv:A2Ph7+1qOwcfofRt5HLYCfSMQRnQKVz9fn6Jbm9ENO4=,tag:FH8V2kYjNLRZ7mSywKkVKA==,type:str]
  kms: []
  gcp_kms: []
  azure_kv: []
  hc_vault: []
    - recipient: age1vt4lmr873png75lskfhz9ymh29wvnr3gydgzea6w8r7wp4al54lsdhw08a
      enc: |
        -----BEGIN AGE ENCRYPTED FILE-----
        -----END AGE ENCRYPTED FILE-----        
  lastmodified: "2023-09-13T11:21:32Z"
  mac: ENC[AES256_GCM,data:p3rVoUXsqWbV6K6dKXtCTDcFvw3I9wijPNwJ1hauivoobjlDEbxtW/jksZaqWnlUH0K1EI37P9Gq49wDT8piyeiIVLZ+KvHeiP24/0JXf9i0+Tk2edgT6TIz2Z5jCP84flcFNWu0IT91eoYQqt1Gu5Np33517QJUC6J8jPY0C6k=,iv:4EPEAgSgCFXdtucSxznvORXYVy91LEAaXAIC+NTKeHA=,tag:J0R6gg5E9RSpPkx9CNc4tA==,type:str]
  pgp: []
  encrypted_regex: ^(data|stringData)$
  version: 3.7.3

Opening the above file with sops cluster-secrets.yaml will then allow us to modify the decrypted secret.

Finally, when the Secret above is created in the cluster, we need to add the following snippet to the relevant Kustomization to tell Flux to decrypt the secret using SOPS and the private key we stored previously:

    provider: sops
      name: sops-age

The secret is now stored in the cluster as a normal Secret:

$ kubectl get secret -n flux-system cluster-secrets -o jsonpath="{.data.PASSWORD}" | base64 --decode

Installing Weave GitOps

With everything in place, it’s time to try installing our first non-flux component. Weave GitOps gives us a UI for Flux, which will help us deploy the subsequent components, so let’s start there.

We’re installing Weave using its Helm chart, so we first have to add a HelmRepository manifest for the Weave GitOps Helm repository, allowing Flux to pull the Helm chart from there. The following resources have been adapted from the Flux example repo.

kind: HelmRepository
  name: weave-gitops
  namespace: flux-system
  type: oci
  interval: 60m0s
  url: oci://

Then we have to create a HelmRelease for the actual helm chart.

kind: HelmRelease
  name: weave-gitops
  namespace: flux-system
  interval: 60m
      chart: weave-gitops
      version: "*"
        kind: HelmRepository
        name: weave-gitops
      interval: 12h
          - ALL
      readOnlyRootFilesystem: true
      runAsNonRoot: true
      runAsUser: 1000
      create: true
      username: admin
      passwordHash: ${WEAVE_PASSWORD_HASH}

The WEAVE_PASSWORD_HASH is stored as a Secret which we’ve instructed Flux to use when building the Kustomization:

    substitute: {}
      - kind: Secret
        name: cluster-secrets

The manifests above (and the password hash secret) are commited and pushed to the repo, and now we wait for Flux to do its magic.

$ flux stats -n flux-system
GitRepository        	1      	0      	0        	40.9 KiB
OCIRepository        	0      	0      	0        	-
HelmRepository       	1      	0      	0        	675.5 KiB
HelmChart            	1      	0      	0        	177.1 KiB
Bucket               	0      	0      	0        	-
Kustomization        	4      	0      	0        	-
HelmRelease          	1      	0      	0        	-
Alert                	0      	0      	0        	-
Provider             	0      	0      	0        	-
Receiver             	0      	0      	0        	-
ImageUpdateAutomation	0      	0      	0        	-
ImagePolicy          	0      	0      	0        	-
ImageRepository      	0      	0      	0        	-

Everything looks good. ✨ We don’t have a way of reaching our deployment from the outside yet, but for now we’ll just forward a port.

$ kubectl -n flux-system port-forward svc/weave-gitops 9001:9001
Forwarding from -> 9001
Forwarding from [::1]:9001 -> 9001

It works, hooray!

What’s next?

We’ve managed to set up the initial cluster with Flux and deployed our first service. We’re one step closer to having a fully functioning setup. Next, we’ll set up Traefik and Cert Manager so we’ll be able to reach our services from the outside.