100% `GitOps` using Flux

18-07-2022 · 14 min read · gitops devxp ·

Share on:

Overview

In a previous article, we've seen how to use Crossplane so that we can manage cloud resources the same way as our applications. ❤️ Declarative approach! There were several steps and command lines in order to get everything working and reach our target to provision a dev Kubernetes cluster.

Here we'll achieve exactly the same thing but we'll do that in the GitOps way. According to the OpenGitOps working group there are 4 GitOps principles:

The desired state of our system must be expressed declaratively.
This state must be stored in a versioning system.
Changes are pulled and applied automatically in the target platform whenever the desired state changes.
If, for any reason, the current state is modified, it will be automatically reconciled with the desired state.

There are several GitOps engine options. The most famous ones are ArgoCD and Flux. We won't compare them here. I chose Flux because I like its composable architecture with different controllers, each one handling a core Flux feature (GitOps toolkit).

Learn more about GitOps toolkit components here.

🎯 Our target

Here we want to declare our desired infrastructure components only by adding git changes. By the end of this article you'll get a GKE cluster provisioned using a local Crossplane instance. We'll discover Flux basics and how to use it in order to build a complete GitOps CD workflow.

☑️ Requirements

📥 Install required tools

First of all we need to install a few tools using asdf

Create a local file .tool-versions

1cd ~/sources/devflux/
2
3cat > .tool-versions <<EOF
4flux2 0.31.3
5kubectl 1.24.3
6kubeseal 0.18.1
7kustomize 4.5.5
8EOF

1for PLUGIN in $(cat .tool-versions | awk '{print $1}'); do asdf plugin-add $PLUGIN; done
2
3asdf install
4Downloading ... 100.0%
5Copying Binary
6...

Check that all the required tools are actually installed.

1asdf current
2flux2           0.31.3          /home/smana/sources/devflux/.tool-versions
3kubectl         1.24.3          /home/smana/sources/devflux/.tool-versions
4kubeseal        0.18.1          /home/smana/sources/devflux/.tool-versions
5kustomize       4.5.5           /home/smana/sources/devflux/.tool-versions

🔑 Create a Github personal access token

In this article the git repository is hosted in Github. In order to be able to use the flux bootstrap a personnal access token is required.

Please follow this procedure.

Warning

Store the Github token in a safe place for later use

🧑‍💻 Clone the devflux repository

All the files used for the upcoming steps can be retrieved from this repository. You should clone it, that will be easier to copy them into your own repository.

1git clone https://github.com/Smana/devflux.git

🚀 Bootstrap flux in the Crossplane cluster

As we will often be using the flux CLI you may want to configure the bash|zsh completion

1source <(flux completion bash)

Warning

Here we consider that you already have a local k3d instance. If not you may want to either go through the whole previous article or just run the local cluster creation.

Ensure that you're working in the right context

1kubectl config current-context
2k3d-crossplane

Run the bootstrap command that will basically deploy all Flux's components in the namespace flux-system. Here I'll create a repository named devflux using my personal Github account.

 1export GITHUB_USER=<YOUR_ACCOUNT>
 2export GITHUB_TOKEN=ghp_<REDACTED> # your personal access token
 3export GITHUB_REPO=devflux
 4
 5flux bootstrap github --owner="${GITHUB_USER}" --repository="${GITHUB_REPO}" --personal --path=clusters/k3d-crossplane
 6► cloning branch "main" from Git repository "https://github.com/<YOUR_ACCOUNT>/devflux.git"
 7...
 8✔ configured deploy key "flux-system-main-flux-system-./clusters/k3d-crossplane" for "https://github.com/<YOUR_ACCOUNT>/devflux"
 9...
10✔ all components are healthy

Check that all the pods are running properly and that the kustomization flux-system has been successfully reconciled.

 1kubectl get po -n flux-system
 2NAME                                       READY   STATUS    RESTARTS   AGE
 3helm-controller-5985c795f8-gs2pc           1/1     Running   0          86s
 4notification-controller-6b7d7485fc-lzlpg   1/1     Running   0          86s
 5kustomize-controller-6d4669f847-9x844      1/1     Running   0          86s
 6source-controller-5fb4888d8f-wgcqv         1/1     Running   0          86s
 7
 8flux get kustomizations
 9NAME            REVISION        SUSPENDED       READY   MESSAGE
10flux-system     main/33ebef1    False           True    Applied revision: main/33ebef1

Running the bootstap command actually creates a github repository if it doesn't exist yet. Clone it now for our upcoming changes. You'll notice that the first commit has been made by Flux.

 1git clone https://github.com/<YOUR_ACCOUNT>/devflux.git
 2Cloning into 'devflux'...
 3
 4cd devflux
 5
 6git log -1
 7commit 2beb6aafea67f3386b50cbc706fb34575844040d (HEAD -> main, origin/main, origin/HEAD)
 8Author: Flux <>
 9Date:   Thu Jul 14 17:13:27 2022 +0200
10
11    Add Flux sync manifests
12
13ls clusters/k3d-crossplane/flux-system/
14gotk-components.yaml  gotk-sync.yaml  kustomization.yaml

📂 Flux repository structure

There are several options for organizing your resources in the Flux configuration repository. Here is a proposition for the sake of this article.

 1tree -d -L 2
 2.
 3├── apps
 4│   ├── base
 5│   └── dev-cluster
 6├── clusters
 7│   ├── dev-cluster
 8│   └── k3d-crossplane
 9├── infrastructure
10│   ├── base
11│   ├── dev-cluster
12│   └── k3d-crossplane
13├── observability
14│   ├── base
15│   ├── dev-cluster
16│   └── k3d-crossplane
17└── security
18    ├── base
19    ├── dev-cluster
20    └── k3d-crossplane

Directory	Description	Example
/apps	our applications	Here we'll deploy a demo application "online-boutique"
/infrastructure	base infrastructure/network components	Crossplane as it will be used to provision cloud resources but we can also find CSI/CNI/EBS drivers...
/observability	All metrics/apm/logging tools	Prometheus of course, Opentelemetry ...
/security	Any component that enhance our security level	SealedSecrets (see below)

Info

For the upcoming steps please refer to the demo repository here

Let's use this structure and begin to deploy applications 🚀.

🔐 SealedSecrets

There are plenty of alternatives when it comes to secrets management in Kubernetes. In order to securely store secrets in a git repository the GitOps way we'll make use of SealedSecrets. It uses a custom resource definition named SealedSecrets in order to encrypt the Kubernetes secret at the client side then the controller is in charge of decrypting and generating the expected secret in the cluster.

🛠️ Deploy the controller using Helm

The first thing to do is to declare the kustomization that handles all the security tools.

clusters/k3d-crossplane/security.yaml

 1apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
 2kind: Kustomization
 3metadata:
 4  name: security
 5  namespace: flux-system
 6spec:
 7  prune: true
 8  interval: 4m0s
 9  sourceRef:
10    kind: GitRepository
11    name: flux-system
12  path: ./security/k3d-crossplane
13  healthChecks:
14    - apiVersion: helm.toolkit.fluxcd.io/v1beta1
15      kind: HelmRelease
16      name: sealed-secrets
17      namespace: kube-system

Info

A Kustomization is a custom resource that comes with Flux. It basically points to a set of Kubernetes resources managed with kustomize The above security kustomization points to a local directory where the kustomize resources are.

1...
2spec:
3  path: ./security/k3d-crossplane
4...

Note

This is worth noting that there are two types on kustomizations. That can be confusing when you start playing with Flux.

One managed by flux's kustomize controller. Its API is kustomization.kustomize.toolkit.fluxcd.io
The other kustomization.kustomize.config.k8s.io is for the kustomize overlay

The kustomization.yaml file is always used for the kustomize overlay. Flux itself doesn't need this overlay in all cases, but if you want to use features of a Kustomize overlay you will occasionally need to create it in order to access them. It provides instructions to the Kustomize CLI.

We will deploy SealedSecrets using the Helm chart. So we need to declare the source of this chart. Using the kustomize overlay system, we'll first create the base files that will be inherited at the cluster level.

security/base/sealed-secrets/source.yaml

1apiVersion: source.toolkit.fluxcd.io/v1beta2
2kind: HelmRepository
3metadata:
4  name: sealed-secrets
5  namespace: flux-system
6spec:
7  interval: 30m
8  url: https://bitnami-labs.github.io/sealed-secrets

Then we'll define the HelmRelease which references the above source. Put the values you want to apply to the Helm chart under spec.values

security/base/sealed-secrets/helmrelease.yaml

 1apiVersion: helm.toolkit.fluxcd.io/v2beta1
 2kind: HelmRelease
 3metadata:
 4  name: sealed-secrets
 5  namespace: kube-system
 6spec:
 7  releaseName: sealed-secrets
 8  chart:
 9    spec:
10      chart: sealed-secrets
11      sourceRef:
12        kind: HelmRepository
13        name: sealed-secrets
14        namespace: flux-system
15      version: "2.4.0"
16  interval: 10m0s
17  install:
18    remediation:
19      retries: 3
20  values:
21    fullnameOverride: sealed-secrets-controller
22    resources:
23      requests:
24        cpu: 80m
25        memory: 100Mi

If you're starting your repository from scratch you'll need to generate the kustomization.yaml file (kustomize overlay).

1kustomize create --autodetect

security/base/sealed-secrets/kustomization.yaml

1apiVersion: kustomize.config.k8s.io/v1beta1
2kind: Kustomization
3resources:
4- helmrelease.yaml
5- source.yaml

Now we declare the sealed-secret kustomization at the cluster level. Just for the example we'll overwrite a value at the cluster level using kustomize's overlay system.

security/k3d-crossplane/sealed-secrets/helmrelease.yaml

 1apiVersion: helm.toolkit.fluxcd.io/v2beta1
 2kind: HelmRelease
 3metadata:
 4  name: sealed-secrets
 5  namespace: kube-system
 6spec:
 7  values:
 8    resources:
 9      requests:
10        cpu: 100m

security/k3d-crossplane/sealed-secrets/kustomization.yaml

1apiVersion: kustomize.config.k8s.io/v1beta1
2kind: Kustomization
3bases:
4  - ../../base
5patches:
6  - helmrelease.yaml

Pushing our changes is the only thing to do in order to get sealed-secrets deployed in the target cluster.

1git commit -m "security: deploy sealed-secrets in k3d-crossplane"
2[security/sealed-secrets 283648e] security: deploy sealed-secrets in k3d-crossplane
3 6 files changed, 66 insertions(+)
4 create mode 100644 clusters/k3d-crossplane/security.yaml
5 create mode 100644 security/base/sealed-secrets/helmrelease.yaml
6 create mode 100644 security/base/sealed-secrets/kustomization.yaml
7 create mode 100644 security/base/sealed-secrets/source.yaml
8 create mode 100644 security/k3d-crossplane/sealed-secrets/helmrelease.yaml
9 create mode 100644 security/k3d-crossplane/sealed-secrets/kustomization.yaml

After a few seconds (1 minutes by default) a new kustomization will appear.

1flux get kustomizations
2NAME            REVISION        SUSPENDED       READY   MESSAGE
3flux-system     main/d36a33c    False           True    Applied revision: main/d36a33c
4security        main/d36a33c    False           True    Applied revision: main/d36a33c

And all the resources that we declared in the flux repository should be available and READY.

1flux get sources helm
2NAME            REVISION                                                                SUSPENDED       READY   MESSAGE
3sealed-secrets  4c0aa1980e3ec9055dea70abd2b259aad1a2c235325ecf51a25a92a39ac4eeee        False           True    stored artifact for revision '4c0aa1980e3ec9055dea70abd2b259aad1a2c235325ecf51a25a92a39ac4eeee'

1flux get helmrelease -n kube-system
2NAME            REVISION        SUSPENDED       READY   MESSAGE
3sealed-secrets  2.2.0           False           True    Release reconciliation succeeded

🧪 A first test SealedSecret

Let's use the CLI kubeseal to test it out. We'll create a SealedSecret that will be decrypted by the sealed-secrets controller in the cluster and create the expected secret foobar

 1kubectl create secret generic foobar -n default --dry-run=client -o yaml --from-literal=foo=bar \
 2| kubeseal --namespace default --format yaml | kubectl apply -f -
 3sealedsecret.bitnami.com/foobar created
 4
 5kubectl get secret -n default foobar
 6NAME     TYPE     DATA   AGE
 7foobar   Opaque   1      3m13s
 8
 9kubectl delete sealedsecrets.bitnami.com foobar
10sealedsecret.bitnami.com "foobar" deleted

☁️ Deploy and configure Crossplane

🔑 Create the Google service account secret

The first thing we need to do in order to get Crossplane working is to create the GCP serviceaccount. The steps have been covered here in the previous article. We'll create a SealedSecret gcp-creds that contains the serviceaccount file crossplane.json.

infrastructure/k3d-crossplane/crossplane/configuration/sealedsecrets.yaml

1kubectl create secret generic gcp-creds --context k3d-crossplane -n crossplane-system --from-file=creds=./crossplane.json --dry-run=client -o yaml \
2| kubeseal --format yaml --namespace crossplane-system - > infrastructure/k3d-crossplane/crossplane/configuration/sealedsecrets.yaml

🔄 Crossplane dependencies

Now we will deploy Crossplane with Flux. I won't put the manifests here you'll find all of them in this repository. However it's important to understand that, in order to deploy and configure Crossplane properly we need to do that in a specific order. Indeed several CRD's (custom resource definitions) are required:

First of all we'll install the crossplane controller.
Then we'll configure the provider because the custom resource is now available thanks to the crossplane controller installation.
Finally a provider installation deploys several CRDs that can be used to configure the provider itself and cloud resources.

The dependencies between kustomizations can be controlled using the parameters dependsOn. Looking at the file clusters/k3d-crossplane/infrastructure.yaml, we can see for example that the kustomization infrastructure-custom-resources depends on the kustomization crossplane_provider which itself depends on crossplane-configuration....

 1---
 2apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
 3kind: Kustomization
 4metadata:
 5  name: crossplane-provider
 6spec:
 7...
 8  dependsOn:
 9    - name: crossplane-core
10---
11apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
12kind: Kustomization
13metadata:
14  name: crossplane-configuration
15spec:
16...
17  dependsOn:
18    - name: crossplane-provider
19---
20apiVersion: kustomize.toolkit.fluxcd.io/v1beta2
21kind: Kustomization
22metadata:
23  name: infrastructure-custom-resources
24spec:
25...
26  dependsOn:
27    - name: crossplane-configuration

Commit and push the changes for the kustomisations to appear. Note that they'll be reconciled in the defined order.

1flux get kustomizations
2NAME                            REVISION        SUSPENDED       READY   MESSAGE
3infrastructure-custom-resources                 False           False   dependency 'flux-system/crossplane-configuration' is not ready
4crossplane-configuration                        False           False   dependency 'flux-system/crossplane-provider' is not ready
5security                        main/666f85a    False           True    Applied revision: main/666f85a
6flux-system                     main/666f85a    False           True    Applied revision: main/666f85a
7crossplane-core                 main/666f85a    False           True    Applied revision: main/666f85a
8crossplane-provider             main/666f85a    False           True    Applied revision: main/666f85a

Then all Crossplane components will be deployed, we can have a look to the HelmRelease status for instance.

 1kubectl describe helmrelease -n crossplane-system crossplane
 2...
 3Status:
 4  Conditions:
 5    Last Transition Time:          2022-07-15T19:12:04Z
 6    Message:                       Release reconciliation succeeded
 7    Reason:                        ReconciliationSucceeded
 8    Status:                        True
 9    Type:                          Ready
10    Last Transition Time:          2022-07-15T19:12:04Z
11    Message:                       Helm upgrade succeeded
12    Reason:                        UpgradeSucceeded
13    Status:                        True
14    Type:                          Released
15  Helm Chart:                      crossplane-system/crossplane-system-crossplane
16  Last Applied Revision:           1.9.0
17  Last Attempted Revision:         1.9.0
18  Last Attempted Values Checksum:  056dc1c6029b3a644adc7d6a69a93620afd25b65
19  Last Release Revision:           2
20  Observed Generation:             1
21Events:
22  Type    Reason  Age   From             Message
23  ----    ------  ----  ----             -------
24  Normal  info    20m   helm-controller  HelmChart 'crossplane-system/crossplane-system-crossplane' is not ready
25  Normal  info    20m   helm-controller  Helm upgrade has started
26  Normal  info    19m   helm-controller  Helm upgrade succeeded

And our GKE cluster should also be created because we defined a bunch of crossplane custom resources in infrastructure/k3d-crossplane/custom-resources/crossplane

1kubectl get cluster
2NAME          READY   SYNCED   STATE     ENDPOINT        LOCATION         AGE
3dev-cluster   True    True     RUNNING   34.x.x.190      europe-west9-a   22m

🚀 Bootstrap flux in the dev cluster

Our local Crossplane cluster is now ready and it created our dev cluster and we also want it to be managed with Flux. So let's configure Flux for this dev cluster using the same bootstrap command.

Authenticate to the newly created cluster. The following command will automatically change your current context.

1gcloud container clusters get-credentials dev-cluster --zone europe-west9-a --project <your_project>
2Fetching cluster endpoint and auth data.
3kubeconfig entry generated for dev-cluster.
4
5kubectl config current-context
6gke_<your_project>_europe-west9-a_dev-cluster

Run the bootstrap command for the dev-cluster.

 1export GITHUB_USER=Smana
 2export GITHUB_TOKEN=ghp_<REDACTED> # your personal access token
 3export GITHUB_REPO=devflux
 4
 5flux bootstrap github --owner="${GITHUB_USER}" --repository="${GITHUB_REPO}" --personal --path=clusters/dev-cluster
 6► cloning branch "main" from Git repository "https://github.com/Smana/devflux.git"
 7...
 8✔ configured deploy key "flux-system-main-flux-system-./clusters/dev-cluster" for "https://github.com/Smana/devflux"
 9...
10✔ all components are healthy

Note

It's worth noting that each Kubernetes cluster generates its own sealing keys. That means that if you recreate the dev-cluster, you must regenerate all the sealedsecrets. In our example we declared a secret in order to set the Grafana credentials. Here's the command you need to run in order to create a new version of the sealedsecret and don't forget to use the proper context 😉.

1kubectl create secret generic kube-prometheus-stack-grafana \
2--from-literal=admin-user=admin --from-literal=admin-password=<yourpassword> --namespace observability --dry-run=client -o yaml \
3| kubeseal --namespace observability --format yaml > observability/dev-cluster/kube-prometheus-stack/sealedsecrets.yaml

After a few seconds we'll get the following kustomizations deployed.

1flux get kustomizations
2NAME            REVISION        SUSPENDED       READY   MESSAGE
3apps            main/1380eaa    False           True    Applied revision: main/1380eaa
4flux-system     main/1380eaa    False           True    Applied revision: main/1380eaa
5observability   main/1380eaa    False           True    Applied revision: main/1380eaa
6security        main/1380eaa    False           True    Applied revision: main/1380eaa

Here we configured the prometheus stack and deployed a demo microservices stack named "online-boutique" This demo application exposes the frontend through a service of type LoadBalancer.

1kubectl get svc -n demo frontend-external
2NAME                TYPE           CLUSTER-IP       EXTERNAL-IP    PORT(S)        AGE
3frontend-external   LoadBalancer   10.140.174.201   34.155.121.2   80:31943/TCP   7m44s

Use the EXTERNAL_IP

🕵️ Troubleshooting

The cheatsheet in Flux's documentation contains many ways for troubleshooting when something goes wrong. Here I'll just give a sample of my favorite command lines.

Objects that aren't ready

1flux get all -A --status-selector ready=false

Checking the logs of a given kustomization

1flux logs --kind kustomization --name infrastructure-custom-resources
22022-07-15T19:38:52.996Z info Kustomization/infrastructure-custom-resources.flux-system - server-side apply completed
32022-07-15T19:38:53.016Z info Kustomization/infrastructure-custom-resources.flux-system - Reconciliation finished in 66.12266ms, next run in 4m0s
42022-07-15T19:11:34.697Z info Kustomization/infrastructure-custom-resources.flux-system - Discarding event, no alerts found for the involved object

Show how a given pod is managed by Flux.

 1flux trace -n crossplane-system pod/crossplane-5dc8d888d7-g95qx
 2
 3Object:         Pod/crossplane-5dc8d888d7-g95qx
 4Namespace:      crossplane-system
 5Status:         Managed by Flux
 6---
 7HelmRelease:    crossplane
 8Namespace:      crossplane-system
 9Revision:       1.9.0
10Status:         Last reconciled at 2022-07-15 21:12:04 +0200 CEST
11Message:        Release reconciliation succeeded
12---
13HelmChart:      crossplane-system-crossplane
14Namespace:      crossplane-system
15Chart:          crossplane
16Version:        1.9.0
17Revision:       1.9.0
18Status:         Last reconciled at 2022-07-15 21:11:36 +0200 CEST
19Message:        pulled 'crossplane' chart with version '1.9.0'
20---
21HelmRepository: crossplane
22Namespace:      crossplane-system
23URL:            https://charts.crossplane.io/stable
24Revision:       362022f8c7ce215a0bb276887115cb5324b35a3169723900c84456adc3538a8d
25Status:         Last reconciled at 2022-07-15 21:11:35 +0200 CEST
26Message:        stored artifact for revision '362022f8c7ce215a0bb276887115cb5324b35a3169723900c84456adc3538a8d'

If you want to check what would be the changes before pushing your commit. In thi given example I just increased the cpu requests for the sealed-secrets controller.

 1flux diff kustomization security  --path security/k3d-crossplane
 2✓  Kustomization diffing...
 3► HelmRelease/kube-system/sealed-secrets drifted
 4
 5metadata.generation
 6  ± value change
 7    - 6
 8    + 7
 9
10spec.values.resources.requests.cpu
11  ± value change
12    - 100m
13    + 120m
14
15⚠️ identified at least one change, exiting with non-zero exit code

🧹 Cleanup

Don't forget to delete the Cloud resources if you don't want to have a bad suprise 💵! Just comment the file infrastructure/k3d-crossplane/custom-resources/crossplane/kustomization.yaml

1apiVersion: kustomize.config.k8s.io/v1beta1
2kind: Kustomization
3resources:
4  # - cluster.yaml
5  - network.yaml

👏 Achievements

With our current setup everything is configured using the GitOps approach:

We can manage infrastructure resources using Crossplane.
Our secrets are securely stored in our git repository.
We have a dev-cluster that we can enable or disable just but commenting a yaml file.
Our demo application can be deployed from scratch in seconds.

💭 final thoughts

Flux is probably the tool I'm using the most on a daily basis. It's really amazing!

When you get familiar with its concepts and the command line it becomes really easy to use and troubleshoot. You can use either Helm when a chart is available or Kustomize.

However we faced a few issues:

It's not straightforward to find an efficient structure depending on the company needs. Especially when you have several Kubernetes controllers that depend on other CRDs.
The Helm controller doesn't maintain a state of the Kubernetes resources deployed by the Helm chart. That means that if you delete a resource which has been deployed through a Helm chart, it won't be reconciled (It will change soon. Being discussed here)
Flux doesn't provide itself a web UI and switching between CLIs (kubectl, flux ...) can be annoying from a developer perspective. (I'm going to test weave-gitops )

I've been using Flux in production for more than a year and we configured it with the image automation so that the only thing a developer has to do is to merge a pull request and the new version of the application is automatically deployed in the target cluster.

I should probably give another try to ArgoCD in order to be able to compare these precisely 🤔.