Going Further with Crossplane: Compositions and Functions

With the emergence of Platform Engineering, we are witnessing a shift towards the creation of self-service solutions for developers. This approach facilitates the standardization of DevOps practices, enhances the developer experience, and reduces the cognitive load associated with managing tools.

Crossplane, an "Incubating" project under the Cloud Native Computing Foundation (CNCF), aims to become the leading framework for creating Cloud Native platforms. In my first article about Crossplane, I introduced this tool and explained how it leverages GitOPs principles for infrastructure, enabling the creation of a GKE cluster.

Now celebrating its 5th anniversary 🎂🎉, the project has matured and expanded its features over time.

In this post, we will explore some of Crossplane's key features, with a particular focus on the Composition Functions that are generating significant interest within the community. Are we about to witness a pivotal moment for the project?

🎯 Our target

The Crossplane documentation is comprehensive, so we'll quickly review the basic concepts to focus on a specific use case: Deploying Harbor on an EKS cluster, adhering to high availability best practices.

Harbor's availability mainly depends on its stateful components. Users are responsible for their implementation, which should be tailored to the target infrastructure. This blog post presents the options I selected for optimal availability.

• Redis deployed using the Bitnami Helm chart in "master/slave" mode.
• Artifacts are stored in an AWS S3 bucket.
• A PostgreSQL RDS instance for the database.

We will now explore how Crossplane simplifies the provisioning of this RDS instance by providing a level of abstraction that exposes only an opinionated set of options. 🚀

🏗️ Prerequisites

Before we can build our Compositions, some groundwork is necessary as several preliminary operations need to be carried out. These steps are performed in a specific order:

1. Deployment of Crossplane's core controller using the Helm chart.
2. Installation of the providers and their configurations.
3. Deployment of various configurations using the previously installed providers, especially the Compositions and Composition Functions.
4. Declarations of Claims to consume the Compositions.

These steps are described through Flux's dependencies and can be viewed here.

📦 The Compositions

Put simply, a Composition in Crossplane is a way to aggregate and automatically manage multiple resources whose configuration can sometimes be complex.

It leverages the Kubernetes API to define and orchestrate not just infrastructure elements like storage and networking, but also various other components (refer to the list of providers). This method provides developers with a simplified interface, representing an abstraction layer that hides the more complex technical details of the underlying infrastructure.

To achieve my goal of creating Harbor's database, I first looked for a relevant example. For this purpose, I used the Upbound marketplace, where a few Compositions can be found that can be used as starting points.

Based on the configuration-rds composition, I wanted to add the following elements:

• 🔑 Allow pods to access the instance.
• ▶️ Creation of a ExternalName Kubernetes service with a predictable name that will be used in Harbor's configuration.
• 💾 Creation of databases and the roles that will own them.

❓ How would this Composition then be used if, for example, a developer wishes to have a database? They is simply done by declaring a Claim which represents the level of abstraction exposed to the users. Let's get a closer look 🔍

tooling/base/harbor/sqlinstance.yaml

 1apiVersion: cloud.ogenki.io/v1alpha1
 2kind: SQLInstance
 3metadata:
 4  name: xplane-harbor
 5  namespace: tooling
 6spec:
 7  parameters:
 8    engine: postgres
 9    engineVersion: "15"
10    size: small
11    storageGB: 20
12    databases:
13      - owner: harbor
14        name: registry
15    passwordSecretRef:
16      namespace: tooling
17      name: harbor-pg-masterpassword
18      key: password
19  compositionRef:
20    name: xsqlinstances.cloud.ogenki.io
21  writeConnectionSecretToRef:
22    name: xplane-harbor-rds

Here we observe that it boils down to a simple resource with few parameters to express our needs:

• A PostgreSQL instance version 15 will be created.
• The instance type is at the discretion of the platform team (the maintainers of the composition). In the above Claim, we ask for a small instance, which is interpreted by the composition as db.t3.small. infrastructure/base/crossplane/configuration/sql-instance-composition.yaml

1transforms:
2  - type: map
3    map:
4      large: db.t3.large
5      medium: db.t3.medium
6      small: db.t3.small

• The master user's password is retrieved from a harbor-pg-masterpassword secret, retrieved from an External Secret.
• Once the instance is created, the connection details are stored in a secret xplane-harbor-rds.

This is where we can fully appreciate the power of Crossplane Compositions! Indeed, many resources are provisionned under the hood, as illustrated by the following diagram:

After a few minutes, all the resources are ready. ℹ️ The Crossplane CLI now enables many operations, including visualizing the resources of a Composition.

 1kubectl get xsqlinstances
 2NAME                  SYNCED   READY   COMPOSITION                     AGE
 3xplane-harbor-jmdhp   True     True    xsqlinstances.cloud.ogenki.io   8m32s
 4
 5crank beta trace xsqlinstances.cloud.ogenki.io xplane-harbor-jmdhp
 6NAME                                                    SYNCED   READY   STATUS
 7XSQLInstance/xplane-harbor-jmdhp                        True     True    Available
 8├─ SecurityGroupIngressRule/xplane-harbor-jmdhp-n785k   True     True    Available
 9├─ SecurityGroup/xplane-harbor-jmdhp-8jnhc              True     True    Available
10├─ Object/external-service-xplane-harbor                True     True    Available
11├─ Object/providersql-xplane-harbor                     True     True    Available
12├─ Database/registry                                    True     True    Available
13├─ Role/harbor                                          True     True    Available
14├─ Instance/xplane-harbor-jmdhp-whv4g                   True     True    Available
15└─ SubnetGroup/xplane-harbor-jmdhp-fjfth                True     True    Available

Et Voilà! Harbor becomes accessible thanks to Cilium and Gateway API (You can take a look at a previous post on the topic 😉)

 1apiVersion: apiextensions.crossplane.io/v1alpha1
 2kind: EnvironmentConfig
 3metadata:
 4  name: eks-environment
 5data:
 6  clusterName: ${cluster_name}
 7  oidcUrl: ${oidc_issuer_url}
 8  oidcHost: ${oidc_issuer_host}
 9  oidcArn: ${oidc_provider_arn}
10  accountId: ${aws_account_id}
11  region: ${region}
12  vpcId: ${vpc_id}
13  CIDRBlock: ${vpc_cidr_block}
14  privateSubnetIds: ${private_subnet_ids}

 1- name: SecurityGroupIngressRule
 2  base:
 3    apiVersion: ec2.aws.upbound.io/v1beta1
 4    kind: SecurityGroupIngressRule
 5    spec:
 6      forProvider:
 7        cidrIpv4: ""
 8  patches:
 9...
10    - fromFieldPath: CIDRBlock
11      toFieldPath: spec.forProvider.cidrIpv4
12      type: FromEnvironmentFieldPath

🛠️ Composition Functions

Composition Functions represent a significant evolution in the development of Compositions. The traditional way of doing patch and transforms within a composition had certain limitations, such as the inability to use conditions, loops in the code, or to execute advanced functions (e.g., subnet calculations, checking the status of external resources).

Composition Functions overcome these limitations and are essentially programs that extend the templating capabilities of resources within Crossplane. They can be written in any programming language, thus offering huge flexibility and power in defining compositions. This allows for complex tasks such as conditional transformations, iterations, and dynamic operations.

These functions are executed in a sequential manner (in Pipeline mode), with each function manipulating and transforming the resources and then passing the result to the next function, opening the door to powerful combinations.

But let's get back to our RDS composition 🔍! It indeed uses this new way of defining Compositions and consists of three steps:

infrastructure/base/crossplane/configuration/sql-instance-composition.yaml

 1apiVersion: apiextensions.crossplane.io/v1
 2kind: Composition
 3metadata:
 4  name: xsqlinstances.cloud.ogenki.io
 5...
 6spec:
 7  mode: Pipeline
 8...
 9  pipeline:
10    - step: patch-and-transform
11      functionRef:
12        name: function-patch-and-transform
13...
14    - step: sql-go-templating
15      functionRef:
16        name: function-go-templating
17...
18    - step: ready
19      functionRef:
20        name: function-auto-ready

1. The syntax of the first step, patch-and-transform, might look familiar 😉. It is indeed the traditional patching method of Crossplane, but this time executed as a function in the Pipeline.
2. The second step involves calling the function-go-templating function, which we will discuss in more detail shortly.
3. Finally, the last step uses the function-auto-ready function, which checks whether the composite resource (XR) is ready. This means that all the resources composing it have reached the Ready state.

1go install github.com/crossplane-contrib/crossplane-migrator@latest

1crossplane-migrator new-pipeline-composition --function-name crossplane-contrib-function-patch-and-transform -i composition.yaml -o composition-pipeline.yaml

🐹 Go templating in Compositions

As mentioned earlier, the power of Composition Functions lies primarily in the fact that any programming language can be used. For instance, it's possible to generate resources from Go templates with the function-go-templating. Creating Composition with it is not so different from writing Helm Charts.

All you need to do is call the function and provide it with a template as input to generate Kubernetes resources. In the SQLInstance composition, the YAMLs are generated directly inline, but it's also possible to load local files (source: Filesystem).

 1    - step: sql-go-templating
 2      functionRef:
 3        name: function-go-templating
 4      input:
 5        apiVersion: gotemplating.fn.crossplane.io/v1beta1
 6        kind: GoTemplate
 7        source: Inline
 8        inline:
 9          template: |
10          ...          

Then it's your turn to play! For example, there is slight difference in generating a MariaDB or PostgreSQL database, so we can formulate conditions as follows:

1{{- $apiVersion := "" }}
2{{- if eq $parameters.engine "postgres" }}
3  {{- $apiVersion = "postgresql.sql.crossplane.io/v1alpha1" }}
4{{- else }}
5  {{- $apiVersion = "mysql.sql.crossplane.io/v1alpha1" }}
6{{- end -}}

This also allowed me to define a list of databases along with their owner.

 1apiVersion: cloud.ogenki.io/v1alpha1
 2kind: SQLInstance
 3metadata:
 4...
 5spec:
 6  parameters:
 7...
 8    databases:
 9      - owner: owner1
10        name: db1
11      - owner: owner2
12        name: db2

Then, I used Golang loops to create them using the SQL provider.

 1{{- range $parameters.databases }}
 2---
 3apiVersion: {{ $apiVersion }}
 4kind: Database
 5metadata:
 6  name: {{ .name | replace "_" "-" }}
 7  annotations:
 8    {{ setResourceNameAnnotation (print "db-" (replace "_" "-" .name)) }}
 9spec:
10...
11{{- end }}

It is even possible to develop more complex logic in go template functions using the usual define and include directives. Here is an excerpt from the examples available in the function's repository.

1{{- define "labels" -}}
2some-text: {{.val1}}
3other-text: {{.val2}}
4{{- end }}
5...
6labels:
7  {{- include "labels" $vals | nindent 4}}
8...

Finally, we can test the Composition and display the rendering of the template with the following command:

1crank beta render tooling/base/harbor/sqlinstance.yaml infrastructure/base/crossplane/configuration/sql-instance-composition.yaml infrastructure/base/crossplane/configuration/function-go-templating.yaml

As we can see, the possibilities are greatly expanded thanks to the ability to construct resources using a programming language. However, it is also necessary to ensure that the composition remains readable and maintainable in the long term. We will likely witness the emergence of best practices as we gain more experience with the use of these functions.

💭 Final Thoughts

When we talk about Infrastructure As Code, Terraform often comes to mind first. This tool, supported by a vast community, with a mature ecosystem, remains a top choice. However, it's interesting to ponder how Terraform has evolved in response to the new paradigms introduced by Kubernetes. We touched on this in our article on terraform controller. Since then, you may have noticed Hashicorp's controversial decision to adopt the Business Source License. This switch sparked many reactions and might have influenced the strategy and roadmap of other solutions...

Without saying that this is a direct reaction, recently, Crossplane updated its charter to expand its scope to the entire ecosystem (providers, functions), notably by integrating the Upjet project under the CNCF umbrella. The goal of this move is to strengthen the governance of associated projects and ultimately improve the developer experience.

Personally, I've been using Crossplane for a while for specific use cases. I even deployed it in production at a company, using a composition to define specific permissions for pods on EKS (IRSA). We also restricted the types of resources a developer could declare.

❓ So, what to think of this new experience with Crossplane?

It is obvious that Composition Functions promise exciting horizons, and we can expect to see many functions emerge in 2024 🚀

However, imho, it is crucial that development and operation tools continue to improve to foster adoption of the project. For instance, a web interface or a k9s plugin would be useful.

Furthermore, for a beginner looking to develop a composition or a function, the first step might seem daunting. Validating a composition is not straightforward, and there aren't many examples to follow. We hope the marketplace will grow over time.

That said, these concerns are being addressed by the Crossplane community, especially by the SIG Dev XP, whose efforts deserve applause and who are currently doing significant work. 👏

I encourage you to closely follow the project's evolution in the coming months 👀, and to try out Crossplane for yourself to form your own opinion.

🔖 References

• Crossplane blog: Improve Crossplane Compositions Authoring with go-templating-function
• Dev XP Roadmap
• Video (Kubecon NA 2023): Crossplane Intro and Deep Dive - the Cloud Native Control Plane Framework
• Video (DevOps Toolkit): Crossplane Composition Functions: Unleashing the Full Potential