Going Further with Crossplane
: Compositions and Functions
Overview
I'm now using the KCL (Kusion Configuration Language) for crossplane compositions.
With the emergence of Platform Engineering, we are witnessing a shift towards the creation of self-service solutions for developers. This approach facilitates the standardization of DevOps practices, enhances the developer experience, and reduces the cognitive load associated with managing tools.
Crossplane
, an "Incubating" project under the Cloud Native Computing Foundation (CNCF), aims to become the leading framework for creating Cloud Native platforms. In my first article about Crossplane, I introduced this tool and explained how it leverages GitOPs principles for infrastructure, enabling the creation of a GKE
cluster.
Now celebrating its 5th anniversary ππ, the project has matured and expanded its features over time.
In this post, we will explore some of Crossplane's key features, with a particular focus on the Composition Functions that are generating significant interest within the community. Are we about to witness a pivotal moment for the project?
π― Our target
The Crossplane documentation is comprehensive, so we'll quickly review the basic concepts to focus on a specific use case: Deploying Harbor on an EKS cluster, adhering to high availability best practices.
Harbor, also from the CNCF, is a security-focused container artifact management solution.
Its primary role is to store, sign, and scan for vulnerabilities container images. Harbor features fine-grained access control, an API, and a web interface, allowing dev teams to access and manage their own images easily.
Harbor's availability mainly depends on its stateful components. Users are responsible for their implementation, which should be tailored to the target infrastructure. This blog post presents the options I selected for optimal availability.
- Redis deployed using the Bitnami Helm chart in "master/slave" mode.
- Artifacts are stored in an AWS S3 bucket.
- A PostgreSQL RDS instance for the database.
We will now explore how Crossplane simplifies the provisioning of this RDS instance by providing a level of abstraction that exposes only an opinionated set of options. π
ποΈ Prerequisites
Before we can build our Compositions, some groundwork is necessary as several preliminary operations need to be carried out. These steps are performed in a specific order:
- Deployment of Crossplane's core controller using the Helm chart.
- Installation of the
providers
and their configurations. - Deployment of various configurations using the previously installed providers, especially the
Compositions
andComposition Functions
. - Declarations of Claims to consume the Compositions.
These steps are described through Flux's dependencies and can be viewed here.
All the actions carried out in this article come from this git repository.
There you can find numerous sources that help me construct my blog posts. π π Feedback is a gift π
π¦ The Compositions
Put simply, a Composition
in Crossplane is a way to aggregate and automatically manage multiple resources whose configuration can sometimes be complex.
It leverages the Kubernetes API to define and orchestrate not just infrastructure elements like storage and networking, but also various other components (refer to the list of providers). This method provides developers with a simplified interface, representing an abstraction layer that hides the more complex technical details of the underlying infrastructure.
To achieve my goal of creating Harbor's database, I first looked for a relevant example. For this purpose, I used the Upbound marketplace, where a few Compositions can be found that can be used as starting points.
Based on the configuration-rds composition, I wanted to add the following elements:
- π Allow pods to access the instance.
- βΆοΈ Creation of a ExternalName Kubernetes service with a predictable name that will be used in Harbor's configuration.
- πΎ Creation of databases and the roles that will own them.
β How would this Composition then be used if, for example, a developer wishes to have a database? They is simply done by declaring a Claim which represents the level of abstraction exposed to the users. Let's get a closer look π
tooling/base/harbor/sqlinstance.yaml
1apiVersion: cloud.ogenki.io/v1alpha1
2kind: SQLInstance
3metadata:
4 name: xplane-harbor
5 namespace: tooling
6spec:
7 parameters:
8 engine: postgres
9 engineVersion: "15"
10 size: small
11 storageGB: 20
12 databases:
13 - owner: harbor
14 name: registry
15 passwordSecretRef:
16 namespace: tooling
17 name: harbor-pg-masterpassword
18 key: password
19 compositionRef:
20 name: xsqlinstances.cloud.ogenki.io
21 writeConnectionSecretToRef:
22 name: xplane-harbor-rds
Here we observe that it boils down to a simple resource with few parameters to express our needs:
- A PostgreSQL instance version 15 will be created.
- The instance type is at the discretion of the platform team (the maintainers of the composition). In the above
Claim
, we ask for a small instance, which is interpreted by the composition asdb.t3.small
. infrastructure/base/crossplane/configuration/sql-instance-composition.yaml
1transforms:
2 - type: map
3 map:
4 large: db.t3.large
5 medium: db.t3.medium
6 small: db.t3.small
- The
master
user's password is retrieved from aharbor-pg-masterpassword
secret, retrieved from an External Secret. - Once the instance is created, the connection details are stored in a secret
xplane-harbor-rds
.
This is where we can fully appreciate the power of Crossplane Compositions! Indeed, many resources are provisionned under the hood, as illustrated by the following diagram:
After a few minutes, all the resources are ready. βΉοΈ The Crossplane CLI now enables many operations, including visualizing the resources of a Composition.
1kubectl get xsqlinstances
2NAME SYNCED READY COMPOSITION AGE
3xplane-harbor-jmdhp True True xsqlinstances.cloud.ogenki.io 8m32s
4
5crank beta trace xsqlinstances.cloud.ogenki.io xplane-harbor-jmdhp
6NAME SYNCED READY STATUS
7XSQLInstance/xplane-harbor-jmdhp True True Available
8ββ SecurityGroupIngressRule/xplane-harbor-jmdhp-n785k True True Available
9ββ SecurityGroup/xplane-harbor-jmdhp-8jnhc True True Available
10ββ Object/external-service-xplane-harbor True True Available
11ββ Object/providersql-xplane-harbor True True Available
12ββ Database/registry True True Available
13ββ Role/harbor True True Available
14ββ Instance/xplane-harbor-jmdhp-whv4g True True Available
15ββ SubnetGroup/xplane-harbor-jmdhp-fjfth True True Available
Et VoilΓ ! Harbor becomes accessible thanks to Cilium and Gateway API (You can take a look at a previous post on the topic π)
The EnvironmentConfigs enable the use of cluster-specific variables. These configuration elements are loaded into memory and can then be used within the composition.
Since the EKS cluster is created with Opentofu, we store its properties using Flux variables. (more info on Flux's variables substitution here)
infrastructure/base/crossplane/configuration/environmentconfig.yaml
1apiVersion: apiextensions.crossplane.io/v1alpha1
2kind: EnvironmentConfig
3metadata:
4 name: eks-environment
5data:
6 clusterName: ${cluster_name}
7 oidcUrl: ${oidc_issuer_url}
8 oidcHost: ${oidc_issuer_host}
9 oidcArn: ${oidc_provider_arn}
10 accountId: ${aws_account_id}
11 region: ${region}
12 vpcId: ${vpc_id}
13 CIDRBlock: ${vpc_cidr_block}
14 privateSubnetIds: ${private_subnet_ids}
These variables can then be used in Compositions via the FromEnvironmentFieldPath
directive. For instance, to allow pods to access our RDS instance, we allow the VPC's CIDR as follows:
infrastructure/base/crossplane/configuration/irsa-composition.yaml
1- name: SecurityGroupIngressRule
2 base:
3 apiVersion: ec2.aws.upbound.io/v1beta1
4 kind: SecurityGroupIngressRule
5 spec:
6 forProvider:
7 cidrIpv4: ""
8 patches:
9...
10 - fromFieldPath: CIDRBlock
11 toFieldPath: spec.forProvider.cidrIpv4
12 type: FromEnvironmentFieldPath
β οΈ As of the time of writing this post, the feature is still in alpha.
π οΈ Composition Functions
Composition Functions
represent a significant evolution in the development of Compositions. The traditional way of doing patch and transforms within a composition had certain limitations, such as the inability to use conditions, loops in the code, or to execute advanced functions (e.g., subnet calculations, checking the status of external resources).
Composition Functions overcome these limitations and are essentially programs that extend the templating capabilities of resources within Crossplane. They can be written in any programming language, thus offering huge flexibility and power in defining compositions. This allows for complex tasks such as conditional transformations, iterations, and dynamic operations.
These functions are executed in a sequential manner (in Pipeline
mode), with each function manipulating and transforming the resources and then passing the result to the next function, opening the door to powerful combinations.
But let's get back to our RDS composition π! It indeed uses this new way of defining Compositions
and consists of three steps:
infrastructure/base/crossplane/configuration/sql-instance-composition.yaml
1apiVersion: apiextensions.crossplane.io/v1
2kind: Composition
3metadata:
4 name: xsqlinstances.cloud.ogenki.io
5...
6spec:
7 mode: Pipeline
8...
9 pipeline:
10 - step: patch-and-transform
11 functionRef:
12 name: function-patch-and-transform
13...
14 - step: sql-go-templating
15 functionRef:
16 name: function-go-templating
17...
18 - step: ready
19 functionRef:
20 name: function-auto-ready
- The syntax of the first step,
patch-and-transform
, might look familiar π. It is indeed the traditional patching method of Crossplane, but this time executed as a function in the Pipeline. - The second step involves calling the function-go-templating function, which we will discuss in more detail shortly.
- Finally, the last step uses the function-auto-ready function, which checks whether the composite resource (
XR
) is ready. This means that all the resources composing it have reached theReady
state.
If you already have Compositions in the previous format (Patch & Transforms), there is a great tool available for migrating to the Pipeline
mode: crossplane-migrator
- Install crossplane-migrator
1go install github.com/crossplane-contrib/crossplane-migrator@latest
- Then execute the following command, which will generate the correct format in
composition-pipeline.yaml
1crossplane-migrator new-pipeline-composition --function-name crossplane-contrib-function-patch-and-transform -i composition.yaml -o composition-pipeline.yaml
βΉοΈ This capabilitiy should be added to the Crossplane CLI in the next release (v1.15)
πΉ Go templating in Compositions
As mentioned earlier, the power of Composition Functions lies primarily in the fact that any programming language can be used. For instance, it's possible to generate resources from Go templates with the function-go-templating. Creating Composition with it is not so different from writing Helm Charts.
All you need to do is call the function and provide it with a template as input
to generate Kubernetes resources. In the SQLInstance composition, the YAMLs are generated directly inline, but it's also possible to load local files (source: Filesystem
).
1 - step: sql-go-templating
2 functionRef:
3 name: function-go-templating
4 input:
5 apiVersion: gotemplating.fn.crossplane.io/v1beta1
6 kind: GoTemplate
7 source: Inline
8 inline:
9 template: |
10 ...
Then it's your turn to play! For example, there is slight difference in generating a MariaDB or PostgreSQL database, so we can formulate conditions as follows:
1{{- $apiVersion := "" }}
2{{- if eq $parameters.engine "postgres" }}
3 {{- $apiVersion = "postgresql.sql.crossplane.io/v1alpha1" }}
4{{- else }}
5 {{- $apiVersion = "mysql.sql.crossplane.io/v1alpha1" }}
6{{- end -}}
This also allowed me to define a list of databases along with their owner.
1apiVersion: cloud.ogenki.io/v1alpha1
2kind: SQLInstance
3metadata:
4...
5spec:
6 parameters:
7...
8 databases:
9 - owner: owner1
10 name: db1
11 - owner: owner2
12 name: db2
Then, I used Golang loops to create them using the SQL provider.
1{{- range $parameters.databases }}
2---
3apiVersion: {{ $apiVersion }}
4kind: Database
5metadata:
6 name: {{ .name | replace "_" "-" }}
7 annotations:
8 {{ setResourceNameAnnotation (print "db-" (replace "_" "-" .name)) }}
9spec:
10...
11{{- end }}
It is even possible to develop more complex logic in go template functions using the usual define
and include
directives. Here is an excerpt from the examples available in the function's repository.
1{{- define "labels" -}}
2some-text: {{.val1}}
3other-text: {{.val2}}
4{{- end }}
5...
6labels:
7 {{- include "labels" $vals | nindent 4}}
8...
Finally, we can test the Composition and display the rendering of the template with the following command:
1crank beta render tooling/base/harbor/sqlinstance.yaml infrastructure/base/crossplane/configuration/sql-instance-composition.yaml infrastructure/base/crossplane/configuration/function-go-templating.yaml
As we can see, the possibilities are greatly expanded thanks to the ability to construct resources using a programming language. However, it is also necessary to ensure that the composition remains readable and maintainable in the long term. We will likely witness the emergence of best practices as we gain more experience with the use of these functions.
π Final Thoughts
When we talk about Infrastructure As Code, Terraform often comes to mind first. This tool, supported by a vast community, with a mature ecosystem, remains a top choice. However, it's interesting to ponder how Terraform has evolved in response to the new paradigms introduced by Kubernetes. We touched on this in our article on terraform controller. Since then, you may have noticed Hashicorp's controversial decision to adopt the Business Source License. This switch sparked many reactions and might have influenced the strategy and roadmap of other solutions...
Without saying that this is a direct reaction, recently, Crossplane
updated its charter to expand its scope to the entire ecosystem (providers, functions), notably by integrating the Upjet project under the CNCF umbrella. The goal of this move is to strengthen the governance of associated projects and ultimately improve the developer experience.
Personally, I've been using Crossplane
for a while for specific use cases. I even deployed it in production at a company, using a composition to define specific permissions for pods on EKS (IRSA). We also restricted the types of resources a developer could declare.
β So, what to think of this new experience with Crossplane?
It is obvious that Composition Functions promise exciting horizons, and we can expect to see many functions emerge in 2024 π
However, imho, it is crucial that development and operation tools continue to improve to foster adoption of the project. For instance, a web interface or a k9s plugin would be useful.
Furthermore, for a beginner looking to develop a composition or a function, the first step might seem daunting. Validating a composition is not straightforward, and there aren't many examples to follow. We hope the marketplace will grow over time.
That said, these concerns are being addressed by the Crossplane community, especially by the SIG Dev XP, whose efforts deserve applause and who are currently doing significant work. π
I encourage you to closely follow the project's evolution in the coming months π, and to try out Crossplane for yourself to form your own opinion.
π References
- Crossplane blog: Improve Crossplane Compositions Authoring with go-templating-function
- Dev XP Roadmap
- Video (Kubecon NA 2023): Crossplane Intro and Deep Dive - the Cloud Native Control Plane Framework
- Video (DevOps Toolkit): Crossplane Composition Functions: Unleashing the Full Potential