Custom Kubernetes resources typically have multiple versions. Operators need to be able to convert between all different versions in all directions. This article describes how to implement this using a simple example.
As applications evolve, custom resource definitions need to be extended. As for every API these changes need to be upwards compatible. Additionally the information from the newer versions needs also to be stored in older versions. This is why conversions need to be done in BOTH directions without loosing information.
This allows Kubernetes to provide the following functionality. See the documentation Versions in CustomResourceDefinitions for details.
- Custom resource is requested in a different version than stored version.
- Watch is created in one version but the changed object is stored in another version.
- Custom resource PUT request is in a different version than storage version.
The best documentation I’ve found about conversion comes from Kubebuilder:
Let’s look at a concrete example. I’m working on a GitHub repo that describes various operator patterns and best practises. There is a custom resource ‘Application’ which has two version: The intial v1alpha1 version and the latest version v1beta1.
This is a resource using the alpha version:
1
2
3
4
5
6
7
8
9
10
apiVersion: application.sample.ibm.com/v1alpha1
kind: Application
metadata:
name: application
namespace: application-alpha
spec:
version: "1.0.0"
amountPods: 1
databaseName: database
databaseNamespace: database
The beta version has one additional property ‘title’.
1
2
3
4
5
6
7
8
9
10
11
apiVersion: application.sample.ibm.com/v1beta1
kind: Application
metadata:
name: application
namespace: application-beta
spec:
version: "1.0.0"
amountPods: 1
databaseName: database
databaseNamespace: database
title: Movies
Once deployed, the application resource can be read via the following kubectl commands. By default the latest version is returned.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
$ kubectl get applications/application -n application-beta -oyaml
or
$ kubectl get applications.v1beta1.application.sample.ibm.com/application -n application-beta -oyaml
apiVersion: application.sample.ibm.com/v1beta1
kind: Application
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: ...
...
spec:
amountPods: 1
databaseName: database
databaseNamespace: database
title: Movies
version: 1.0.0
You can also request a specific version, in this case the alpha version from the application-alpha resource. In the sample the ‘title’ is missing since it wasn’t part of the resource when it was created.
1
2
3
4
5
6
7
8
9
10
11
12
$ kubectl get applications.v1alpha1.application.sample.ibm.com/application -n application-alpha -oyaml
apiVersion: application.sample.ibm.com/v1alpha1
kind: Application
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: ...
...
spec:
amountPods: 1
databaseName: database
databaseNamespace: database
version: 1.0.0
Furthermore you can request the beta version of the application-alpha resource. In this case there is a title which has the value ‘Undefined’ since it was not set initially.
1
2
3
4
5
6
7
8
9
10
11
12
13
$ kubectl get applications.v1beta1.application.sample.ibm.com/application -n application-alpha -oyaml | grep -A6 -e "spec:" -e "apiVersion: application.sample.ibm.com/"
apiVersion: application.sample.ibm.com/v1beta1
kind: Application
metadata:
annotations:
kubectl.kubernetes.io/last-applied-configuration: ...
...
spec:
amountPods: 1
databaseName: database
databaseNamespace: database
title: Undefined
version: 1.0.0
You can even request the application-beta resource in the alpha version. In this case the title can not be stored in the ‘spec’ part. The trick is to use annotations. Annotations are part of every resource in the metadata section. They are basically a ‘generic schema’ which name/values pairs.
1
2
3
4
5
6
7
8
9
10
11
12
13
$ kubectl get applications.v1alpha1.application.sample.ibm.com/application -n application-beta -oyaml | grep -A6 -e "spec:" -e "apiVersion: application.sample.ibm.com/"
apiVersion: application.sample.ibm.com/v1alpha1
kind: Application
metadata:
annotations:
applications.application.sample.ibm.com/title: Movies
kubectl.kubernetes.io/last-applied-configuration: ...
...
spec:
amountPods: 1
databaseName: database
databaseNamespace: database
version: 1.0.0
Next let me describe how to implement this scenario. First you need to define which of the versions should be used to store the resources in etcd via ‘+kubebuilder:storageversion’ (code).
1
2
3
4
5
6
7
8
9
//+kubebuilder:object:root=true
//+kubebuilder:subresource:status
//+kubebuilder:storageversion
type Application struct {
metav1.TypeMeta `json:",inline"`
metav1.ObjectMeta `json:"metadata,omitempty"`
Spec ApplicationSpec `json:"spec,omitempty"`
Status ApplicationStatus `json:"status,omitempty"`
}
Next you need to define which of the versions is your hub. All other ones are spokes. See the Kubebuilder documentation. I’ve defined the latest as hub which only contains the empty Hub() function (code).
1
2
package v1beta1
func (*Application) Hub() {}
Next the spokes need to implement ConvertTo() and ConvertFrom(). Here is the ConvertFrom() function that converts from the latest to the initial version.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
// convert from the hub version (src= v1beta1) to this version (dst = v1alpha1)
func (dst *Application) ConvertFrom(srcRaw conversion.Hub) error {
src := srcRaw.(*v1beta1.Application)
dst.ObjectMeta = src.ObjectMeta
dst.Status.Conditions = src.Status.Conditions
dst.Spec.AmountPods = src.Spec.AmountPods
dst.Spec.DatabaseName = src.Spec.DatabaseName
dst.Spec.DatabaseNamespace = src.Spec.DatabaseNamespace
dst.Spec.SchemaUrl = src.Spec.SchemaUrl
dst.Spec.Version = src.Spec.Version
if dst.ObjectMeta.Annotations == nil {
dst.ObjectMeta.Annotations = make(map[string]string)
}
dst.ObjectMeta.Annotations[variables.ANNOTATION_TITLE] = string(src.Spec.Title)
return nil
}
And here is the ConvertTo() function that converts from the initial to the latest version.
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
// convert this version (src = v1alpha1) to the hub version (dst = v1beta1)
func (src *Application) ConvertTo(dstRaw conversion.Hub) error {
dst := dstRaw.(*v1beta1.Application)
dst.Spec.AmountPods = src.Spec.AmountPods
dst.Spec.DatabaseName = src.Spec.DatabaseName
dst.Spec.DatabaseNamespace = src.Spec.DatabaseNamespace
dst.Spec.SchemaUrl = src.Spec.SchemaUrl
dst.Spec.Version = src.Spec.Version
if src.ObjectMeta.Annotations == nil {
dst.Spec.Title = variables.DEFAULT_ANNOTATION_TITLE
} else {
title, annotationFound := src.ObjectMeta.Annotations[variables.ANNOTATION_TITLE]
if annotationFound {
dst.Spec.Title = title
} else {
dst.Spec.Title = variables.DEFAULT_ANNOTATION_TITLE
}
}
dst.ObjectMeta = src.ObjectMeta
dst.Status.Conditions = src.Status.Conditions
return nil
}
The implementation of the conversion webhooks is rather straight forward. The setup of the webhooks is a little bit more tricky. Check out my earlier blog Configuring Webhooks for Kubernetes Operators.
Try the sample operator which demonstrates the capabilities outlined above as well as many other operator patterns.