heidloff.net - Building is my Passion
Post
Cancel

Scaling Applications automatically with Operators

The real power or Kubernetes operators are not day 1 tasks like the initial deployments, but the automation of day 2 operations. This article describes a sample operator that scales up an application automatically based on the number of API requests.

The complete source code from this article is available in the ibm/operator-sample-go repo. The repo includes operator samples that demonstrate patterns and best practises. It also includes another day 2 sample scenario: Automatically Archiving Data with Kubernetes Operators.

The sample contains the following components:

  • Prometheus: Stores metrics from various sources and provides query capabilities
  • Sample microservice: Provides a /hello endpoint which exposes a counter to Prometheus
  • Application operator (core): Deploys the microservice
  • Application operator’s CronJob: Separate container which scales up the number of pod instances based on the amount of /hello invocations

To set up and configure Prometheus, check our my previous article Exporting Metrics from Kubernetes Apps for Prometheus. Below I focus on the implementation of the auto-scaler.

The microservice has been implemented with Quarkus. It uses Eclipse MicroProfile to track the number of invocations (see code).

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
import org.eclipse.microprofile.config.inject.ConfigProperty;
import org.eclipse.microprofile.metrics.annotation.Counted;

@Path("/hello")
public class GreetingResource {
  @ConfigProperty(name = "greeting.message") 
  String message;

  @GET
  @Produces(MediaType.TEXT_PLAIN)
  @Counted(name = "countHelloEndpointInvoked", description = "How often /hello has been invoked")
  public String hello() {
    return String.format("Hello %s", message);        
  }
}

To allow Prometheus to scrape these metrics, a ServiceMonitor is used.

1
2
3
4
5
6
7
8
9
10
11
12
13
apiVersion: monitoring.coreos.com/v1
kind: ServiceMonitor
metadata:
  labels:
    app: myapplication
  name: myapplication-metrics-monitor
  namespace: application-beta
spec:
  endpoints:
    - path: /q/metrics
  selector:
    matchLabels:
      app: myapplication

With the Prometheus user interface queries to this data can be done.

image

To develop the auto-scaler, a separate image/container is used. This container is an extension to the application controller. The application controller sets up a CronJob for the auto-scaler container so that it is run on a scheduled basis. The CronJob that is created by the controller looks like this. Note that the application name and namespace are passed in as parameter.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
apiVersion: batch/v1
kind: CronJob
metadata:
  name: application-scaler
  namespace: operator-application-system
spec:
  schedule: "0 * * * *"
  jobTemplate:
    spec:
      template:
        spec:
          containers:
          - name: application-scale
            image: docker.io/nheidloff/operator-application-scaler:v1.0.2
            imagePullPolicy: IfNotPresent
            env:
            - name: APPLICATION_RESOURCE_NAME
              value: "application"
            - name: APPLICATION_RESOURCE_NAMESPACE
              value: "application-beta"
          restartPolicy: OnFailure

The implementation of the actual auto-scaler is trivial. I’ve used the Prometheus Go client library. Note that this library is still considered experimental. Alternatively you can use the Prometheus HTTP API.

1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
prometheusAddress := "http://prometheus-operated.monitoring:9090"
queryAmountHelloEndpointInvocations := "application_net_heidloff_GreetingResource_countHelloEndpointInvoked_total"
client, err := api.NewClient(api.Config{
  Address: prometheusAddress,
})
if err != nil {
  os.Exit(1)
}
v1api := v1.NewAPI(client)
ctx, cancel := context.WithTimeout(context.Background(), 10*time.Second)
defer cancel()
result, warnings, err := v1api.Query(ctx, queryAmountHelloEndpointInvocations, time.Now())
if err != nil {
  os.Exit(1)
}
resultVector, conversionSuccessful := (result).(model.Vector)
if conversionSuccessful == true {
  if resultVector.Len() > 0 {
    firstElement := resultVector[0]
    if firstElement.Value > 5 {
      // Note: '5' is only used for demo purposes
      scaleUp()
    } 
  }
}

To learn more about operator patterns and best practices, check out the repo operator-sample-go. The instructions how to run the auto-scaler demo are in the documentation.

Featured Blog Posts
Disclaimer
The postings on this site are my own and don't necessarily represent my employer IBM's positions, strategies or opinions.
Trending Tags