Skip to content

Latest commit

 

History

History
214 lines (161 loc) · 12.8 KB

README.md

File metadata and controls

214 lines (161 loc) · 12.8 KB
Raw

KEDA External Scaler for Azure Cosmos DB

This directory contains the source code for two sample projects to demonstrate the functionality of the external scaler.

  • Keda.CosmosDb.Scaler.Demo.OrderGenerator - used to push fake purchase orders to a test Cosmos DB container.
  • Keda.CosmosDb.Scaler.Demo.OrderProcessor - used to read these orders from the container and process them.

We will later deploy the order-processor application to Kubernetes cluster and use KEDA along with the external scaler from Keda.CosmosDb.Scaler project to scale the application.

Prerequisites

Testing sample application locally on Docker

  1. Open command prompt or shell and change to the root directory of the cloned repo.

  2. Run the below commands to build the Docker container images for order-generator and order-processor applications.

    # docker build --file .\src\Scaler.Demo\OrderGenerator\Dockerfile --force-rm --tag cosmosdb-order-generator .
# docker build --file .\src\Scaler.Demo\OrderProcessor\Dockerfile --force-rm --tag cosmosdb-order-processor .

  3. Create test-database and test-container within the database in Cosmos DB account by running the order-generator application inside the container with setup option. Make sure to put the connection string of Cosmos DB account in the command below.

    # docker run --env CosmosDbConfig__Connection="<connection-string>" --interactive --rm --tty cosmosdb-order-generator setup

    Caution The default application settings provision a throughput of 11,000 RU/s to the test Cosmos DB container. This sets the number of its physical partitions to 2. If you have a free Azure account which offers limited throughput, or if you want to limit the cost of running the sample, be sure to update the throughput to 400 RU/s by setting the value of property CosmosDb:ConnectionThroughput to 400 in file src/Scaler.Demo/OrderGenerator/appsettings.json before building the container image. That would still allow testing of KEDA scaling between 0 and 1 instances but not upto 2 instances. Also, at any point, you can run order-generator with teardown option to delete the database and Cosmos DB container inside.

  4. Start a second shell instance and run the order-processor application in a new container. You can put the same connection string in both places in the command below. Note that the sample applications are written to handle different Cosmos DB accounts for monitored and lease containers but having two different accounts is not a requirement.

    # docker run --env CosmosDbConfig__Connection="<connection-string>" --env CosmosDbConfig__LeaseConnection="<connection-string>" --interactive --rm --tty cosmosdb-order-processor

    The order-processor application will create lease database and container if they do not exist. The default application settings would share the same database between the monitored and lease containers. The order-processor application will then activate a change-feed processor to monitor and process new changes in the monitored container.

  5. Keep the order-processor application running. Go back to the first shell instance and run order-generator application with generate option to add fake orders to the test Cosmos DB container.

    # docker run --env CosmosDbConfig__Connection="<connection-string>" --interactive --tty --rm cosmosdb-order-generator generate
Let's queue some orders, how many do you want? 3
Do you want to limit orders to single article (to put them in a single partition)? (Y/N) N
Creating order 1f157b6e-c51f-4e02-a492-295e7fd47fbe - 3 unit(s) of Mouse for Cleta Padberg
Creating order ffeb4822-43e6-4b80-a439-d2ad09a278c1 - 9 unit(s) of Soap for Uriel Cormier
Creating order b0e396d7-e140-43f5-872c-f47df6bccf2e - 4 unit(s) of Chair for Granville Auer
That's it, see you later!

  6. Go back to the second shell where the order-processor application is running. Check the console output and verify that the orders were processed.

    2021-09-03 06:52:34 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Started change feed processor instance Instance-588a09449616
2021-09-03 06:52:34 info: Microsoft.Hosting.Lifetime[0]
    Application started. Press Ctrl+C to shut down.
2021-09-03 06:52:34 info: Microsoft.Hosting.Lifetime[0]
    Hosting environment: Production
2021-09-03 06:52:34 info: Microsoft.Hosting.Lifetime[0]
    Content root path: /app
2021-09-03 06:52:59 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    1 orders received
2021-09-03 06:52:59 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Processing order e5912b0a-939b-45bc-87de-9baa314ca65e - 9 unit(s) of Soap bought by Uriel Cormier
2021-09-03 06:53:00 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    2 orders received
2021-09-03 06:53:00 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Processing order 6daae206-027c-4055-8d39-97274de1ab59 - 3 unit(s) of Mouse bought by Cleta Padberg
2021-09-03 06:53:01 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Order e5912b0a-939b-45bc-87de-9baa314ca65e processed
2021-09-03 06:53:02 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Order 6daae206-027c-4055-8d39-97274de1ab59 processed
2021-09-03 06:53:02 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Processing order 135e1fb9-807a-489e-90f0-0cb4c3768a36 - 4 unit(s) of Chair bought by Granville Auer
2021-09-03 06:53:04 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Order 135e1fb9-807a-489e-90f0-0cb4c3768a36 processed

  7. Stop order-processor container from the first shell.

    # docker container ls --no-trunc --format "{{.Image}} {{.Names}}"
cosmosdb-order-processor jolly_wilbur
# docker container stop jolly_wilbur
jolly_wilbur

Deploying KEDA and external scaler to cluster

  1. Follow one of the steps on Deploying KEDA documentation page to deploy KEDA on your Kubernetes cluster.

  2. Open command prompt or shell and change to the root directory of the cloned repo.

  3. Build container image for the external scaler and push the image to Docker Hub. Make sure to replace <docker-id> in below commands with your Docker ID.

    # docker build --file .\src\Scaler\Dockerfile --force-rm --tag cosmosdb-scaler .
# docker login --username <docker-id>
# docker tag cosmosdb-scaler:latest <docker-id>/cosmosdb-scaler:latest
# docker push <docker-id>/cosmosdb-scaler:latest

  4. Update your Docker ID in the image path in manifest file src/Scaler/deploy.yaml and apply it to deploy the external scaler application.

    kubectl apply --filename=src/Scaler/deploy.yaml

Deploying sample application to cluster

  1. Build the Docker container image for order-processor application (if you haven't already) and push the container image to Docker Hub. Make sure to replace <docker-id> in below commands with your Docker ID.

    # docker build --file .\src\Scaler.Demo\OrderProcessor\Dockerfile --force-rm --tag cosmosdb-order-processor .
# docker tag cosmosdb-order-processor:latest <docker-id>/cosmosdb-order-processor:latest
# docker push <docker-id>/cosmosdb-order-processor:latest

  2. Update your Docker ID in the image path in manifest file src/Scaler.Demo/OrderProcessor/deploy.yaml. Also, update the values of connection strings to point to the test Cosmos DB account. Apply the manifest to deploy the order-processor application.

    kubectl apply --filename=src/Scaler.Demo/OrderProcessor/deploy.yaml

  3. Ensure that the order-processor application is running correctly on the cluster by checking application logs. The application will create lease database and container if they do not exist, hence it is needed to run for a few seconds before we enable auto-scaling for it, as that would immediately bring replicas to 0 if there are no orders pending to be processed.

    # kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
cosmosdb-order-processor-b59956989-bcbzg   1/1     Running   0          50s
cosmosdb-scaler-64dd48678c-zjcgd           1/1     Running   0          23m
# kubectl logs cosmosdb-order-processor-b59956989-bcbzg
2021-09-03 08:05:01 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Started change feed processor instance Instance-cosmosdb-order-processor-b59956989-bcbzg
...

  4. In the manifest file for scaled object, src/Scaler.Demo/OrderProcessor/deploy-scaledobject.yaml, update the values of connection strings to point to the test Cosmos DB account and apply it.

    kubectl apply --filename=src/Scaler.Demo/OrderProcessor/deploy-scaledobject.yaml

    Note Ideally, we would have created TriggerAuthentication resource that would enable sharing of the connection strings as secrets between the scaled object and the target application. However, this is not possible since at the moment, the triggers of external type do not support referencing a TriggerAuthentication resource (link).

Testing auto-scaling for sample application

  1. Verify that there are no pods for order-processor running after the scaled object was created.

    # kubectl get pods
NAME                               READY   STATUS    RESTARTS   AGE
cosmosdb-scaler-64dd48678c-d6dqq   1/1     Running   0          10m

  2. Add new orders to the Cosmos DB container by running the order-generator. Select option to generate all orders for the same article.

    # docker run --env CosmosDbConfig__Connection="<connection-string>" --interactive --tty --rm cosmosdb-order-generator generate
Let's queue some orders, how many do you want? 150
Do you want to limit orders to single article (to put them in a single partition)? (Y/N) Y
...

    This would restrict the orders to a single logical partition (and a single physical partition thereof). The external scaler scales the targets according to the number of change feeds that have non-zero pending messages remaining to be processed. The total number of change feeds (with or without pending messages) equals the number of physical partitions in the Cosmos DB container.

  3. Verify that only one pod is created for the order-processor. It may take a few seconds for the pod to show up.

    # kubectl get pods
NAME                                       READY   STATUS    RESTARTS   AGE
cosmosdb-order-processor-b59956989-fscsb   1/1     Running   0          30s
cosmosdb-scaler-64dd48678c-d6dqq           1/1     Running   0          20m

  4. Now, add more orders to the Cosmos DB container but this time, select option to generate orders of different articles.

    # docker run --env CosmosDbConfig__Connection="<connection-string>" --interactive --tty --rm cosmosdb-order-generator generate
Let's queue some orders, how many do you want? 250
Do you want to limit orders to single article (to put them in a single partition)? (Y/N) N
...

  5. Verify that two pods are created for the order-processor.

    NAME                                       READY   STATUS    RESTARTS   AGE
cosmosdb-order-processor-b59956989-88dc5   1/1     Running   0          15s
cosmosdb-order-processor-b59956989-dxp4b   1/1     Running   0          3s
cosmosdb-scaler-64dd48678c-d6dqq           1/1     Running   0          35m

  6. You can also verify that both order-processor pods are able to share the processing of orders.

    # kubectl logs cosmosdb-order-processor-b59956989-dxp4b --tail=4
2021-09-03 12:57:41 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Order 5ba7f503-0185-49f6-9fce-3da999464049 processed
2021-09-03 12:57:41 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Processing order ce1f05ad-08ff-4535-858f-3158de41971b - 8 unit(s) of Computer bought by Jaren Tremblay

# kubectl logs cosmosdb-order-processor-b59956989-88dc5 --tail=4
2021-09-03 12:57:53 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Order e881c998-1318-411e-8181-fa638335910e processed
2021-09-03 12:57:53 info: Keda.CosmosDb.Scaler.Demo.OrderProcessor.Worker[0]
    Processing order ca17597f-7aa2-4b04-abd8-724139b2c370 - 1 unit(s) of Gloves bought by Donny Shanahan