App Definition Authoring for KubeDirector

The application expert preparing an application for KubeDirector deployment is responsible for providing three components: application metadata, a container image (or multiple images), and a setup package. Together these components form an entry in the KubeDirector application catalog.

Metadata

The metadata for an application describes the services the cluster will provide, the "roles" that members (virtual hosts) can take on within the application cluster, the location of the runtime setup package, and the choices available to the user that deploys instances of this application.

Each role will be associated with some container image and a set of services. A role can also have other characteristics such as an allowed number of role members or minimum memory size per member.

A sample metadata document for defining a Cassandra application instance is provided below. Keep in mind that this document is not interpreted by any k8s service, but is instead used by KubeDirector.

This example describes an arrangement that has two or more "seed" role members and a variable number of "worker" role members. Two services are defined, both of which run on the members of either role. The final few properties declare that extra deployment steps must be taken to support systemd within the container and provide certain additional container privileges required by the app; also, if persistent storage is used to deploy this app, the "/data" directory should be placed there (along with other default persisted directories "/usr", "/opt", "/var", and "/etc").

This is a simple app type definition in many respects. Roles can differ from each other in several ways, not only in the services that run on a role member but also in the container image and setup package used within the role. To keep this example straightforward, such role features are not used here. The "config" part of the app type definition can also be elaborated further to define deploy-time choices that affect which roles are active in a given instance and which services run on those roles; this example does not define any such choices.

apiVersion: kubedirector.bluedata.com/v1alpha1
kind: KubeDirectorApp
metadata:
  name: cassandra311
spec:
  label:
    name: Cassandra 3.11
    description: Cassandra 3.11 on CentOS 7.x
  distro_id: bluedata/cassandra311
  version: '3.0'
  image:
    imageRepoTag: docker.io/bluedata/cassandra:3.0
  setup_package:
    config_api_version: 7
    checksum: 3ed586747775a98ef780448addb70d73
    import_url: https://s3.amazonaws.com/bluek8s/kubedirector/cassandra311/appconfig.tgz
  services:
  - id: ssh
    label:
      name: SSH
    endpoint:
      port: '22'
      is_dashboard: false
    id: ssh
  - id: cassandra
    label:
      name: cql
    endpoint:
      url_scheme: cql
      port: '9042'
      is_dashboard: false
  node_roles:
  - id: seed
    cardinality: 2+
  - id: worker
    cardinality: 0+
  config:
    node_services:
    - role_id: seed
      service_ids:
      - ssh
      - cassandra
    - role_id: worker
      service_ids:
      - ssh
      - cassandra
    selected_roles:
    - seed
    - worker
  systemctlMounts: true
  persist_dirs:
  - "/data"
  capabilities:
  - SYS_RESOURCE
  - IPC_LOCK

Container Images

A container image for a role contains the application software already installed and ready to start after some final runtime configuration. Images can be shared among roles, but if the roles provide widely divergent services then it is better to use unique per-role images to minimize image size.

Note that each container in the cluster will be an independent network entity, likely running multiple services on the usual ports. Preparing the image will be very similar to installing the application on a bare-metal host.

The images must be hosted on a container image repository, such as Docker Hub or quay.io, that will be accessible to the k8s nodes.

Setup Package

At runtime, the application will need to be configured with information about the identities of the cluster members and any choices made by the deploying user. KubeDirector will automatically do this via the following steps inside each container:

• Create a KubeDirector config file tailored for that container's role and specific identity, derived from the application metadata, the runtime state of the cluster, and user choices.
• Fetch the application setup package designated by the metadata, and unpack it.
• Run the startscript from the setup package, with arguments that specify an initial configuration. (This script will also be used later to notify when other members are added to or removed from the cluster.)

Therefore the author of the catalog entry must provide that setup package for KubeDirector to use during cluster deployment. The setup package must be hosted at a webserver accessible by the k8s nodes. In the future, other methods of package hosting may be possible.

An edited example of a startscript for the Cassandra application is provided below. Note that the setup package does not only contain this script; it will also contain some standard shell macro/utility definitions for reading the KubeDirector config file, and any application config file templates or other artifacts that the startscript will make use of.

#!/bin/bash

# The script flow will initially diverge depending on whether arguments
# indicate that it was invoked for initial configuration, or as a notification
# of other members entering/leaving the cluster. We'll assume initial
# configuration in this example.

source macros.sh # utilities common for all app setups

# Edit various app config file templates. The all-caps macros such as
# GET_FQDN_LIST provide info from this container's KubeDirector-provided
# configuration info.

REPLACE_PATTERN @@@@SEEDS@@@@ /etc/cassandra/default.conf/cassandra.yaml GET_FQDN_LIST seed

REPLACE_PATTERN @@@@LISTEN_ADDRESS@@@@ /etc/cassandra/default.conf/cassandra.yaml GET_NODE_FQDN

REPLACE_PATTERN @@@@CONCURRENT_COMPACTORS@@@@ /etc/cassandra/default.conf/cassandra.yaml GET_TOTAL_VCORES

# etc. ...

# Start services appropriate to this member's role and the deployment choices.
# This could use macros to determine the current member's role and the
# appropriate services for that role; in this case however we know that both
# the seed role and the worker role always want to run the cassandra service.

REGISTER_START_SERVICE_SYSCTL cassandra cassandra