Skip to content

cfcontainerizationbot/cf-operator

 
 

Repository files navigation

cf-operator

godoc master go report card Coveralls github

Nightly build nightly

cf-operator enables the deployment of BOSH Releases, especially Cloud Foundry, to Kubernetes.

It's implemented as a k8s operator, an active controller component which acts upon custom k8s resources.

Features

cf-operator deploys dockerized BOSH releases onto existing Kubernetes cluster

  • Supports operations files to modify manifest
  • Service instance groups become pods, each job in one container
  • Errand instance groups become QuarksJobs

To do this it relies on three Kubernetes components:

  • QuarksSecret, a custom resource and controller for the generation and rotation of secrets
  • QuarksJob, templating for Kubernetes jobs, which can trigger jobs on configuration changes and persist their output to secrets
  • QuarksStatefulSet, adds canary, zero-downtime deployment, zones and active-passive probe support

BOSH configuration related:

  • BOSH links can be provided by existing Kubernetes secrets
  • Provides BOSH link properties as Kubernetes secrets
  • Generates explicit variables, e.g. password, certificate, and SSH keys
  • Reads implicit variables from secrets
  • Secret rotation for individual secrets

BOSH releases related:

  • Pre-render scripts to patch releases, which are incompatible with Kubernetes

BOSH lifecycle related:

  • Restart only affected instance groups on update
  • Sequential startup of instance groups
  • Kubernetes healthchecks instead of monit

Kubernetes related:

  • Supports RBAC
  • Uses immutable, versioned secrets internally

BOSH Compatibily

  • Supports BOSH deployment manifests, including links and addons
  • Uses available BPM information from job releases
  • Renders ERB job templates in an init container, before starting the dockerized BOSH release
  • Adds endpoints and services for instance groups
  • BOSH DNS support
  • Uses Kubernetes zones for BOSH AZs

Prerequisites

The cf-operator assumes that the cluster root CA is also used for signing CSRs via the certificates.k8s.io API and will embed this CA in the generated certificate secrets. If your cluster is set up to use a different cluster-signing CA the generated certificates will have the wrong CA embedded. See https://kubernetes.io/docs/tasks/tls/managing-tls-in-a-cluster/ for more information on cluster trust.

Installing

Using the helm chart

The cf-operator can be installed via helm. You can use our helm repository.

See the releases page for up-to-date instructions on how to install the operator.

For more information about the cf-operator helm chart and how to configure it, please refer to deploy/helm/cf-operator/README.md

Recovering from a crash

If the operator pod crashes, it cannot be restarted in the same namespace before the existing mutating webhook configuration for that namespace is removed. The operator uses mutating webhooks to modify pods on the fly and Kubernetes fails to create pods if the webhook server is unreachable. The webhook configurations are installed cluster wide and don't belong to a single namespace, just like custom resources.

To remove the webhook configurations for the cf-operator namespace run:

CF_OPERATOR_NAMESPACE=cf-operator
kubectl delete mutatingwebhookconfiguration "cf-operator-hook-$CF_OPERATOR_NAMESPACE"
kubectl delete validatingwebhookconfiguration "cf-operator-hook-$CF_OPERATOR_NAMESPACE"

From Kubernetes 1.15 onwards, it is possible to instead patch the webhook configurations for the cf-operator namespace via:

CF_OPERATOR_NAMESPACE=cf-operator
kubectl patch mutatingwebhookconfigurations "cf-operator-hook-$CF_OPERATOR_NAMESPACE" -p '
webhooks:
- name: mutate-pods.quarks.cloudfoundry.org
  objectSelector:
    matchExpressions:
    - key: name
      operator: NotIn
      values:
      - "cf-operator"
'

Custom Resources

The cf-operator watches four different types of custom resources:

The cf-operator requires the according CRDs to be installed in the cluster in order to work as expected. By default, the cf-operator applies CRDs in your cluster automatically.

To verify that the CRD´s are installed:

$ kubectl get crds
NAME                                            CREATED AT
boshdeployments.quarks.cloudfoundry.org        2019-06-25T07:08:37Z
quarksjobs.quarks.cloudfoundry.org           2019-06-25T07:08:37Z
quarkssecrets.quarks.cloudfoundry.org        2019-06-25T07:08:37Z
quarksstatefulsets.quarks.cloudfoundry.org   2019-06-25T07:08:37Z

Variables

BOSH releases consume two types of variables, explicit and implicit ones.

Implicit Variables

Implicit variables have to be created before creating a BOSH deployment resource. The previous example creates a secret named nats-deployment.var-custom-password. That value will be used to fill ((custom-password)) place holders in the BOSH manifest.

The name of the secret has to follow this scheme: '<bosh-deployment-cr.name>.var-'

Missing implicit variables are treated as an error.

Explicit Variables

Explicit variables are explicitly defined in the BOSH manifest. They are generated automatically upon deployment and stored in secrets.

The naming scheme is the same as for implicit variables.

If an explicit variable secret already exists, it will not be generated. This allows users to set their own passwords, etc.

Using your fresh installation

With a running cf-operator pod, you can try one of the files (see docs/examples/bosh-deployment/boshdeployment-with-custom-variable.yaml ), as follows:

kubectl -n cf-operator apply -f docs/examples/bosh-deployment/boshdeployment-with-custom-variable.yaml

The above will spawn two pods in your cf-operator namespace (which needs to be created upfront), running the BOSH nats release.

You can access the cf-operator logs by following the operator pod's output:

kubectl logs -f -n cf-operator cf-operator

Or look at the k8s event log:

kubectl get events -n cf-operator --watch

Modifying the deployment

The main input to the operator is the BOSH deployment custom resource and the according manifest config map or secret. Changes to the Spec or Data fields of either of those will trigger the operator to recalculate the desired state and apply the required changes from the current state.

Besides that there are more things the user can change which will trigger an update of the deployment:

  • ops files can be added or removed from the BOSH deployment. Existing ops file config maps and secrets can be modified
  • generated secrets for explicit variables can be modified
  • secrets for implicit variables have to be created by the user beforehand anyway, but can also be changed after the initial deployment

Development and Tests

For more information about the operator development, see docs/development.md

For more information about testing, see docs/testing.md

For more information about building the operator from source, see docs/building.md

For more information about how to develop a BOSH release using Quarks and SCF, see the SCFv3 docs

Nice tools to use

The following is a list of tools with their respective main features that can help you to simplify your development work when dealing with cf-operator and kubecf

It provides an easy way to navigate through your k8s resources, while watching lively to changes on them. Main features that can be helpful for containerized CF are:

  • inmediate access to resources YAMLs definition

  • inmediate access to services endpoints

  • inmediate access to pods/container logs

  • sort resources(e.g. pods) by cpu or memory consumption

  • inmediate access to a container secure shell

A tool-kit with different features around k8s and CloudFoundry

  • top, to get an overview on the cpu/memory/load of the cluster, per ns and pods.

  • logs, to download all logs from all pods into your local system

  • pod-exec, to open a shell into containers. This can execute cmds in different containers simultaneously.

  • node-exec, to open a shell into nodes. This can execute cmds in different containers simultaneously.

Allows you to tail multiple pods on k8s and multiple containers within the pod.

A more user friendly to navigate your k8s cluster resources.

About

BOSH releases deployed on Kubernetes

Resources

License

Stars

Watchers

Forks

Packages

No packages published

Languages

  • Go 95.1%
  • HTML 2.8%
  • Shell 1.8%
  • Other 0.3%