Skip to content
This repository has been archived by the owner on Feb 22, 2022. It is now read-only.

Latest commit

 

History

History
104 lines (72 loc) · 7.06 KB

README.md

File metadata and controls

104 lines (72 loc) · 7.06 KB
Raw

⚠️ Repo Archive Notice

As of Nov 13, 2020, charts in this repo will no longer be updated. For more information, see the Helm Charts Deprecation and Archive Notice, and Update.

Helm Chart for Spark Operator

This is the Helm chart for the Kubernetes Operator for Apache Spark.

DEPRECATION NOTICE

This chart is deprecated and no longer supported.

Prerequisites

The Operator requires Kubernetes version 1.13 or above to use the subresource support for CustomResourceDefinitions, which became beta in 1.13 and is enabled by default in 1.13 and higher.

Installing the chart

First add the incubator repo:

$ helm repo add incubator http://storage.googleapis.com/kubernetes-charts-incubator

If using Helm 2, then the chart can be installed by running:

$ helm install incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default

Note that you need to use the --namespace flag during helm install to specify in which namespace you want to install the operator. The namespace can be existing or not. When it's not available, Helm would take care of creating the namespace. Note that this namespace has no relation to the namespace where you would like to deploy Spark jobs (i.e. the setting sparkJobNamespace shown in the table below). They can be the same namespace or different ones.

If using Helm 3, then install the chart by running:

$ helm install incubator/sparkoperator --generate-name --namespace spark-operator --set sparkJobNamespace=default

or

$ helm install [RELEASE-NAME] incubator/sparkoperator --namespace spark-operator --set sparkJobNamespace=default

if you don't want Helm to automatically generate a name for you.

Configuration

The following table lists the configurable parameters of the Spark operator chart and their default values.

Parameter Description Default
operatorImageName The name of the operator image gcr.io/spark-operator/spark-operator
operatorVersion The version of the operator to install v1beta2-1.2.0-3.0.0
imagePullPolicy Docker image pull policy IfNotPresent
imagePullSecrets Docker image pull secrets
replicas The number of replicas of the operator Deployment 1
sparkJobNamespace K8s namespace where Spark jobs are to be deployed ``
enableWebhook Whether to enable mutating admission webhook false
enableMetrics Whether to expose metrics to be scraped by Prometheus true
controllerThreads Number of worker threads used by the SparkApplication controller 10
ingressUrlFormat Ingress URL format ""
logLevel Logging verbosity level 2
installCrds Whether the release should install CRDs. true
metricsPort Port for the metrics endpoint 10254
metricsEndpoint Metrics endpoint "/metrics"
metricsPrefix Prefix for the metrics ""
nodeSelector Node labels for pod assignment {}
tolerations Tolerations for the sparkoperator deployment []
podAnnotations Annotations to be added to pods {}
resyncInterval Informer resync interval in seconds 30
webhookPort Service port of the webhook server 8080
webhookNamespaceSelector The webhook will only operate on namespaces with this label, specified in the form key1=value1,key2=value2 ""
resources Resources needed for the sparkoperator deployment {}
enableBatchScheduler Whether to enable batch scheduler for pod scheduling false
enableResourceQuotaEnforcement Whether to enable the ResourceQuota enforcement for SparkApplication resources. Requires the webhook to be enabled by setting enableWebhook to true. false
leaderElection.enable Whether to enable leader election when the operator Deployment has more than one replica, i.e., when replicas is greater than 1. false
leaderElection.lockName Lock name to use for leader election spark-operator-lock
leaderElection.lockNamespace Namespace to use for leader election (namespace of release)
securityContext Defines security context for operator container. {}
istio.enabled Whether Jobs will run in service mesh false

Specify each parameter using the --set key=value[,key=value] argument to helm install.

Upgrading

To 0.6.5
  • enableLeaderElection has been renamed leaderElection.enable to keep all of the leader election stuff together
To 0.6.2
Breaking changes
  • cleanupCrdsBeforeInstall has been removed for Helm 3 compatibility. If you wish to replicate this behavior before upgrading, do so manually (kubectl delete CustomResourceDefinition sparkapplications.sparkoperator.k8s.io scheduledsparkapplications.sparkoperator.k8s.io)
Non-breaking changes
  • app.kubernetes.io/name=sparkoperator label is added to CRDs if installed at this version, for easier manual cleanup after chart deletion (kubectl delete CustomResourceDefinition -l app.kubernetes.io/name=sparkoperator)

Contributing

When making changes to values.yaml, update the files in ci/ by running hack/update-ci.sh.