Skip to content

Commit

Permalink
Initial commit (#623)
Browse files Browse the repository at this point in the history
  • Loading branch information
Techassi authored Sep 21, 2023
1 parent 24b19cc commit a1927ae
Show file tree
Hide file tree
Showing 3 changed files with 55 additions and 25 deletions.
19 changes: 11 additions & 8 deletions docs/modules/kafka/pages/getting_started/installation.adoc
Original file line number Diff line number Diff line change
@@ -1,20 +1,22 @@
= Installation

On this page you will install the Stackable Operator for Apache Kafka and operators for its dependencies - ZooKeeper - as well as the commons and secret operator which are required by all Stackable Operators.
On this page you will install the Stackable Operator for Apache Kafka and operators for its dependencies - ZooKeeper -
as well as the commons and secret operator which are required by all Stackable Operators.

== Stackable Operators

There are 2 ways to install Stackable Operators:

1. Using xref:stackablectl::index.adoc[stackablectl]

2. Using Helm
. Using xref:management:stackablectl:index.adoc[stackablectl]
. Using Helm

=== stackablectl

The stackablectl command line tool is the recommended way to interact with operators and dependencies. Follow the xref:stackablectl::installation.adoc[installation steps] for your platform if you choose to work with stackablectl.
The `stackablectl` command line tool is the recommended way to interact with operators and dependencies. Follow the
xref:management:stackablectl:installation.adoc[installation steps] for your platform if you choose to work with
`stackablectl`.

After you have installed stackablectl, run the following command to install all operators necessary for Kafka:
After you have installed `stackablectl`, run the following command to install all operators necessary for Kafka:

[source,bash]
----
Expand All @@ -28,7 +30,7 @@ The tool will show
include::example$getting_started/install-operator-output.txt[tag=stackablectl-install-operators-output]
----

TIP: Consult the xref:stackablectl::quickstart.adoc[] to learn more about how to use stackablectl.
TIP: Consult the xref:management:stackablectl:quickstart.adoc[] to learn more about how to use `stackablectl`.

=== Helm

Expand All @@ -46,7 +48,8 @@ Then install the Stackable Operators:
include::example$getting_started/getting_started.sh[tag=helm-install-operators]
----

Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache Kafka service (as well as the CRDs for the required operators). You are now ready to deploy Apache Kafka in Kubernetes.
Helm will deploy the operators in a Kubernetes Deployment and apply the CRDs for the Apache Kafka service (as well as
the CRDs for the required operators). You are now ready to deploy Apache Kafka in Kubernetes.

== What's next

Expand Down
45 changes: 33 additions & 12 deletions docs/modules/kafka/pages/index.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,45 +2,66 @@
:description: The Stackable Operator for Apache Superset is a Kubernetes operator that can manage Apache Kafka clusters. Learn about its features, resources, dependencies and demos, and see the list of supported Kafka versions.
:keywords: Stackable Operator, Apache Kafka, Kubernetes, operator, SQL, engineer, broker, big data, CRD, StatefulSet, ConfigMap, Service, Druid, ZooKeeper, NiFi, S3, demo, version

The Stackable Operator for Apache Kafka is an operator that can deploy and manage https://kafka.apache.org/[Apache Kafka] clusters on Kubernetes.
:metadata-quorum: https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum

The Stackable Operator for Apache Kafka is an operator that can deploy and manage https://kafka.apache.org/[Apache Kafka]
clusters on Kubernetes.
// what is Kafka?
Apache Kafka is a distributed streaming platform designed to handle large volumes of data in real-time. It is commonly used for real-time data processing, data ingestion, event streaming, and messaging between applications.
Apache Kafka is a distributed streaming platform designed to handle large volumes of data in real-time. It is commonly
used for real-time data processing, data ingestion, event streaming, and messaging between applications.

== Getting started

Follow the xref:kafka:getting_started/index.adoc[] which will guide you through installing The Stackable Kafka and ZooKeeper Operators, setting up ZooKeeper and Kafka and testing your Kafka using kcat.
Follow the xref:kafka:getting_started/index.adoc[] which will guide you through installing The Stackable Kafka and
ZooKeeper Operators, setting up ZooKeeper and Kafka and testing your Kafka using kcat.

== Resources

The _KafkaCluster_ custom resource contains your Kafka cluster configuration. It defines a single `broker` xref:concepts:roles-and-role-groups.adoc[role].
The _KafkaCluster_ custom resource contains your Kafka cluster configuration. It defines a single `broker`
xref:concepts:roles-and-role-groups.adoc[role].

image::kafka_overview.drawio.svg[A diagram depicting the Kubernetes resources created by the operator.]

For every xref:concepts:roles-and-role-groups.adoc#_role_groups[role group] in the `broker` role the Operator creates a StatefulSet. Multiple Services are created - one at role level, one per role group as well as one for every individual Pod - to allow access to the entire Kafka cluster, parts of it or just individual brokers.
For every xref:concepts:roles-and-role-groups.adoc#_role_groups[role group] in the `broker` role the Operator creates a
StatefulSet. Multiple Services are created - one at role level, one per role group as well as one for every individual
Pod - to allow access to the entire Kafka cluster, parts of it or just individual brokers.

For every StatefulSet (role group) a ConfigMap is deployed containing a `log4j.properties` file for xref:usage-guide/logging.adoc[logging] configuration and a `server.properties` file containing the whole Kafka configuration which is derived from the KafkaCluster resource.
For every StatefulSet (role group) a ConfigMap is deployed containing a `log4j.properties` file for
xref:usage-guide/logging.adoc[logging] configuration and a `server.properties` file containing the whole Kafka
configuration which is derived from the KafkaCluster resource.

The Operator creates a xref:concepts:service_discovery.adoc[] for the whole KafkaCluster which references the Service for the whole cluster. Other operators use this ConfigMap to connect to a Kafka cluster simply by name and it can also be used by custom third party applications to find the connection endpoint.
The Operator creates a xref:concepts:service_discovery.adoc[] for the whole KafkaCluster which references the Service
for the whole cluster. Other operators use this ConfigMap to connect to a Kafka cluster simply by name and it can also
be used by custom third party applications to find the connection endpoint.

== Dependencies

Kafka requires xref:zookeeper:index.adoc[Apache ZooKeeper] for coordination purposes (it will not be needed in the future as it will be replaced with a https://cwiki.apache.org/confluence/display/KAFKA/KIP-500%3A+Replace+ZooKeeper+with+a+Self-Managed+Metadata+Quorum[built-in solution]).
Kafka requires xref:zookeeper:index.adoc[Apache ZooKeeper] for coordination purposes (it will not be needed in the
future as it will be replaced with a {metadata-quorum}[built-in solution]).

== Connections to other products

Since Kafka often takes on a bridging role, many other products connect to it. In the <<demos, demos>> below you will find example data pipelines that use xref:nifi:index.adoc[Apache NiFi with the Stackable Operator] to write to Kafka and xref:nifi:index.adoc[Apache Druid with the Stackable Operator] to read from Kafka. But you can also connect using xref:spark-k8s:index.adoc[Apache Spark] or with a custom Job written in various languages.
Since Kafka often takes on a bridging role, many other products connect to it. In the <<demos, demos>> below you will
find example data pipelines that use xref:nifi:index.adoc[Apache NiFi with the Stackable Operator] to write to Kafka and
xref:nifi:index.adoc[Apache Druid with the Stackable Operator] to read from Kafka. But you can also connect using
xref:spark-k8s:index.adoc[Apache Spark] or with a custom Job written in various languages.

== [[demos]]Demos

xref:stackablectl::index.adoc[] supports installing xref:stackablectl::demos/index.adoc[] with a single command. The demos are complete data piplines which showcase multiple components of the Stackable platform working together and which you can try out interactively. Both demos below inject data into Kafka using NiFi and read from the Kafka topics using Druid.
xref:management:stackablectl:index.adoc[] supports installing xref:demos:index.adoc[] with a single command. The demos
are complete data piplines which showcase multiple components of the Stackable platform working together and which you
can try out interactively. Both demos below inject data into Kafka using NiFi and read from the Kafka topics using Druid.

=== Waterlevel Demo

The xref:stackablectl::demos/nifi-kafka-druid-water-level-data.adoc[] demo uses data from https://www.pegelonline.wsv.de/webservice/ueberblick[PEGELONLINE] to visualize water levels in rivers and coastal regions of Germany from historic and real time data.
The xref:demos:nifi-kafka-druid-water-level-data.adoc[] demo uses data from
https://www.pegelonline.wsv.de/webservice/ueberblick[PEGELONLINE] to visualize water levels in rivers and coastal
regions of Germany from historic and real time data.

=== Earthquake Demo

The xref:stackablectl::demos/nifi-kafka-druid-earthquake-data.adoc[] demo ingests https://earthquake.usgs.gov/[earthquake data] into a similar pipeline as is used in the waterlevel demo.
The xref:demos:nifi-kafka-druid-earthquake-data.adoc[] demo ingests https://earthquake.usgs.gov/[earthquake data] into
a similar pipeline as is used in the waterlevel demo.


== Supported Versions
Expand Down
16 changes: 11 additions & 5 deletions docs/modules/kafka/pages/usage-guide/security.adoc
Original file line number Diff line number Diff line change
Expand Up @@ -2,7 +2,8 @@

== Encryption

The internal and client communication can be encrypted TLS. This requires the xref:secret-operator::index.adoc[Secret Operator] to be present in order to provide certificates. The utilized certificates can be changed in a top-level config.
The internal and client communication can be encrypted TLS. This requires the xref:secret-operator:index.adoc[Secret
Operator] to be present in order to provide certificates. The utilized certificates can be changed in a top-level config.

[source,yaml]
----
Expand All @@ -27,7 +28,7 @@ spec:
<1> The `spec.clusterConfig.tls.serverSecretClass` refers to the client-to-server encryption. Defaults to the `tls` secret. Can be deactivated by setting `serverSecretClass` to `null`.
<2> The `spec.clusterConfig.tls.internalSecretClass` refers to the broker-to-broker internal encryption. This must be explicitly set or defaults to `tls`. May be disabled by setting `internalSecretClass` to `null`.

The `tls` secret is deployed from the xref:secret-operator::index.adoc[Secret Operator] and looks like this:
The `tls` secret is deployed from the xref:secret-operator:index.adoc[Secret Operator] and looks like this:

[source,yaml]
----
Expand All @@ -46,11 +47,14 @@ spec:
autoGenerate: true
----

You can create your own secrets and reference them e.g. in the `spec.clusterConfig.tls.serverSecretClass` or `spec.clusterConfig.tls.internalSecretClass` to use different certificates.
You can create your own secrets and reference them e.g. in the `spec.clusterConfig.tls.serverSecretClass` or
`spec.clusterConfig.tls.internalSecretClass` to use different certificates.

== Authentication

The internal or broker-to-broker communication is authenticated via TLS. In order to enforce TLS authentication for client-to-server communication, you can set an `AuthenticationClass` reference in the custom resource provided by the xref:commons-operator::index.adoc[Commons Operator].
The internal or broker-to-broker communication is authenticated via TLS. In order to enforce TLS authentication for
client-to-server communication, you can set an `AuthenticationClass` reference in the custom resource provided by the
xref:commons-operator:index.adoc[Commons Operator].

[source,yaml]
----
Expand Down Expand Up @@ -101,7 +105,9 @@ spec:

== [[authorization]]Authorization

If you wish to include integration with xref:opa::index.adoc[Open Policy Agent] and already have an OPA cluster, then you can include an `opa` field pointing to the OPA cluster discovery `ConfigMap` and the required package. The package is optional and will default to the `metadata.name` field:
If you wish to include integration with xref:opa:index.adoc[Open Policy Agent] and already have an OPA cluster, then you
can include an `opa` field pointing to the OPA cluster discovery `ConfigMap` and the required package. The package is
optional and will default to the `metadata.name` field:

[source,yaml]
----
Expand Down

0 comments on commit a1927ae

Please sign in to comment.