Skip to content

Commit

Permalink
Support ClickHouse Cluster deployment
Browse files Browse the repository at this point in the history 
Signed-off-by: Yanjun Zhou <zhouya@vmware.com>
  • Loading branch information
@yanjunz97
yanjunz97 committed Jun 24, 2022
1 parent 09c55fd commit a526502
Show file tree
Hide file tree
Showing 14 changed files with 811 additions and 206 deletions.
14 changes: 10 additions & 4 deletions build/charts/theia/README.md
View file
Original file line number Diff line number Diff line change
Expand Up @@ -18,6 +18,12 @@ Kubernetes: `>= 1.16.0-0`

| Key | Type | Default | Description |
|-----|------|---------|-------------|
| clickhouse.cluster.enable | bool | `false` | Determine whether to deploy ClickHouse as a cluster. |
| clickhouse.cluster.installZookeeper | object | `{"size":"5Gi"}` | Specification to install a default ZooKeeper cluster. This is ignored if zookeeperHosts are provided. The default ZooKeeper cluster is deployed with 3 replicas. |
| clickhouse.cluster.podDistribution | list | `[{"number":1,"topologyKey":"kubernetes.io/hostname","type":"MaxNumberPerNode"}]` | Affinity for the ClickHouse Pods. By default, it allows only one ClickHouse instance per Node. Please refer to <https://github.com/Altinity/clickhouse-operator/blob/master/docs/chi-examples/99-clickhouseinstallation-max.yaml> for other distributions. |
| clickhouse.cluster.replicas | int | `2` | Number of ClickHouse replicas in each shard. |
| clickhouse.cluster.shards | int | `2` | Number of ClickHouse shards in the cluster. |
| clickhouse.cluster.zookeeperHosts | list | `[]` | To use a pre-installed ZooKeeper for ClickHouse data replication, please provide a list of your ZooKeeper hosts. To install a customized ZooKeeper, refer to <https://github.com/Altinity/clickhouse-operator/blob/master/docs/zookeeper_setup.md> |
| clickhouse.connectionSecret | object | `{"password":"clickhouse_operator_password","username":"clickhouse_operator"}` | Credentials to connect to ClickHouse. They will be stored in a secret. |
| clickhouse.image | object | `{"pullPolicy":"IfNotPresent","repository":"projects.registry.vmware.com/antrea/theia-clickhouse-server","tag":"21.11"}` | Container image used by ClickHouse. |
| clickhouse.monitor.deletePercentage | float | `0.5` | The percentage of records in ClickHouse that will be deleted when the storage grows above threshold. Vary from 0 to 1. |
Expand All @@ -30,14 +36,14 @@ Kubernetes: `>= 1.16.0-0`
| clickhouse.service.tcpPort | int | `9000` | TCP port number for the ClickHouse service. |
| clickhouse.service.type | string | `"ClusterIP"` | The type of Service exposing ClickHouse. It can be one of ClusterIP, NodePort or LoadBalancer. |
| clickhouse.storage.createPersistentVolume.local.affinity | object | `{}` | Affinity for the Local Persistent Volume. By default it requires to label the Node used to store the ClickHouse data with "antrea.io/clickhouse-data-node=". |
| clickhouse.storage.createPersistentVolume.local.path | string | `"/data"` | The local path. Required when type is "Local". |
| clickhouse.storage.createPersistentVolume.local.path | string | `"/data"` | The local path. Required when type is "Local". To use this Local PV for cluster, please create dir <path>/vol<i> on the Node, where i varies from 0 to shards * replicas. |
| clickhouse.storage.createPersistentVolume.nfs.host | string | `""` | The NFS server hostname or IP address. Required when type is "NFS". |
| clickhouse.storage.createPersistentVolume.nfs.path | string | `""` | The path exported on the NFS server. Required when type is "NFS". |
| clickhouse.storage.createPersistentVolume.nfs.path | string | `""` | The path exported on the NFS server. Required when type is "NFS". To use this NFS PV for cluster, please create dir <path>/vol<i> on the NFS server, where i varies from 0 to shards * replicas. |
| clickhouse.storage.createPersistentVolume.type | string | `""` | Type of PersistentVolume. Can be set to "Local" or "NFS". Please set this value to use a PersistentVolume created by Theia. |
| clickhouse.storage.persistentVolumeClaimSpec | object | `{}` | Specification for PersistentVolumeClaim. This is ignored if createPersistentVolume.type is non-empty. To use a custom PersistentVolume, please set storageClassName: "" volumeName: "<my-pv>". To dynamically provision a PersistentVolume, please set storageClassName: "<my-storage-class>". Memory storage is used if both createPersistentVolume.type and persistentVolumeClaimSpec are empty. |
| clickhouse.storage.persistentVolumeClaimSpec | object | `{}` | Specification for PersistentVolumeClaim. This is ignored if createPersistentVolume.type is non-empty. To use a custom PersistentVolume, please set storageClassName: "" volumeName: "<my-pv>". To dynamically provision a PersistentVolume, please set storageClassName: "<my-storage-class>". Memory storage is used if both createPersistentVolume.type and persistentVolumeClaimSpec are empty. |
| clickhouse.storage.size | string | `"8Gi"` | ClickHouse storage size. Can be a plain integer or as a fixed-point number using one of these quantity suffixes: E, P, T, G, M, K. Or the power-of-two equivalents: Ei, Pi, Ti, Gi, Mi, Ki. |
| clickhouse.ttl | string | `"1 HOUR"` | Time to live for data in the ClickHouse. Can be a plain integer using one of these unit suffixes SECOND, MINUTE, HOUR, DAY, WEEK, MONTH, QUARTER, YEAR. |
| grafana.dashboards | list | `["flow_records_dashboard.json","pod_to_pod_dashboard.json","pod_to_service_dashboard.json","pod_to_external_dashboard.json","node_to_node_dashboard.json","networkpolicy_dashboard.json"]` | The dashboards to be displayed in Grafana UI. The files must be put under provisioning/dashboards. |
| grafana.dashboards | list | `["flow_records_dashboard.json","pod_to_pod_dashboard.json","pod_to_service_dashboard.json","pod_to_external_dashboard.json","node_to_node_dashboard.json","networkpolicy_dashboard.json"]` | The dashboards to be displayed in Grafana UI. The files must be put under provisioning/dashboards. |
| grafana.enable | bool | `true` | Determine whether to install Grafana. It is used as a data visualization and monitoring tool. |
| grafana.image | object | `{"pullPolicy":"IfNotPresent","repository":"projects.registry.vmware.com/antrea/theia-grafana","tag":"8.3.3"}` | Container image used by Grafana. |
| grafana.installPlugins | list | `["https://downloads.antrea.io/artifacts/grafana-custom-plugins/theia-grafana-sankey-plugin-1.0.1.zip;theia-grafana-sankey-plugin","https://downloads.antrea.io/artifacts/grafana-custom-plugins/theia-grafana-chord-plugin-1.0.0.zip;theia-grafana-chord-plugin","grafana-clickhouse-datasource 1.0.1"]` | Grafana plugins to install. |
Expand Down
62 changes: 60 additions & 2 deletions build/charts/theia/provisioning/datasources/create_table.sh
View file
Original file line number Diff line number Diff line change
Expand Up @@ -23,11 +23,14 @@
{{- else if eq $ttl._1 "HOUR" }}
{{- $ttlTimeout = min (mul $ttl._0 60 60) $ttlTimeout }}
{{- end }}

set -e
clickhouse client -n -h 127.0.0.1 <<-EOSQL
{{- if .Values.clickhouse.cluster.enable }}
CREATE TABLE IF NOT EXISTS flows_local (
{{- else }}
CREATE TABLE IF NOT EXISTS flows (
{{- end }}
timeInserted DateTime DEFAULT now(),
flowStartSeconds DateTime,
flowEndSeconds DateTime,
Expand Down Expand Up @@ -77,13 +80,22 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
reverseThroughputFromSourceNode UInt64,
reverseThroughputFromDestinationNode UInt64,
trusted UInt8 DEFAULT 0
{{- if .Values.clickhouse.cluster.enable }}
) engine=ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
{{- else }}
) engine=MergeTree
{{- end }}
ORDER BY (timeInserted, flowEndSeconds)
TTL timeInserted + INTERVAL {{ .Values.clickhouse.ttl }}
SETTINGS merge_with_ttl_timeout = {{ $ttlTimeout }};
{{- if and .Values.clickhouse.cluster.enable }}
CREATE MATERIALIZED VIEW IF NOT EXISTS flows_pod_view_local
ENGINE = ReplicatedSummingMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
{{- else }}
CREATE MATERIALIZED VIEW IF NOT EXISTS flows_pod_view
ENGINE = SummingMergeTree
{{- end }}
ORDER BY (
timeInserted,
flowEndSeconds,
Expand Down Expand Up @@ -123,7 +135,11 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
sum(reverseThroughput) AS reverseThroughput,
sum(throughputFromSourceNode) AS throughputFromSourceNode,
sum(throughputFromDestinationNode) AS throughputFromDestinationNode
{{- if .Values.clickhouse.cluster.enable }}
FROM flows_local
{{- else }}
FROM flows
{{- end }}
GROUP BY
timeInserted,
flowEndSeconds,
Expand All @@ -140,8 +156,13 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
sourceTransportPort,
destinationTransportPort;
{{- if .Values.clickhouse.cluster.enable }}
CREATE MATERIALIZED VIEW IF NOT EXISTS flows_node_view_local
ENGINE = ReplicatedSummingMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
{{- else }}
CREATE MATERIALIZED VIEW IF NOT EXISTS flows_node_view
ENGINE = SummingMergeTree
{{- end }}
ORDER BY (
timeInserted,
flowEndSeconds,
Expand Down Expand Up @@ -171,7 +192,11 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
sum(reverseThroughputFromSourceNode) AS reverseThroughputFromSourceNode,
sum(throughputFromDestinationNode) AS throughputFromDestinationNode,
sum(reverseThroughputFromDestinationNode) AS reverseThroughputFromDestinationNode
{{- if .Values.clickhouse.cluster.enable }}
FROM flows_local
{{- else }}
FROM flows
{{- end }}
GROUP BY
timeInserted,
flowEndSeconds,
Expand All @@ -182,8 +207,13 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
sourcePodNamespace,
destinationPodNamespace;
{{- if .Values.clickhouse.cluster.enable }}
CREATE MATERIALIZED VIEW IF NOT EXISTS flows_policy_view_local
ENGINE = ReplicatedSummingMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
{{- else }}
CREATE MATERIALIZED VIEW IF NOT EXISTS flows_policy_view
ENGINE = SummingMergeTree
{{- end }}
ORDER BY (
timeInserted,
flowEndSeconds,
Expand Down Expand Up @@ -235,7 +265,11 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
sum(reverseThroughputFromSourceNode) AS reverseThroughputFromSourceNode,
sum(throughputFromDestinationNode) AS throughputFromDestinationNode,
sum(reverseThroughputFromDestinationNode) AS reverseThroughputFromDestinationNode
{{- if .Values.clickhouse.cluster.enable }}
FROM flows_local
{{- else }}
FROM flows
{{- end }}
GROUP BY
timeInserted,
flowEndSeconds,
Expand All @@ -257,12 +291,36 @@ clickhouse client -n -h 127.0.0.1 <<-EOSQL
destinationServicePortName,
destinationIP;
{{- if .Values.clickhouse.cluster.enable }}
CREATE TABLE IF NOT EXISTS recommendations_local (
{{- else }}
CREATE TABLE IF NOT EXISTS recommendations (
{{- end }}
id String,
type String,
timeCreated DateTime,
yamls String
{{- if .Values.clickhouse.cluster.enable }}
) engine=ReplicatedMergeTree('/clickhouse/tables/{shard}/{database}/{table}', '{replica}')
{{- else }}
) engine=MergeTree
{{- end }}
ORDER BY (timeCreated);
{{- if .Values.clickhouse.cluster.enable }}
CREATE TABLE IF NOT EXISTS flows AS flows_local
engine=Distributed('{cluster}', default, flows_local, rand());
CREATE TABLE IF NOT EXISTS flows_pod_view AS flows_pod_view_local
engine=Distributed('{cluster}', default, flows_pod_view_local, rand());
CREATE TABLE IF NOT EXISTS flows_node_view AS flows_node_view_local
engine=Distributed('{cluster}', default, flows_node_view_local, rand());
CREATE TABLE IF NOT EXISTS flows_policy_view AS flows_policy_view_local
engine=Distributed('{cluster}', default, flows_pod_view_local, rand());
CREATE TABLE IF NOT EXISTS recommendations AS recommendations_local
engine=Distributed('{cluster}', default, recommendations_local, rand());
{{- end }}
EOSQL
70 changes: 70 additions & 0 deletions build/charts/theia/templates/_helpers.tpl
View file
Original file line number Diff line number Diff line change
@@ -0,0 +1,70 @@
{{- define "clickhouse.monitor.container" }}
{{- $clickhouse := .clickhouse }}
- name: clickhouse-monitor
image: {{ $clickhouse.monitor.image.repository }}:{{ $clickhouse.monitor.image.tag }}
imagePullPolicy: {{ $clickhouse.monitor.image.pullPolicy }}
env:
- name: CLICKHOUSE_USERNAME
valueFrom:
secretKeyRef:
name: clickhouse-secret
key: username
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: clickhouse-secret
key: password
- name: DB_URL
value: "tcp://localhost:9000"
- name: TABLE_NAME
{{- if $clickhouse.cluster.enable }}
value: "default.flows_local"
{{- else }}
value: "default.flows"
{{- end }}
- name: MV_NAMES
{{- if $clickhouse.cluster.enable }}
value: "default.flows_pod_view_local default.flows_node_view_local default.flows_policy_view_local"
{{- else }}
value: "default.flows_pod_view default.flows_node_view default.flows_policy_view"
{{- end }}
- name: STORAGE_SIZE
value: {{ $clickhouse.storage.size | quote }}
- name: THRESHOLD
value: {{ $clickhouse.monitor.threshold | quote }}
- name: DELETE_PERCENTAGE
value: {{ $clickhouse.monitor.deletePercentage | quote }}
- name: EXEC_INTERVAL
value: {{ $clickhouse.monitor.execInterval }}
- name: SKIP_ROUNDS_NUM
value: {{ $clickhouse.monitor.skipRoundsNum | quote }}
{{- end }}

{{- define "clickhouse.server.container" }}
{{- $clickhouse := .clickhouse }}
{{- $enablePV := .enablePV }}
- name: clickhouse
image: {{ $clickhouse.image.repository }}:{{ $clickhouse.image.tag }}
imagePullPolicy: {{ $clickhouse.image.pullPolicy }}
volumeMounts:
- name: clickhouse-configmap-volume
mountPath: /docker-entrypoint-initdb.d
{{- if not $enablePV }}
- name: clickhouse-storage-volume
mountPath: /var/lib/clickhouse
{{- end }}
{{- end }}

{{- define "clickhouse.volume" }}
{{- $clickhouse := .clickhouse }}
{{- $enablePV := .enablePV }}
- name: clickhouse-configmap-volume
configMap:
name: clickhouse-mounted-configmap
{{- if not $enablePV }}
- name: clickhouse-storage-volume
emptyDir:
medium: Memory
sizeLimit: {{ $clickhouse.storage.size }}
{{- end }}
{{- end }}
107 changes: 56 additions & 51 deletions build/charts/theia/templates/clickhouse/clickhouseinstallation.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -14,8 +14,39 @@ spec:
clusters:
- name: "clickhouse"
layout:
{{- if .Values.clickhouse.cluster.enable }}
shards:
{{- range $i, $_ := until (int .Values.clickhouse.cluster.shards) }}
- name: {{ $i | quote }}
replicasCount: {{ $.Values.clickhouse.cluster.replicas }}
replicas:
{{- range $j, $_ := until (int $.Values.clickhouse.cluster.replicas) }}
- name: {{ $i }}-{{ $j }}
templates:
{{- if eq $j (sub $.Values.clickhouse.cluster.replicas 1) }}
podTemplate: pod-template
{{- else }}
podTemplate: pod-template-without-monitor
{{- end }}
{{- end }}
{{- end }}
shardsCount: {{ .Values.clickhouse.cluster.shards }}
replicasCount: {{ .Values.clickhouse.cluster.replicas }}
{{- else }}
shardsCount: 1
replicasCount: 1
{{- end }}
{{- if .Values.clickhouse.cluster.enable }}
zookeeper:
nodes:
{{- if eq (len .Values.clickhouse.cluster.zookeeperHosts) 0 }}
- host: zookeeper.{{ .Release.Namespace }}
{{- else }}
{{- range $host := .Values.clickhouse.cluster.zookeeper.zookeeperHosts }}
- host: {{ $host }}
{{- end }}
{{- end }}
{{- end }}
defaults:
templates:
podTemplate: pod-template
Expand All @@ -34,61 +65,35 @@ spec:
- name: tcp
port: {{ .Values.clickhouse.service.tcpPort }}
podTemplates:
- name: pod-template
{{- if .Values.clickhouse.cluster.enable }}
- name: pod-template-without-monitor
spec:
containers:
- name: clickhouse
image: {{ .Values.clickhouse.image.repository }}:{{ .Values.clickhouse.image.tag }}
imagePullPolicy: {{ .Values.clickhouse.image.pullPolicy }}
volumeMounts:
- name: clickhouse-configmap-volume
mountPath: /docker-entrypoint-initdb.d
{{- if not $enablePV }}
- name: clickhouse-storage-volume
mountPath: /var/lib/clickhouse
{{- end }}
{{- if .Values.clickhouse.monitor.enable}}
- name: clickhouse-monitor
image: {{ .Values.clickhouse.monitor.image.repository }}:{{ .Values.clickhouse.monitor.image.tag }}
imagePullPolicy: {{ .Values.clickhouse.monitor.image.pullPolicy }}
env:
- name: CLICKHOUSE_USERNAME
valueFrom:
secretKeyRef:
name: clickhouse-secret
key: username
- name: CLICKHOUSE_PASSWORD
valueFrom:
secretKeyRef:
name: clickhouse-secret
key: password
- name: DB_URL
value: "tcp://localhost:9000"
- name: TABLE_NAME
value: "default.flows"
- name: MV_NAMES
value: "default.flows_pod_view default.flows_node_view default.flows_policy_view"
- name: STORAGE_SIZE
value: {{ .Values.clickhouse.storage.size | quote }}
- name: THRESHOLD
value: {{ .Values.clickhouse.monitor.threshold | quote }}
- name: DELETE_PERCENTAGE
value: {{ .Values.clickhouse.monitor.deletePercentage | quote }}
- name: EXEC_INTERVAL
value: {{ .Values.clickhouse.monitor.execInterval }}
- name: SKIP_ROUNDS_NUM
value: {{ .Values.clickhouse.monitor.skipRoundsNum | quote }}
{{- end}}
{{- include "clickhouse.server.container" (dict "clickhouse" .Values.clickhouse "enablePV" $enablePV) | indent 12 }}
volumes:
- name: clickhouse-configmap-volume
configMap:
name: clickhouse-mounted-configmap
{{- if not $enablePV }}
- name: clickhouse-storage-volume
emptyDir:
medium: Memory
sizeLimit: {{ .Values.clickhouse.storage.size }}
{{- include "clickhouse.volume" (dict "clickhouse" .Values.clickhouse "enablePV" $enablePV) | indent 12 }}
{{- if .Values.clickhouse.cluster.podDistribution }}
podDistribution:
{{- with .Values.clickhouse.cluster.podDistribution }}
{{ toJson . | trim }}
{{- end }}
{{- end }}
{{- end }}
- name: pod-template
spec:
containers:
{{- include "clickhouse.server.container" (dict "clickhouse" .Values.clickhouse "enablePV" $enablePV) | indent 12 }}
{{- if .Values.clickhouse.monitor.enable }}
{{- include "clickhouse.monitor.container" (dict "clickhouse" .Values.clickhouse) | indent 12 }}
{{- end }}
volumes:
{{- include "clickhouse.volume" (dict "clickhouse" .Values.clickhouse "enablePV" $enablePV) | indent 12 }}
{{- if and .Values.clickhouse.cluster.enable .Values.clickhouse.cluster.podDistribution }}
podDistribution:
{{- with .Values.clickhouse.cluster.podDistribution }}
{{ toJson . | trim }}
{{- end }}
{{- end }}
{{- if $enablePV }}
volumeClaimTemplates:
- name: clickhouse-storage-template
Expand Down
2 changes: 1 addition & 1 deletion build/charts/theia/templates/clickhouse/configmap.yaml
View file
Original file line number Diff line number Diff line change
Expand Up @@ -5,4 +5,4 @@ metadata:
namespace: {{ .Release.Namespace }}
data:
create_table.sh: |-
{{ tpl (.Files.Get "provisioning/datasources/create_table.sh") . | indent 4}}
{{ tpl (.Files.Get "provisioning/datasources/create_table.sh") . | indent 4 }}
Loading

0 comments on commit a526502

Please sign in to comment.