Document how to set up remote clusters across k8s boundaries (#2593)

Co-authored-by: Charith Ellawala <52399125+charith-elastic@users.noreply.github.com>

docs/elasticsearch-specification.asciidoc

@@ -29,6 +29,7 @@ Before you deploy and run ECK, take some time to look at the basic and advanced
 - <<{p}-advanced-node-scheduling,Advanced Elasticsearch node scheduling>>
 - <<{p}-orchestration>>
 - <<{p}-snapshots,Create automated snapshots>>
+- <<{p}-remote-clusters,Remote clusters>>
 - <<{p}-readiness>>
 - <<{p}-prestop>>
 
@@ -547,6 +548,7 @@ spec:
 include::orchestration.asciidoc[]
 include::advanced-node-scheduling.asciidoc[]
 include::snapshots.asciidoc[]
+include::remote-clusters.asciidoc[]
 
 
 [id="{p}-readiness"]

docs/remote-clusters.asciidoc

@@ -0,0 +1,128 @@
+ifdef::env-github[]
+****
+link:https://www.elastic.co/guide/en/cloud-on-k8s/master/k8s-remote-clusters.html[View this document on the Elastic website]
+****
+endif::[]
+[id="{p}-remote-clusters"]
+=== Remote clusters
+
+The link:https://www.elastic.co/guide/en/elasticsearch/reference/current/modules-remote-clusters.html[remote clusters module] in Elasticsearch enables you to establish uni-directional connections to a remote cluster. This functionality is used in cross-cluster replication and cross-cluster search.
+
+When using remote cluster connections with ECK, the setup process depends on where the remote cluster is deployed.
+
+[id="{p}-remote-clusters-connect-internal"]
+==== Connect from an Elasticsearch cluster running in the same Kubernetes cluster
+
+TBD
+
+[id="{p}-remote-clusters-connect-external"]
+==== Connect from an Elasticsearch cluster running outside the Kubernetes cluster
+
+NOTE: While it is technically possible to configure remote cluster connections using older versions of Elasticsearch, this guide only covers the setup for Elasticsearch 7.6 and later. The setup process is significantly simplified in Elasticsearch 7.6 due to improved support for the indirection of Kubernetes services.
+
+You can configure a remote cluster connection to an ECK-managed Elasticsearch cluster from another cluster running outside the Kubernetes cluster as follows:
+
+. Ensure that both clusters trust each other's certificate authority.
+. Configure the remote cluster connection via the Elasticsearch REST API.
+
+For illustration purposes, consider the following example:
+
+* `cluster-one` resides inside Kubernetes and is managed by ECK
+* `cluster-two` is not hosted inside the same Kubernetes cluster as `cluster-one` and may not even be managed by ECK 
+
+To configure `cluster-one` as a remote cluster in `cluster-two`:
+
+
+===== Ensure both clusters trust each others certificate authority
+
+The certificate authority (CA) used by ECK to issue certificates for the Elasticsearch transport layer is stored in a secret named `<cluster_name>-es-transport-certs-public`. Extract the certificate for `cluster-one` as follows:
+
+[source,sh]
+----
+kubectl get secret cluster-one-es-transport-certs-public \
+-o go-template='{{index .data "ca.crt" | base64decode}}' > remote.ca.crt
+----
+
+You then need to configure the CA as one of the trusted CAs in `cluster-two`. If that cluster is hosted outside of Kubernetes, simply add the CA certificate extracted in the above step to the list of CAs in link:https://www.elastic.co/guide/en/elasticsearch/reference/current/security-settings.html#_pem_encoded_files_3[`xpack.security.transport.ssl.certificate_authorities`].
+
+If `cluster-two` is also managed by an ECK instance, proceed as follows:
+
+Create a secret with the CA certificate you just extracted:
+[source,sh]
+----
+kubectl create secret generic remote-certs --from-file=remote.ca.crt
+----
+
+Use this secret to configure `cluster-one`'s CA as a trusted CA in `cluster-two`:
+
+[source,yaml,subs="attributes"]
+----
+apiVersion: elasticsearch.k8s.elastic.co/{eck_crd_version}
+kind: Elasticsearch
+metadata:
+  name: cluster-two
+spec:
+  nodeSets:
+  - config:
+      xpack.security.transport.ssl.certificate_authorities:
+      - /usr/share/elasticsearch/config/other/remote.ca.crt
+    count: 3
+    name: default
+    podTemplate:
+      spec:
+        containers:
+        - name: elasticsearch
+          volumeMounts:
+          - mountPath: /usr/share/elasticsearch/config/other
+            name: remote-certs
+        volumes:
+        - name: remote-certs
+          secret:
+            secretName: remote-certs
+  version: {version}
+----
+
+Repeat the above steps to add the CA of `cluster-two` to `cluster-one` as well.
+
+===== Configure the remote cluster connection via the Elasticsearch REST API
+
+Expose the transport layer of `cluster-one`.
+
+[source,yaml]
+----
+apiVersion: v1
+kind: Service
+metadata:
+  name: cluster-one-es-transport
+spec:
+  selector:
+    common.k8s.elastic.co/type: elasticsearch
+    elasticsearch.k8s.elastic.co/cluster-name: cluster-one
+  type: LoadBalancer <1>
+  ports:
+  - protocol: TCP
+    port: 9300
+    targetPort: 9300
+----
+<1> On cloud providers which support external load balancers, setting the type field to LoadBalancer provisions a load balancer for your Service. Alternatively expose the service via a Kubernetes link:https://kubernetes.io/docs/concepts/services-networking/ingress/[Ingress].
+
+Finally, configure `cluster-one` as a remote cluster in `cluster-two` using the Elasticsearch REST API:
+
+[source,sh]
+----
+PUT _cluster/settings
+{
+  "persistent": {
+    "cluster": {
+      "remote": {
+        "cluster-one": {
+          "mode": "proxy", <1>
+          "proxy_address": "${LOADBALANCER_IP}:9300" <2>
+        }
+      }
+    }
+  }
+}
+----
+<1> Use "proxy" mode as `cluster-two` will be connecting to `cluster-one` through the Kubernetes service abstraction.
+<2> Replace `${LOADBALANCER_IP}` with the IP address assigned to the `LoadBalancer` configured above. if you have configured a DNS entry for the service, you can use the DNS name instead of the IP address as well.