Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Support remote clusters using API keys #8089

Merged
merged 23 commits into from
Oct 31, 2024
Merged

Support remote clusters using API keys #8089

Show file tree
Hide file tree
Changes from 18 commits
Commits
Show all changes
23 commits
Select commit Hold shift + click to select a range
851219b
Remote Clusters using API Keys
barkbay Sep 24, 2024
fb34ee7
Cosmetic changes
barkbay Oct 9, 2024
ed29168
Publish the remote cluster Service to the client, not the Pod IP
barkbay Oct 10, 2024
4c2c6fe
Use existing headless service
barkbay Oct 10, 2024
8735592
[DOC] Update issuing node transport certificates with third-party tools
barkbay Oct 11, 2024
5ec1e8d
[E2E] Add end-to-end test
barkbay Oct 11, 2024
875c169
[E2E] Also attempt to search + more details about errors
barkbay Oct 11, 2024
c1223bd
Only delete the keystore which has been initially loaded
barkbay Oct 11, 2024
ab442d1
Add API keystore Secret expectations
barkbay Oct 11, 2024
2a8cd2d
Update from review
barkbay Oct 15, 2024
b6bf462
Add support for access.search.query
barkbay Oct 15, 2024
3225307
Add support for allow_restricted_indices
barkbay Oct 15, 2024
6576910
Fix unit tests
barkbay Oct 15, 2024
e9fd909
Apply Peter's suggestions
barkbay Oct 21, 2024
8c0a6fb
Add ForgetChangeFor
barkbay Oct 21, 2024
c42c3f1
Handle conflict
barkbay Oct 21, 2024
ba62848
Merge remote-tracking branch 'origin/main' into rcs2-pr
barkbay Oct 21, 2024
9af83ee
typos
barkbay Oct 21, 2024
823ee1e
Apply suggestions from code review
barkbay Oct 30, 2024
e4e053a
Update comments
barkbay Oct 30, 2024
c1dab18
Merge remote-tracking branch 'origin/main' into rcs2-pr
barkbay Oct 30, 2024
a35d056
make generate
barkbay Oct 30, 2024
610423a
Fix expected license
barkbay Oct 30, 2024
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions cmd/manager/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -83,7 +83,7 @@ import (
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/logstash"
lsvalidation "github.com/elastic/cloud-on-k8s/v2/pkg/controller/logstash/validation"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/maps"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/remoteca"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/remotecluster"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/stackconfigpolicy"
"github.com/elastic/cloud-on-k8s/v2/pkg/controller/webhook"
"github.com/elastic/cloud-on-k8s/v2/pkg/dev"
Expand Down Expand Up @@ -899,7 +899,7 @@ func registerControllers(mgr manager.Manager, params operator.Parameters, access
name string
registerFunc func(manager.Manager, rbac.AccessReviewer, operator.Parameters) error
}{
{name: "RemoteCA", registerFunc: remoteca.Add},
{name: "RemoteCA", registerFunc: remotecluster.Add},
{name: "APM-ES", registerFunc: associationctl.AddApmES},
{name: "APM-KB", registerFunc: associationctl.AddApmKibana},
{name: "KB-ES", registerFunc: associationctl.AddKibanaES},
Expand Down
57 changes: 57 additions & 0 deletions config/crds/v1/all-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4794,13 +4794,70 @@ spec:
type: string
type: object
type: object
remoteClusterServer:
description: |-
RemoteClusterServer specifies if the remote cluster server must be enabled.
This must be enabled if this cluster is a remote cluster which is expected to be accessed using Cross-Cluster API key APIs.
properties:
enabled:
type: boolean
type: object
remoteClusters:
description: RemoteClusters enables you to establish uni-directional
connections to a remote Elasticsearch cluster.
items:
description: RemoteCluster declares a remote Elasticsearch cluster
connection.
properties:
apiKey:
description: 'APIKey can be used to enable remote cluster access
using Cross-Cluster API keys: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html'
properties:
access:
description: Access is the name of the API Key. It is automatically
generated if not set or empty.
properties:
replication:
properties:
names:
items:
type: string
type: array
required:
- names
type: object
search:
properties:
allow_restricted_indices:
type: boolean
field_security:
properties:
except:
items:
type: string
type: array
grant:
items:
type: string
type: array
required:
- except
- grant
type: object
names:
items:
type: string
type: array
query:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- names
type: object
type: object
required:
- access
type: object
elasticsearchRef:
description: ElasticsearchRef is a reference to an Elasticsearch
cluster running within the same k8s cluster.
Expand Down
57 changes: 57 additions & 0 deletions config/crds/v1/resources/elasticsearch.k8s.elastic.co_elasticsearches.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -9263,13 +9263,70 @@ spec:
type: string
type: object
type: object
remoteClusterServer:
description: |-
RemoteClusterServer specifies if the remote cluster server must be enabled.
This must be enabled if this cluster is a remote cluster which is expected to be accessed using Cross-Cluster API key APIs.
properties:
enabled:
type: boolean
type: object
remoteClusters:
description: RemoteClusters enables you to establish uni-directional
connections to a remote Elasticsearch cluster.
items:
description: RemoteCluster declares a remote Elasticsearch cluster
connection.
properties:
apiKey:
description: 'APIKey can be used to enable remote cluster access
using Cross-Cluster API keys: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html'
properties:
access:
description: Access is the name of the API Key. It is automatically
generated if not set or empty.
properties:
replication:
properties:
names:
items:
type: string
type: array
required:
- names
type: object
search:
properties:
allow_restricted_indices:
type: boolean
field_security:
properties:
except:
items:
type: string
type: array
grant:
items:
type: string
type: array
required:
- except
- grant
type: object
names:
items:
type: string
type: array
query:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- names
type: object
type: object
required:
- access
type: object
elasticsearchRef:
description: ElasticsearchRef is a reference to an Elasticsearch
cluster running within the same k8s cluster.
Expand Down
81 changes: 81 additions & 0 deletions config/recipes/remoteclusters/elasticsearch.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
@@ -0,0 +1,81 @@
apiVersion: v1
kind: Namespace
metadata:
name: ns1
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: cluster1
namespace: ns1
spec:
version: 8.15.0
remoteClusters:
- name: to-ns2-cluster2
elasticsearchRef:
name: cluster2
namespace: ns2
apiKey:
access:
search:
names:
## This requires the sample data available at https://kibana_url/app/home#/tutorial_directory/sampleData
- kibana_sample_data_ecommerce
nodeSets:
- name: default
config:
node.store.allow_mmap: false
count: 3
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
namespace: ns1
spec:
# http:
# service:
# spec:
# # expose this cluster Service with a LoadBalancer
# type: LoadBalancer
version: 8.15.0
count: 1
elasticsearchRef:
name: "cluster1"
---
apiVersion: v1
kind: Namespace
metadata:
name: ns2
---
apiVersion: elasticsearch.k8s.elastic.co/v1
kind: Elasticsearch
metadata:
name: cluster2
namespace: ns2
spec:
version: 8.15.0
## Required for this cluster to be accessed using remote cluster API keys.
remoteClusterServer:
enabled: true
nodeSets:
- name: default
config:
node.store.allow_mmap: false
count: 3
---
apiVersion: kibana.k8s.elastic.co/v1
kind: Kibana
metadata:
name: kibana
namespace: ns2
spec:
# http:
# service:
# spec:
# # expose this cluster Service with a LoadBalancer
# type: LoadBalancer
version: 8.15.0
count: 1
elasticsearchRef:
name: "cluster2"
57 changes: 57 additions & 0 deletions deploy/eck-operator/charts/eck-operator-crds/templates/all-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4836,13 +4836,70 @@ spec:
type: string
type: object
type: object
remoteClusterServer:
description: |-
RemoteClusterServer specifies if the remote cluster server must be enabled.
This must be enabled if this cluster is a remote cluster which is expected to be accessed using Cross-Cluster API key APIs.
properties:
enabled:
type: boolean
type: object
remoteClusters:
description: RemoteClusters enables you to establish uni-directional
connections to a remote Elasticsearch cluster.
items:
description: RemoteCluster declares a remote Elasticsearch cluster
connection.
properties:
apiKey:
description: 'APIKey can be used to enable remote cluster access
using Cross-Cluster API keys: https://www.elastic.co/guide/en/elasticsearch/reference/current/security-api-create-cross-cluster-api-key.html'
properties:
access:
description: Access is the name of the API Key. It is automatically
generated if not set or empty.
properties:
replication:
properties:
names:
items:
type: string
type: array
required:
- names
type: object
search:
properties:
allow_restricted_indices:
type: boolean
field_security:
properties:
except:
items:
type: string
type: array
grant:
items:
type: string
type: array
required:
- except
- grant
type: object
names:
items:
type: string
type: array
query:
type: object
x-kubernetes-preserve-unknown-fields: true
required:
- names
type: object
type: object
required:
- access
type: object
elasticsearchRef:
description: ElasticsearchRef is a reference to an Elasticsearch
cluster running within the same k8s cluster.
Expand Down
6 changes: 5 additions & 1 deletion ...hestrating-elastic-stack-applications/elasticsearch/transport-settings.asciidoc
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -115,12 +115,16 @@ spec:
volumeAttributes:
csi.cert-manager.io/issuer-name: ca-cluster-issuer <2>
csi.cert-manager.io/issuer-kind: ClusterIssuer
csi.cert-manager.io/dns-names: "${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local"
csi.cert-manager.io/dns-names: "${POD_NAME}.${POD_NAMESPACE}.svc.cluster.local" <3>
----
<1> Disables the self-signed certificates generated by ECK for the transport layer.

<2> The example assumes that a `ClusterIssuer` by the name of `ca-cluster-issuer` exists and a PEM encoded version of the CA certificate is available in a ConfigMap (in the example named `trust`). The CA certificate must be in a file called `ca.crt` inside the ConfigMap in the same namespace as the Elasticsearch resource.

<3> If the remote cluster server is enabled, then the DNS names must also include both:
* The DNS name for the related Kubernetes `Service`: `<cluster-name>-es-remote-cluster.${POD_NAMESPACE}.svc`
* The Pod DNS name: `${POD_NAME}.<cluster-name>-es-<nodeset-name>.${POD_NAMESPACE}.svc`

The following manifest is only provided to illustrate how these certificates can be configured in principle, using the trust-manager Bundle resource and cert-manager provisioned certificates:

[source,yaml,subs="attributes,callouts"]
Expand Down
Loading