Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[Stack Monitoring] Kibana monitoring with Metricbeat and Filebeat as sidecars #4618

Merged
merged 34 commits into from
Jul 12, 2021
Merged

[Stack Monitoring] Kibana monitoring with Metricbeat and Filebeat as sidecars #4618

Show file tree
Hide file tree
Changes from 24 commits
Commits
Show all changes
34 commits
Select commit Hold shift + click to select a range
0822acb
Add Monitoring fields to the Kibana resource
thbkrkr Jul 6, 2021
795d8ce
Kibana Monitoring association and controller
thbkrkr Jul 6, 2021
7788081
Run generate
thbkrkr Jul 6, 2021
97a72fd
Cleanup duplicate EsMonitoringAssociationType
thbkrkr Jul 6, 2021
3770ea7
Factorize Metricbeat base config
thbkrkr Jul 6, 2021
5d75ca6
Differentiate config secret beat with assoc type
thbkrkr Jul 6, 2021
eac7d0d
Kibana stackmon package
thbkrkr Jul 6, 2021
06b8630
Plug monitoring into the driver
thbkrkr Jul 6, 2021
e075fce
Run generate
thbkrkr Jul 7, 2021
1c8b208
Kibana sidecar unit tests
thbkrkr Jul 7, 2021
0b0a279
Make Kibana namer and HTTPPort reusable
thbkrkr Jul 7, 2021
db48a85
Refactoring to factorize more
thbkrkr Jul 7, 2021
a7b08e4
Factorize validations
thbkrkr Jul 7, 2021
b47395d
E2E test TestKBStackMonitoring
thbkrkr Jul 7, 2021
c3c81f6
Package checks to avoid import cycle
thbkrkr Jul 7, 2021
d0b26ae
monitoring and validations packages to fix import cycle
thbkrkr Jul 7, 2021
e8a97a4
Sort imports
thbkrkr Jul 7, 2021
dc46d1e
Renaming
thbkrkr Jul 7, 2021
7c474be
Fix E2E test package and go tags
thbkrkr Jul 8, 2021
c1dc8a5
Merge remote-tracking branch 'upstream/master' into kibana-stack-moni…
thbkrkr Jul 8, 2021
b0666f9
Apply suggestions from code review
thbkrkr Jul 8, 2021
398177d
Validate Kibana ES association
thbkrkr Jul 8, 2021
7aa1757
Unit tests for monitoring package
thbkrkr Jul 8, 2021
30579ac
Unit tests for BuildMetricbeatBaseConfig
thbkrkr Jul 8, 2021
2ebbab2
Improve validation error messages
thbkrkr Jul 9, 2021
669f6f6
Merge remote-tracking branch 'upstream/master' into kibana-stack-moni…
thbkrkr Jul 9, 2021
010be29
Configure audit logs to be written to disk
thbkrkr Jul 9, 2021
e786717
Disable audit logs by default to let the user choose
thbkrkr Jul 9, 2021
20422f6
Fix SetAssociationStatusMap
thbkrkr Jul 9, 2021
4772364
Adjust E2E test
thbkrkr Jul 9, 2021
7ce7984
Temporarily disable CheckMetricbeatIndex
thbkrkr Jul 9, 2021
b600503
Introduce Monitored interface
barkbay Jul 12, 2021
ee1120e
Remove duplicated metav1.Object
barkbay Jul 12, 2021
dc69c18
new lines in imports
barkbay Jul 12, 2021
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
1 change: 1 addition & 0 deletions cmd/manager/main.go
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -679,6 +679,7 @@ func registerControllers(mgr manager.Manager, params operator.Parameters, access
{name: "AGENT-ES", registerFunc: associationctl.AddAgentES},
{name: "EMS-ES", registerFunc: associationctl.AddMapsES},
{name: "ES-MONITORING", registerFunc: associationctl.AddEsMonitoring},
{name: "KB-MONITORING", registerFunc: associationctl.AddKbMonitoring},
}

for _, c := range assocControllers {
Expand Down
81 changes: 81 additions & 0 deletions config/crds/v1/all-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -6738,6 +6738,80 @@ spec:
image:
description: Image is the Kibana Docker image to deploy.
type: string
monitoring:
description: Monitoring enables you to collect and ship log and monitoring
data of this Kibana. See https://www.elastic.co/guide/en/kibana/current/xpack-monitoring.html.
Metricbeat and Filebeat are deployed in the same Pod as sidecars
and each one sends data to one or two different Elasticsearch monitoring
clusters running in the same Kubernetes cluster.
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
will receive log data from this Kibana.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
required:
- name
type: object
type: array
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which will receive monitoring data from this Kibana.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
required:
- name
type: object
type: array
type: object
type: object
podTemplate:
description: PodTemplate provides customisation options (labels, annotations,
affinity rules, resource requests, and so on) for the Kibana pods
Expand Down Expand Up @@ -6819,6 +6893,13 @@ spec:
health:
description: Health of the deployment.
type: string
monitoringAssociationStatus:
additionalProperties:
description: AssociationStatus is the status of an association resource.
type: string
description: MonitoringAssociationStatus is the status of any auto-linking
to monitoring Elasticsearch clusters.
type: object
selector:
description: Selector is the label selector used to find all pods.
type: string
Expand Down
81 changes: 81 additions & 0 deletions config/crds/v1/bases/kibana.k8s.elastic.co_kibanas.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -544,6 +544,80 @@ spec:
image:
description: Image is the Kibana Docker image to deploy.
type: string
monitoring:
description: Monitoring enables you to collect and ship log and monitoring
data of this Kibana. See https://www.elastic.co/guide/en/kibana/current/xpack-monitoring.html.
Metricbeat and Filebeat are deployed in the same Pod as sidecars
and each one sends data to one or two different Elasticsearch monitoring
clusters running in the same Kubernetes cluster.
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
will receive log data from this Kibana.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
required:
- name
type: object
type: array
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which will receive monitoring data from this Kibana.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of
monitoring Elasticsearch clusters running in the same Kubernetes
cluster. Due to existing limitations, only a single Elasticsearch
cluster is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If
empty, defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing
Kubernetes service which is used to make requests
to the referenced object. It has to be in the same
namespace as the referenced resource. If left empty,
the default HTTP service of the referenced resource
is used.
type: string
required:
- name
type: object
type: array
type: object
type: object
podTemplate:
description: PodTemplate provides customisation options (labels, annotations,
affinity rules, resource requests, and so on) for the Kibana pods
Expand Down Expand Up @@ -7310,6 +7384,13 @@ spec:
health:
description: Health of the deployment.
type: string
monitoringAssociationStatus:
additionalProperties:
description: AssociationStatus is the status of an association resource.
type: string
description: MonitoringAssociationStatus is the status of any auto-linking
to monitoring Elasticsearch clusters.
type: object
selector:
description: Selector is the label selector used to find all pods.
type: string
Expand Down
79 changes: 79 additions & 0 deletions config/crds/v1beta1/all-crds.yaml
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -4359,6 +4359,78 @@ spec:
image:
description: Image is the Kibana Docker image to deploy.
type: string
monitoring:
description: Monitoring enables you to collect and ship log and monitoring
data of this Kibana. See https://www.elastic.co/guide/en/kibana/current/xpack-monitoring.html.
Metricbeat and Filebeat are deployed in the same Pod as sidecars and
each one sends data to one or two different Elasticsearch monitoring
clusters running in the same Kubernetes cluster.
properties:
logs:
description: Logs holds references to Elasticsearch clusters which
will receive log data from this Kibana.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of monitoring
Elasticsearch clusters running in the same Kubernetes cluster.
Due to existing limitations, only a single Elasticsearch cluster
is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If empty,
defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which is used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty, the default HTTP service of
the referenced resource is used.
type: string
required:
- name
type: object
type: array
type: object
metrics:
description: Metrics holds references to Elasticsearch clusters
which will receive monitoring data from this Kibana.
properties:
elasticsearchRefs:
description: ElasticsearchRefs is a reference to a list of monitoring
Elasticsearch clusters running in the same Kubernetes cluster.
Due to existing limitations, only a single Elasticsearch cluster
is currently supported.
items:
description: ObjectSelector defines a reference to a Kubernetes
object.
properties:
name:
description: Name of the Kubernetes object.
type: string
namespace:
description: Namespace of the Kubernetes object. If empty,
defaults to the current namespace.
type: string
serviceName:
description: ServiceName is the name of an existing Kubernetes
service which is used to make requests to the referenced
object. It has to be in the same namespace as the referenced
resource. If left empty, the default HTTP service of
the referenced resource is used.
type: string
required:
- name
type: object
type: array
type: object
type: object
podTemplate:
description: PodTemplate provides customisation options (labels, annotations,
affinity rules, resource requests, and so on) for the Kibana pods
Expand Down Expand Up @@ -4439,6 +4511,13 @@ spec:
health:
description: Health of the deployment.
type: string
monitoringAssociationStatus:
additionalProperties:
description: AssociationStatus is the status of an association resource.
type: string
description: MonitoringAssociationStatus is the status of any auto-linking
to monitoring Elasticsearch clusters.
type: object
selector:
description: Selector is the label selector used to find all pods.
type: string
Expand Down
Loading