draft: Add proposal for grafana dashboard plugin #2708
Conversation
k8s-ci-robot
added
the
cncf-cla: yes
|
Hi @Kavinjsir. Thanks for your PR. I'm waiting for a kubernetes-sigs member to verify that this patch is reasonable to test. If it is, they should reply with Once the patch is verified, the new status will be reflected by the I understand the commands that are listed here. Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes/test-infra repository. |
k8s-ci-robot
added
needs-ok-to-test
| Also, the dashboard should provide filters and selectors to focus on certain objects. This can be implemented by utilizing [Grafana Query Variables](https://grafana.com/docs/grafana/latest/datasources/prometheus/#query-variable). | ||
|
|
||
| An example can be: | ||
|  |
There was a problem hiding this comment.
You can add this image by:
a) Click on for example open a new issue in the repo
b) Then, drag the image to the new issue
c) You will see that the GitHub will convert that in a syntax to render the image ( you can click on in the preview to check the result )
d) So, you can copy and past this syntax and use here in the markdown so that will work
There was a problem hiding this comment.
I loved the dashboard, my question here is do we need to build them manually by importing the json content OR have we a way to generate the whole content and just apply that to check its result?
There was a problem hiding this comment.
Regardless of how it provides the way for user to 'install' dashboards (or auto-installation without manual process), my origin idea is:
Basically, the plugin will scaffold out such json file from static code, which is similar as how service-monitor manifest gets generated by kustomize.
Which means:
- We will have a stable content of the json file which may get stored by a constant string.
- The plugin acts similarly as the
kustomizeplugin, which will generate these json file (considered as grafana manifests). - To visualize all default metrics, that may lead to multiple constant strings to store these json content.
There was a problem hiding this comment.
I am not sure if there are some approaches to generate the manifests pragmatically. (Compared with the above static way)
And it seems like drawing panels in the Web UI remains the main methods to build dashboards.
(Or if there are better ways to deal with this, I'd be glad to apply.)
There was a problem hiding this comment.
HI @Kavinjsir and @varshaprasad96,
I could check it out. See that with Grafana we can create a config and load the JSON file on it so that we would be able to make a target which when the user executes it the dashboard would be created.
See the configmap: https://rancher.com/docs/rancher/v2.5/en/monitoring-alerting/guides/persist-grafana/#2-create-a-configmap-using-the-grafana-json-model
That would mean the target would be like: kubectlc create configmap <dashboard-name> --from-file=/path/to/json/file -n <operator-ns> OR better yet we scaffold the configMap and we just apply it on the same ns of the Operator.
So that means:
- The grafana plugin would scaffold the JSON as described here
- Also, it would add a new target that allows the users just run that to create the Dashboard
-The first init PR to add plugin we can only create the JSON file and we can add the target and improve it incrementally in the follow-ups
ASAP I will try to provide further info and suggestions for this EP.
But you also can see the : https://www.redhat.com/fr/blog/custom-grafana-dashboards-red-hat-openshift-container-platform-4
There was a problem hiding this comment.
See here : https://www.lisenet.com/2021/install-and-configure-grafana-on-kubernetes/
@Kavinjsir sharing for you to be able to do manual tests as well and see how we need to config.
I need a little time to be able to finish the suggestions on this doc but you can also apply what was recommended already so that we can close a few comments
|
|
||
| 1. How should the plugin behave when it is used with/without `kustomize` being triggered? Would there be any duplications or coverings on the same path of the layout? Or if it maybe some contridictions when using both of them? | ||
| 2. The current metrics are exposed with restriction so that when the user want to query these metrics on his/her Grafana Dashboard, will it be possible to have authentication/authorization issue? | ||
|
|
There was a problem hiding this comment.
This might need more discussion to figure out whether necessary and how.
| 3. For Grafana Cloud, users may copy raw json to the ui. | ||
|
|
||
| To handle it generally, we may consider [Grafana HTTP API](https://grafana.com/docs/grafana/latest/http_api/dashboard/#create--update-dashboard). | ||
|
|
There was a problem hiding this comment.
This might need more discussion to figure out whether necessary and how.
|
|
||
| ## Open Questions [optional] | ||
|
|
||
| - TBD |
There was a problem hiding this comment.
WDYT about :
a) Should we scaffold a dir with the config to install the grafana? See: https://grafana.com/docs/grafana/latest/installation/kubernetes/
b) What should be the name of the default directory and where it should be scaffold?
It seems that we could scaffold: (root dir) something like:
grafana/
grafana/dashboards
grafana/config
OR
inside of the config dir another grafana dir with the k8s manifests to be applied on the cluster when the plugin is used
There was a problem hiding this comment.
We can also add a few other pointers like:
- How is the plugin backwards compatible for users who have prometheus scaffolded?
- Is it compatible only with specific versions of prometheus? Or do can we have an existing kb project and add this plugin to scaffold grafana manifests smoothly. Do we need any other changes to an old project structure.
There was a problem hiding this comment.
That's great! I like all these questions, they helps exploring the project. Thx @camilamacedo86 @varshaprasad96
|
|
||
| This proposed implementation is to create a new plugin, `grafana`, that scaffolds the manifests which are loaded by Grafana to display dashboards for controllers' status. | ||
|
|
||
| Basically, the manifest can be raw json file such as `controller-dashboard.json` that can be directly applied to Grafana Web UI. |
There was a problem hiding this comment.
Is required to be a JSON file OR could we have the manifest with the dashboard?
So that, users would only apply the manifests by command line and check the dashboards with graphs and end etc on the UI directly?
There was a problem hiding this comment.
The JSON file provides the dashboard template, (ie) UI related. How do we apply it on cluster? Could we elaborate on that? In the sense, do the users have to just import this JSON specified dashboard in their grafana instance or is there any other way to do so?
There was a problem hiding this comment.
For this json file, the basic approach is to let the user copy the content and paste it in their Grafana Web UI.
So this would assume:
- The users have the write access to their grafana service
- The users' Grafana service allows dashboards creation and storage from Web UI.
There was a problem hiding this comment.
The above idea may not be ideal. It assumes users are commonly doing dashboards creation work on Web UI directly.
I know for prometheus-operator (or kube-prometheus), the json content can be written in a K8s ConfigMap. That means:
- The dashboard is installed when the configmap is applied to the cluster.
- The installation process can be handled in Makefile with a simple command.
I personally like this approach, but I'm not sure how common is prometheus-operator or kube-prometheus.
Or can we assume people would use these stacks?
There was a problem hiding this comment.
In other case, like some people who are familiar with the Grafana community, they may setup their monitoring stacks through jsonnet. The grafana community provides tanka to provision k8s resources that can operate prometheus and grafana.
In that case, it maybe hard to install the dashboard at our side. So this maybe another story.
There was a problem hiding this comment.
So it seems challenging to provide a general approach to fit all cases. The dashboard installation process seems depends on how users deploy their Grafana.
This is also the reason why I come back to the very basic way as just scaffolding out the raw json file.
There was a problem hiding this comment.
Would you have some ideas or suggestions on this?
|
|
||
| For instance: work queue, reconciliation, performance, webhook. | ||
|
|
||
| As a result, the layout given by the plugin can be a folder of `config/grafana/`: |
There was a problem hiding this comment.
In the config/ dir we have only manifests to be used with kustomize. See that SDK add new features by adding manifests there, i.e for scorecard (https://github.com/operator-framework/operator-sdk/tree/master/testdata/go/v3/memcached-operator/config/scorecard) however, all is using kustomize, so I am afraid that we should not add JSON files on the config/ dir. However, we can create a new directory in the root like grafana/ where we can add all there.
There was a problem hiding this comment.
WDYT? @Kavinjsir @varshaprasad96
There was a problem hiding this comment.
Yeah, this sounds better!
| │ ├── reconcilier.json # Reconciliation instruments | ||
| │ ├── webhook.jon # Webhook Ops | ||
| │ └── workqueue.json # Work queue status | ||
| └── README.md # Docs for the usage |
There was a problem hiding this comment.
What we would have in this README? I see another option as well:
Could we try to append the info to the default README when the scaffold with this plugin is done, i.e : add a new section grafana on https://github.com/kubernetes-sigs/kubebuilder/blob/master/testdata/project-v3/README.md ?
However, what would be the cons/pros of each approach?
There was a problem hiding this comment.
The specific .json file to refer to each metric, right? What happens when that metric is unavailable for a controller. Do we just get a blank dashbaord with no metrics? Also would we be hardcoding the all the metrics that controller-runtime exports? If so, we should probably start by scaffolding only a subset which are really important/widely used.
There was a problem hiding this comment.
Yeah, @varshaprasad96 's idea is mainly what I thought. Since the controller-runtime provides plenty metrics, it seems not necessary/good to display all of them in one dashboard.
In brief when the queries failed (controller down/authentication failure/syntax error), the panels will be blank. By that time, the dashboard itself is added into the Grafana service, the user may get some error notification when open the page.
I am planning to hardcoding. Since we hope to have the dashboard available on the Grafana market, that means a stable json file, which may be likely as a statice file maintained in the plugin implementation folder?
How would you think of this? Would you have some suggestions or ideas to deal with this more elegantly?
|
|
||
| ## Open Questions [optional] | ||
|
|
||
| - TBD |
There was a problem hiding this comment.
We can also add a few other pointers like:
- How is the plugin backwards compatible for users who have prometheus scaffolded?
- Is it compatible only with specific versions of prometheus? Or do can we have an existing kb project and add this plugin to scaffold grafana manifests smoothly. Do we need any other changes to an old project structure.
|
|
||
| ## Summary | ||
|
|
||
| This EP will provide a way for the operator author to visualize the runtime metrics in order to observe the operator status. |
There was a problem hiding this comment.
| This EP will provide a way for the operator author to visualize the runtime metrics in order to observe the operator status. | |
| This EP will provide a way for the operator author to visualize the runtime metrics in order to observe the operator status using Grafana. |
|
|
||
| The functionality will be implemented by a new plugin that can generate the 'manifests' to be loaded on Grafana. | ||
|
|
||
| In the context of this EP, the Grafana 'manifests' are mainly: |
There was a problem hiding this comment.
How about elaborating this a bit more on why plugin will add value. Something on the lines of - we currently have controller-runtime metrics exposed by default for all KB operators at an endpoint through Prometheus. In order to visualize those same metrics, every operator author has to individually put in the effort to create a dashboard. This plugin intends to reduce the effort of operator authors to create dashboards and instead provides with the manifests that will allow them to integrate their controller metrics with grafana easily. On top of the basic dashboard provided by us having default metrics, operator authors can also add custom metrics and visualize them in the format they are comfortable in.
| In the context of this EP, the Grafana 'manifests' are mainly: | ||
|
|
||
| - JSON files to be directly loaded on Grafana dashboards | ||
| - TBD |
There was a problem hiding this comment.
Will the grafana dashboards utilize any of the existing promethtues manifests?
There was a problem hiding this comment.
Also, what would those JSON files contain? In the sense just the layout information, or any metrics specifics?
|
|
||
| ### Non-Goals | ||
|
|
||
| - TBD |
There was a problem hiding this comment.
Some non-goals could be:
- We do not deal with the kind of metrics which will be exported.
- The proposal will talk not only about scaffolding basic manifests (or templates) for integrating grafana dashboards. It does not deal with dashboard designs.
|
|
||
| This proposed implementation is to create a new plugin, `grafana`, that scaffolds the manifests which are loaded by Grafana to display dashboards for controllers' status. | ||
|
|
||
| Basically, the manifest can be raw json file such as `controller-dashboard.json` that can be directly applied to Grafana Web UI. |
There was a problem hiding this comment.
The JSON file provides the dashboard template, (ie) UI related. How do we apply it on cluster? Could we elaborate on that? In the sense, do the users have to just import this JSON specified dashboard in their grafana instance or is there any other way to do so?
| │ ├── reconcilier.json # Reconciliation instruments | ||
| │ ├── webhook.jon # Webhook Ops | ||
| │ └── workqueue.json # Work queue status | ||
| └── README.md # Docs for the usage |
There was a problem hiding this comment.
The specific .json file to refer to each metric, right? What happens when that metric is unavailable for a controller. Do we just get a blank dashbaord with no metrics? Also would we be hardcoding the all the metrics that controller-runtime exports? If so, we should probably start by scaffolding only a subset which are really important/widely used.
| 3. Errors: Error rates, or the operations/requests that takes unexpectedly long time. | ||
| 4. Saturation: The measure of the possibility of overflows such as deep work queue depth, concurrencies of reconciliations, large cpu/memory usage. | ||
|
|
||
| Also, the dashboard should provide filters and selectors to focus on certain objects. This can be implemented by utilizing [Grafana Query Variables](https://grafana.com/docs/grafana/latest/datasources/prometheus/#query-variable). |
There was a problem hiding this comment.
These queries will be specified in JSON files, or the user needs to filter them manually in a dashboard?
Kavinjsir
force-pushed
the
proposal/grafana-plugin
branch
2 times, most recently
from
ead612b to
9fd6066
Compare
|
|
||
| ## Summary | ||
|
|
||
| This EP will provide a way for the operator author to visualize the runtime metrics in order to observe the operator status. |
There was a problem hiding this comment.
| This EP will provide a way for the operator author to visualize the runtime metrics in order to observe the operator status. | |
| This proposal aims to provide an optional helper via a new alpha plugin in the ecosystem for the operator author to visualize the runtime metrics which are exported by default in any project that is based on [controller runtime](https://github.com/kubernetes-sigs/controller-runtime) and enable the metrics endpoint to observe the operator status using Grafana. For further information about the metrics see [its section in. the Kubebuilder docs](https://book.kubebuilder.io/reference/metrics.html) | |
| By providing this new plugin Kubebuilder users as other projects which consume Kubebuilder like SDK will be able to also take advantage of this new feature and help its users to check and monitor these metrics via Grafana dashboards. |
There was a problem hiding this comment.
Varsha's suggestion + a little nits
There was a problem hiding this comment.
Also, see the suggestion in the Motivation section where we try to write down the value that it brings.
| We currently have controller-runtime metrics exposed by default for all KB operators at an endpoint through Prometheus. | ||
|
|
||
| In order to visualize those same metrics, every operator author has to individually put in the effort to create a dashboard. | ||
|
|
||
| This plugin intends to reduce the effort of operator authors to create dashboards and instead provides with the manifests that will allow them to integrate their controller metrics with grafana easily. | ||
|
|
||
| On top of the basic dashboard provided by us having default metrics, operator authors can also add custom metrics and visualize them in the format they are comfortable in. | ||
|
|
||
| The functionality will be implemented by a new plugin that can generate the 'manifests' to be loaded on Grafana. | ||
|
|
||
| In the context of this EP, the Grafana 'manifests' are mainly: |
There was a problem hiding this comment.
I would add these parts on the technical details, I think the summary like https://github.com/kubernetes-sigs/kubebuilder/pull/2708/files#r892017769 shows its value as @varshaprasad96 suggested and is a good overview.
|
|
||
| In the context of this EP, the Grafana 'manifests' are mainly: | ||
|
|
||
| - JSON files to be directly loaded on Grafana dashboards |
There was a problem hiding this comment.
It is about how it will be done so IHMO also fits in the technical details
| The [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) provides rich metrics to measure controllers' status from various perspectives. | ||
|
|
||
| Currently, users of Kubebuilder need to design and create their own Grafana dashboards to visualize the metrics. | ||
| It would be great if we can provide a unified and comprehensive approach so that users may: | ||
|
|
||
| - easily visualize controller/operator metrics | ||
| - avoid duplicate creation of Grafana dashboards file | ||
| - improve the observability towards the opeartor, keep it safe and robust |
There was a problem hiding this comment.
WDYT about
| The [controller-runtime](https://github.com/kubernetes-sigs/controller-runtime) provides rich metrics to measure controllers' status from various perspectives. | |
| Currently, users of Kubebuilder need to design and create their own Grafana dashboards to visualize the metrics. | |
| It would be great if we can provide a unified and comprehensive approach so that users may: | |
| - easily visualize controller/operator metrics | |
| - avoid duplicate creation of Grafana dashboards file | |
| - improve the observability towards the opeartor, keep it safe and robust | |
| Kubebuilder and SDK users are looking for how to monitor their solutions using Grafana and the Prometheus metrics. Therefore, they are looking for helpers to speed up their solutions and to know how they can do it for their own customized metrics. For further information check some issues raised: | |
| - [Grafana dashboard for visualizing controller-runtime metrics #2183](https://github.com/kubernetes-sigs/kubebuilder/issues/2183) | |
| - [Custom resource metrics #2286](https://github.com/kubernetes-sigs/kubebuilder/issues/2286) | |
| Therefore for SDK users and for those that are looking to distribute their solution via [OLM](https://olm.operatorframework.io/) in the [OperatorHub.io](https://operatorhub.io/) or via vendors like Openshift this new helper plugin will help them out achieve further [Operator Capability levels](https://sdk.operatorframework.io/docs/overview/operator-capabilities/) since for Operator projects are able to achieve the ultimate levels Deep Insight and auto-pilot the must provide mechanisms to allow cluster admins Monitoring their solutions. | |
| You can also check that Ansible/Helm Operator authors which use SDK to build their solutions have been looking for that see for i.e. [operator-framework/operator-sdk/issues/2300](https://github.com/operator-framework/operator-sdk/issues/2300). | |
| Also, note that Grafana is commonly adopted and for i.e on Openshift it is installed and provided by default. [More info](https://www.redhat.com/fr/blog/custom-grafana-dashboards-red-hat-openshift-container-platform-4). |
| - Provide manifests that can display operator status for common needs. | ||
| - Add a plugin or enhance the current kustomize plugin that scaffolds Grafana manifests. |
There was a problem hiding this comment.
Here we need to speak more about the business goals/generic and not how to achieve them. So, wdyt about something like:
| - Provide manifests that can display operator status for common needs. | |
| - Add a plugin or enhance the current kustomize plugin that scaffolds Grafana manifests. | |
| - Provide an option to help Operator Authors provide monitoring features and projects with further quality which allow customer admins to check the exposed metrics with Grafana Dashboards | |
| - Encourage good practices and increase the quality of the Operator solutions distributed | |
| - Provide good examples and reduce the TOLL for Operator Authors add value to their solutions and to help them to skill up |
|
|
||
| ### Non-Goals | ||
|
|
||
| - TBD |
There was a problem hiding this comment.
| - TBD | |
| - Provided dashboards or helps for metrics which are not exported by default |
Kavinjsir
force-pushed
the
proposal/grafana-plugin
branch
from
9fd6066 to
4e71485
Compare
Kavinjsir
force-pushed
the
proposal/grafana-plugin
branch
2 times, most recently
from
4219822 to
3863c84
Compare
Kavinjsir
force-pushed
the
proposal/grafana-plugin
branch
from
3863c84 to
2963ab8
Compare
|
[APPROVALNOTIFIER] This PR is NOT APPROVED This pull-request has been approved by: Kavinjsir The full list of commands accepted by this bot can be found here.
Needs approval from an approver in each of these files:
Approvers can indicate their approval by writing |
Kavinjsir
force-pushed
the
proposal/grafana-plugin
branch
from
2963ab8 to
b17ca11
Compare
|
Hi @Kavinjsir, I am closing this one because the plugin is done already so I think we can drop the proposal. |
Description of the Change
Add a proposal for the grafana dashboard plugin
Motivation of the Change
This is an enhanced proposal for the GSoC 2022 project.
The origin comes from #2183.
The content of the proposal also gets much reference from
The project as well as the proposal are in the very early stage and so the plans/designs/implementation can/will be welcome to be discussed and flexibly adjusted.