[ISSUE #340]Integrate With OpenTelemetry for metrics in EventMesh #467

Roc-00 · 2021-07-25T11:29:18Z

Fixes ISSUE#<340>.

Motivation

Export metrics data with open telemetry and use Prometheus for visual observation

Modifications

There is a readme document in the code I submitted. I use idea to run locally. I download Prometheus in the window version of the official website. The operation method of docker written in readme is given by reference to open telemetry. I haven't actually operated it. I was successful in running locally. I'm not sure whether the code I wrote meets the requirements, so I didn't finish all the metrics, but wrote three. If there is anything wrong, please put it forward and I will modify and improve it. When the code is perfect and correct, I will continue to write the rest of the code.

Documentation

Does this pull request introduce a new feature? yes
If yes, how is the feature documented? README.md

… observation

# Conflicts: # eventmesh-runtime/build.gradle

codecov-commenter · 2021-07-25T11:36:40Z

Codecov Report

Merging #467 (33cf1b7) into develop (4eaebde) will decrease coverage by 0.03%.
The diff coverage is 0.00%.

@@             Coverage Diff              @@
##             develop    #467      +/-   ##
============================================
- Coverage       9.84%   9.80%   -0.04%     
  Complexity       283     283              
============================================
  Files            228     230       +2     
  Lines          10829   10870      +41     
  Branches         923     923              
============================================
  Hits            1066    1066              
- Misses          9666    9707      +41     
  Partials          97      97

Impacted Files	Coverage Δ
...ntmesh/runtime/metrics/http/HTTPMetricsServer.java	`0.00% <0.00%> (ø)`
...e/metrics/openTelemetry/OpenTelemetryExporter.java	`0.00% <0.00%> (ø)`
...nTelemetry/OpenTelemetryExporterConfiguration.java	`0.00% <0.00%> (ø)`

Continue to review full report at Codecov.

Legend - Click here to learn more
Δ = absolute <relative> (impact), ø = not affected, ? = missing data
Powered by Codecov. Last update 4eaebde...33cf1b7. Read the comment docs.

… observation

tydhot · 2021-07-25T12:11:51Z

.../src/main/java/org/apache/eventmesh/runtime/metrics/openTelemetry/OpenTelemetryExporter.java

+
+        //maxHTTPCost
+        meter
+                .longValueObserverBuilder("max.HTTPCost")


It's strange to spell metric with hump.

how should I spell？

eventmesh.http.request.elapsed.max

tydhot · 2021-07-26T02:43:27Z

@qqeasonchen I found there is many metric in eventmesh now. Maybe it is a good opportunity to make them in otel.

tydhot

good job, anyway. I think is it a good start for your open source journy。

tydhot · 2021-07-26T02:48:58Z

btw, your document can be formatted with some tools.

ruanwenjun

Is it possible for us to use plugin implementation of exporter? I think maybe not everyone will use prometheus.

ruanwenjun · 2021-07-26T03:01:52Z

eventmesh-runtime/build.gradle

+List open_telemetry = [
+        "io.opentelemetry:opentelemetry-api:1.3.0",
+        "io.opentelemetry:opentelemetry-sdk:1.3.0",
+        "io.opentelemetry:opentelemetry-sdk-metrics:1.3.0-alpha",


Can we use other stable version? Using alpha does not seem to be a good choice.

It is ok to use that. Otel's sdk metrics is in alpha although, it is widely used in https://github.com/open-telemetry/opentelemetry-java-instrumentation. This agent has been validated in enough prod environments.

ruanwenjun · 2021-07-26T03:06:19Z

...a/org/apache/eventmesh/runtime/metrics/openTelemetry/OpenTelemetryExporterConfiguration.java

+
+    private  HTTPServer server;//Prometheus server
+
+    int prometheusPort = 19090;//the endpoint to export metrics


It is better to add this to configuration properties.

I've thought about it, but I'm not very good at it. What language is involved? script?

Most configuration is at eventmesh-runtime/conf/eventmesh.properties, in the future, I think it may be split into multiple files, but now you can just add the prometheusPort in this file, and load this config, you can see code in EventMeshHTTPConfiguration.

I will have a try.

Roc-00 · 2021-07-26T03:57:46Z

btw, your document can be formatted with some tools.

what kind of tools? I have no experience, but I will try my best to do that.

tydhot · 2021-07-27T02:23:54Z

@Roc-00 https://doocs.gitee.io/md/#/
you can try this

qqeasonchen · 2021-07-27T06:33:57Z

@qqeasonchen I found there is many metric in eventmesh now. Maybe it is a good opportunity to make them in otel.

yep.

lrhkobe · 2021-07-27T11:21:48Z

@Roc-00 Maybe the metrics about tcp in 'EventMeshTcpMonitor.java' can also be processed in the same way.

Roc-00 · 2021-07-27T13:25:12Z

@Roc-00 https://doocs.gitee.io/md/#/
you can try this

Is this similar to typora

Roc-00 · 2021-07-27T13:27:40Z

@Roc-00 Maybe the metrics about tcp in 'EventMeshTcpMonitor.java' can also be processed in the same way.

I will find out the difference between HTTP and TCP,then complete TCP and HTTP metrics

qqeasonchen · 2021-07-31T14:34:18Z

@Roc-00 please re-submit this pr to branch opentelemetry, and add a design doc first like https://github.com/apache/incubator-eventmesh/blob/develop/docs/en/features/eventmesh-stream-design.md.

Roc-00 · 2021-08-01T06:42:09Z

@Roc-00重请提交这个pr 到分支开放遥测，并首先添加一个设计文档，如https://github.com/apache/incubator-eventmesh/blob/develop/docs/en/features/eventmesh-stream-design .MD。

OK.

qqeasonchen

@Roc-00重请提交这个pr 到分支开放遥测，并首先添加一个设计文档，如https://github.com/apache/incubator-eventmesh/blob/develop/docs/en/features/eventmesh-stream-design .MD。

OK.

please avoid chinese here.

qqeasonchen · 2021-07-27T08:26:53Z

eventmesh-runtime/src/main/java/org/apache/eventmesh/runtime/metrics/openTelemetry/README.md

@@ -0,0 +1,41 @@
+# Open Telemetry exporter


readme file should not exist in src

Should I delete it，or put it in the design doc

qqeasonchen · 2021-07-27T08:27:22Z

...mesh-runtime/src/main/java/org/apache/eventmesh/runtime/metrics/openTelemetry/prometheus.yml

@@ -0,0 +1,46 @@
+#


yml file should not exist in src

Can I put it in conf like this

I think is ok.

Roc-00 · 2021-08-02T06:12:09Z

@Roc-00 please re-submit this pr to branch opentelemetry, and add a design doc first like https://github.com/apache/incubator-eventmesh/blob/develop/docs/en/features/eventmesh-stream-design.md.

Should I close this PR so that I can re-submit this pr to branch opentelemetry?

xwm1992 · 2021-08-02T08:29:01Z

@Roc-00 please re-submit this pr to branch opentelemetry, and add a design doc first like https://github.com/apache/incubator-eventmesh/blob/develop/docs/en/features/eventmesh-stream-design.md.

Should I close this PR so that I can re-submit this pr to branch opentelemetry?

you can edit this pr in this page, change the branch you want to merge

Roc-00 · 2021-08-04T08:35:45Z

Is this the tcp metrics?what is the meaning of "s" and "t"?

Roc-00 · 2021-08-04T08:39:30Z

Why the class TcpSummaryMetrics was never used?

Roc-00 · 2021-08-05T13:26:07Z

I only found the following tcp metrics.Are there any more?

* [ISSUE #340]Integrate With OpenTelemetry for metrics in EventMesh (#467) * The Chinese in the notes is translated into English. * Translation improvement * Translation improvement * fix Chinese annotation on runtime module * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Improper modification * Improper modification * improve * improve * improve * tcp metrics export * improve * improve * merge from otel branch * check lisence Co-authored-by: ZePeng Chen <84842773+Roc-00@users.noreply.github.com>

* [ISSUE apache#340]Integrate With OpenTelemetry for metrics in EventMesh (apache#467) * The Chinese in the notes is translated into English. * Translation improvement * Translation improvement * fix Chinese annotation on runtime module * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Improper modification * Improper modification * improve * improve * improve * tcp metrics export * improve * improve * merge from otel branch * check lisence Co-authored-by: ZePeng Chen <84842773+Roc-00@users.noreply.github.com>

* [ISSUE #340]Integrate With OpenTelemetry for metrics in EventMesh (#467) * The Chinese in the notes is translated into English. * Translation improvement * Translation improvement * fix Chinese annotation on runtime module * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Export metrics data with open telemetry and use Prometheus for visual observation * Improper modification * Improper modification * improve * improve * improve * tcp metrics export * improve * improve * merge from otel branch * check lisence Co-authored-by: ZePeng Chen <84842773+Roc-00@users.noreply.github.com>

Roc-00 and others added 10 commits July 10, 2021 16:05

The Chinese in the notes is translated into English.

518c6fd

Translation improvement

64e1331

Merge branch 'apache:develop' into develop

69d4887

Merge remote-tracking branch 'origin/develop' into develop

b1be2dd

Translation improvement

4b82183

fix Chinese annotation on runtime module

fad0381

Merge branch 'apache:develop' into develop

90cfc5c

Export metrics data with open telemetry and use Prometheus for visual…

0cab2ce

… observation

Merge branch 'apache:develop' into develop

091516f

Merge remote-tracking branch 'origin/develop' into develop

0bc9d50

# Conflicts: # eventmesh-runtime/build.gradle

Roc-00 added 3 commits July 25, 2021 19:42

Export metrics data with open telemetry and use Prometheus for visual…

f5a79c2

… observation

Export metrics data with open telemetry and use Prometheus for visual…

16d9bca

… observation

Export metrics data with open telemetry and use Prometheus for visual…

02df827

… observation

tydhot reviewed Jul 25, 2021

View reviewed changes

Roc-00 added 2 commits July 25, 2021 23:11

Improper modification

a7b4c08

Improper modification

33cf1b7

tydhot approved these changes Jul 26, 2021

View reviewed changes

ruanwenjun reviewed Jul 26, 2021

View reviewed changes

qqeasonchen changed the title ~~[ISSUE #340]Integrate With OpenTelemetry~~ [ISSUE #340]Integrate With OpenTelemetry for metrics in EventMesh Jul 27, 2021

qqeasonchen reviewed Aug 2, 2021

View reviewed changes

Roc-00 changed the base branch from develop to opentelemetry August 2, 2021 11:03

Roc-00 added 2 commits August 3, 2021 20:09

improve

da8e552

improve

45a063d

Roc-00 added 2 commits August 4, 2021 19:45

improve

d7b6ae2

tcp metrics export

0ce9316

improve

5ca22bd

Roc-00 changed the base branch from opentelemetry to develop August 10, 2021 09:45

improve

e29832b

Roc-00 changed the base branch from develop to opentelemetry August 10, 2021 10:11

qqeasonchen merged commit f0d8b55 into apache:opentelemetry Aug 10, 2021

Roc-00 deleted the develop branch September 8, 2021 12:25


		private HTTPServer server;//Prometheus server

		int prometheusPort = 19090;//the endpoint to export metrics

[ISSUE #340]Integrate With OpenTelemetry for metrics in EventMesh #467

[ISSUE #340]Integrate With OpenTelemetry for metrics in EventMesh #467

Conversation

Roc-00 commented Jul 25, 2021

Motivation

Modifications

Documentation

codecov-commenter commented Jul 25, 2021 • edited Loading

Codecov Report

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

tydhot commented Jul 26, 2021

tydhot left a comment

Choose a reason for hiding this comment

tydhot commented Jul 26, 2021

ruanwenjun left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roc-00 commented Jul 26, 2021

tydhot commented Jul 27, 2021

qqeasonchen commented Jul 27, 2021

lrhkobe commented Jul 27, 2021 • edited Loading

Roc-00 commented Jul 27, 2021

Roc-00 commented Jul 27, 2021

qqeasonchen commented Jul 31, 2021

Roc-00 commented Aug 1, 2021

qqeasonchen left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Roc-00 commented Aug 2, 2021

xwm1992 commented Aug 2, 2021

Roc-00 commented Aug 4, 2021

Roc-00 commented Aug 4, 2021

Roc-00 commented Aug 5, 2021

codecov-commenter commented Jul 25, 2021 •

edited

Loading

lrhkobe commented Jul 27, 2021 •

edited

Loading