[checkoutservice] add basic metric support #339

fatsheep9146 · 2022-08-23T14:03:27Z

Signed-off-by: Ziqi Zhao zhaoziqi9146@gmail.com

related #69.

Changes

Try to enhance checkoutservice with metric support.

Add basic runtime metrics for checkoutservice, using runtime metric instrumentation library.

Here is the metrics example, process_runtime_go_goroutines

Following paragraph is the earlier content of this pull request, but these efforts are blocked by open-telemetry/opentelemetry-go-contrib#2700, so remove them for now.

According to https://github.com/open-telemetry/opentelemetry-specification/blob/main/specification/metrics/semantic_conventions/rpc.md and as the first try, I first add a metrics 'rpc.server.duration' for unary service.

~~the metrics with four label~~
~~- rpc.system~~
~~- rpc.service~~
~~- rpc.method~~
~~- rpc.grpc.status_code~~

I made some experiments with new metric, I stop 'shipservice' manually to mock the fail down of shipservice and restart it after a few minutes, and draw a graph indicates the success ratio of checkout service.

~~the query I used is sum(rate(rpc_server_duration_count{rpc_grpc_status_code="0"}[1m])) / sum(rate(rpc_server_duration_count[1m]))~~

~~and the graph is like following with a drop in the middle~~

src/checkoutservice/main.go

src/checkoutservice/middleware.go

reyang

#339 (comment)

github-actions · 2022-09-01T03:57:41Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

fatsheep9146 · 2022-09-01T04:47:08Z

#339 (comment)

open-telemetry/opentelemetry-go-contrib#2700

@reyang @austinlparker

I also started to add metrics support for grpc instrumentation library. Should we wait until this pr merged and then update checkout service's library?

github-actions · 2022-09-09T03:55:53Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

fatsheep9146 · 2022-09-09T04:54:13Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

Related pull requests open-telemetry/opentelemetry-go-contrib#2700 are still being reviewed, not stale.

github-actions · 2022-09-17T03:52:37Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

github-actions · 2022-09-26T04:08:09Z

This PR was marked stale due to lack of activity. It will be closed in 7 days.

cartersocha · 2022-09-29T16:18:47Z

hey @fatsheep9146 lets try to push this forward for the hero scenario asap. Our development freeze starts October 10th.

All we need in the checkout service is a gauge on the cache size + instrumentation libraries

cartersocha · 2022-09-29T16:19:05Z

@austinlparker, @julianocosta89, @puckpuck fyi lets try to push this forward

docker-compose.yml

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

fatsheep9146 · 2022-09-30T09:42:44Z

I refract this pr to use the new version of golang metric sdk and using runtime instrumentation library. Please help review this =D
@cartersocha @austinlparker @puckpuck @julianocosta89

docker-compose.yml

PR has been pivoted and needs a fresh review

julianocosta89

I wasn't able to test this out.

Building the service is failing with the following message:

=> ERROR [builder 10/10] RUN go build -o /go/bin/checkoutservice/ ./                                                                    6.1s 
------                                                                                                                                        
 > [builder 10/10] RUN go build -o /go/bin/checkoutservice/ ./:                                                                               
#0 3.911 # go.opentelemetry.io/otel/sdk/metric/metricdata                                                                                     
#0 3.911 /go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v0.32.1/metricdata/data.go:62:14: syntax error: unexpected int64, expecting ]        
#0 3.911 /go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v0.32.1/metricdata/data.go:67:15: syntax error: unexpected ), expecting type         
#0 3.911 /go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v0.32.1/metricdata/data.go:70:12: syntax error: unexpected int64, expecting ]        
#0 3.911 /go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v0.32.1/metricdata/data.go:80:13: syntax error: unexpected ), expecting type
#0 3.911 /go/pkg/mod/go.opentelemetry.io/otel/sdk/metric@v0.32.1/metricdata/data.go:83:18: syntax error: unexpected int64, expecting ]
#0 3.911 note: module requires Go 1.18
------
failed to solve: executor failed running [/bin/sh -c go build -o /go/bin/checkoutservice/ ./]: exit code: 2

I've tried updating the go version within the Dockerfileand go.mod, but then the build failed with the following:

=> ERROR [builder  9/10] RUN protoc -I ./proto/ ./proto/demo.proto --go_out=./ --go-grpc_out=./                                         0.4s 
------                                                                                                                                        
 > [builder  9/10] RUN protoc -I ./proto/ ./proto/demo.proto --go_out=./ --go-grpc_out=./:
#0 0.412 protoc-gen-go: program not found or is not executable
#0 0.412 Please specify a program using absolute path or make sure the program is available in your PATH system variable
#0 0.412 --go_out: protoc-gen-go: Plugin failed with status code 1.
------
failed to solve: executor failed running [/bin/sh -c protoc -I ./proto/ ./proto/demo.proto --go_out=./ --go-grpc_out=./]: exit code: 1

austinlparker · 2022-10-03T15:58:27Z

I'm investigating this - looks like part of it is that the go version in the container isn't 1.18+

julianocosta89 · 2022-10-03T16:08:35Z

@austinlparker yes, but when I tried updating the go version within the Dockerfile and go.mod, the build failed with the following:

=> ERROR [builder  9/10] RUN protoc -I ./proto/ ./proto/demo.proto --go_out=./ --go-grpc_out=./                                                       0.6s
------
 > [builder  9/10] RUN protoc -I ./proto/ ./proto/demo.proto --go_out=./ --go-grpc_out=./:
#0 0.552 protoc-gen-go: program not found or is not executable
#0 0.552 Please specify a program using absolute path or make sure the program is available in your PATH system variable
#0 0.553 --go_out: protoc-gen-go: Plugin failed with status code 1.
------
failed to solve: executor failed running [/bin/sh -c protoc -I ./proto/ ./proto/demo.proto --go_out=./ --go-grpc_out=./]: exit code: 1

austinlparker · 2022-10-03T16:26:58Z

yeah, I think we've encountered a dependency hell loop. we may have to update our go stuff to use google.golang.org/protobuf rather than protoc-gen-go so we can break into 1.18+... lemme see if that's an easy upgrade

austinlparker · 2022-10-03T16:37:26Z

@fatsheep9146 it looks like in order to take newer versions of otel-go we will need to upgrade all go services to support 1.18+. I believe we will also need to update to the newer protobuf package. Can you take care of that for this PR as well?

fatsheep9146 · 2022-10-04T01:01:52Z

@fatsheep9146 it looks like in order to take newer versions of otel-go we will need to upgrade all go services to support 1.18+. I believe we will also need to update to the newer protobuf package. Can you take care of that for this PR as well?

Yes, I already found this problem and started doing it. I also found that in 1.18, go get can would not install the package, so the dockerfile should also be updated. @austinlparker

austinlparker · 2022-10-04T01:03:39Z

@fatsheep9146 it looks like in order to take newer versions of otel-go we will need to upgrade all go services to support 1.18+. I believe we will also need to update to the newer protobuf package. Can you take care of that for this PR as well?

Yes, I already found this problem and started doing it. I also found that in 1.18, go get can would not install the package, so the dockerfile should also be updated. @austinlparker

Yeah - I would just upgrade to 1.19 across the board, as well as switch to the newer Google protobuf package. Maybe best to do that in a new PR just for cleanliness sake.

fatsheep9146 · 2022-10-04T01:41:15Z

@fatsheep9146 it looks like in order to take newer versions of otel-go we will need to upgrade all go services to support 1.18+. I believe we will also need to update to the newer protobuf package. Can you take care of that for this PR as well?

Yes, I already found this problem and started doing it. I also found that in 1.18, go get can would not install the package, so the dockerfile should also be updated. @austinlparker

Yeah - I would just upgrade to 1.19 across the board, as well as switch to the newer Google protobuf package. Maybe best to do that in a new PR just for cleanliness sake.

Ok, I will first submit a new pull request to update the dockerfile and protobuf version. After that pr is merged I will rebase this pr then.

julianocosta89 · 2022-10-06T21:57:09Z

Hello all,

I've just sent a PR with the required update: #406

Once, this PR is merged we can simply rebase and merge this one.
Already tested locally and it is working 🥳

cartersocha · 2022-10-07T00:50:38Z

@fatsheep9146 the go version is now updated! thanks @julianocosta89

fatsheep9146 · 2022-10-07T01:30:22Z

@fatsheep9146 the go version is now updated! thanks @julianocosta89

I also tested it locally, it worked, thanks! @cartersocha @julianocosta89

julianocosta89

LGTM

julianocosta89 · 2022-10-07T19:31:14Z

@reyang anything missing in here?

reyang · 2022-10-07T19:32:32Z

@reyang anything missing in here?

Let me take a quick look.

reyang

LGTM.

reyang · 2022-10-07T19:34:51Z

I don't know if this needs a changelog entry or not. Not a blocker though.

cartersocha · 2022-10-07T19:37:14Z

I'll cover the changelog update in a separate PR to move this forward

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com> Co-authored-by: Carter Socha <43380952+cartersocha@users.noreply.github.com> Co-authored-by: Juliano Costa <julianocosta89@outlook.com>

fatsheep9146 requested a review from a team August 23, 2022 14:03

cedricziel reviewed Aug 23, 2022

View reviewed changes

src/checkoutservice/main.go Outdated Show resolved Hide resolved

reyang reviewed Aug 23, 2022

View reviewed changes

src/checkoutservice/middleware.go Outdated Show resolved Hide resolved

reyang reviewed Aug 23, 2022

View reviewed changes

src/checkoutservice/middleware.go Outdated Show resolved Hide resolved

reyang reviewed Aug 23, 2022

View reviewed changes

src/checkoutservice/middleware.go Outdated Show resolved Hide resolved

reyang previously requested changes Aug 23, 2022

View reviewed changes

github-actions bot added the Stale label Sep 1, 2022

fatsheep9146 removed the Stale label Sep 1, 2022

github-actions bot added the Stale label Sep 9, 2022

fatsheep9146 removed the Stale label Sep 9, 2022

github-actions bot added the Stale label Sep 17, 2022

fatsheep9146 removed the Stale label Sep 18, 2022

github-actions bot added the Stale label Sep 26, 2022

reyang removed the Stale label Sep 27, 2022

arminru reviewed Sep 30, 2022

View reviewed changes

docker-compose.yml Show resolved Hide resolved

[checkoutservice] add basic runtime metric

b4b653b

Signed-off-by: Ziqi Zhao <zhaoziqi9146@gmail.com>

fatsheep9146 force-pushed the checkoutservice-metrics-enhancement branch from 770cd2f to b4b653b Compare September 30, 2022 09:31

cijothomas reviewed Sep 30, 2022

View reviewed changes

docker-compose.yml Show resolved Hide resolved

cartersocha requested a review from reyang September 30, 2022 17:14

Merge branch 'main' into checkoutservice-metrics-enhancement

13979d4

julianocosta89 reviewed Oct 1, 2022

View reviewed changes

julianocosta89 mentioned this pull request Oct 6, 2022

Update go version to use go otel metrics #406

Closed

2 tasks

mic-max approved these changes Oct 7, 2022

View reviewed changes

cartersocha added 2 commits October 6, 2022 17:50

Merge branch 'main' into checkoutservice-metrics-enhancement

c89b7b3

Merge branch 'main' into checkoutservice-metrics-enhancement

7ce2a27

julianocosta89 approved these changes Oct 7, 2022

View reviewed changes

Merge branch 'main' into checkoutservice-metrics-enhancement

14779e9

reyang approved these changes Oct 7, 2022

View reviewed changes

cartersocha merged commit ca30e16 into open-telemetry:main Oct 7, 2022

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[checkoutservice] add basic metric support #339

[checkoutservice] add basic metric support #339

fatsheep9146 commented Aug 23, 2022 •

edited

Loading

reyang left a comment

github-actions bot commented Sep 1, 2022

fatsheep9146 commented Sep 1, 2022 •

edited

Loading

github-actions bot commented Sep 9, 2022

fatsheep9146 commented Sep 9, 2022

github-actions bot commented Sep 17, 2022

github-actions bot commented Sep 26, 2022

cartersocha commented Sep 29, 2022

cartersocha commented Sep 29, 2022

fatsheep9146 commented Sep 30, 2022

julianocosta89 left a comment

austinlparker commented Oct 3, 2022

julianocosta89 commented Oct 3, 2022

austinlparker commented Oct 3, 2022

austinlparker commented Oct 3, 2022

fatsheep9146 commented Oct 4, 2022

austinlparker commented Oct 4, 2022

fatsheep9146 commented Oct 4, 2022

julianocosta89 commented Oct 6, 2022

cartersocha commented Oct 7, 2022 •

edited

Loading

fatsheep9146 commented Oct 7, 2022

julianocosta89 left a comment

julianocosta89 commented Oct 7, 2022

reyang commented Oct 7, 2022

reyang left a comment

reyang commented Oct 7, 2022

cartersocha commented Oct 7, 2022

[checkoutservice] add basic metric support #339

[checkoutservice] add basic metric support #339

Conversation

fatsheep9146 commented Aug 23, 2022 • edited Loading

Changes

reyang left a comment

Choose a reason for hiding this comment

github-actions bot commented Sep 1, 2022

fatsheep9146 commented Sep 1, 2022 • edited Loading

github-actions bot commented Sep 9, 2022

fatsheep9146 commented Sep 9, 2022

github-actions bot commented Sep 17, 2022

github-actions bot commented Sep 26, 2022

cartersocha commented Sep 29, 2022

cartersocha commented Sep 29, 2022

fatsheep9146 commented Sep 30, 2022

julianocosta89 left a comment

Choose a reason for hiding this comment

austinlparker commented Oct 3, 2022

julianocosta89 commented Oct 3, 2022

austinlparker commented Oct 3, 2022

austinlparker commented Oct 3, 2022

fatsheep9146 commented Oct 4, 2022

austinlparker commented Oct 4, 2022

fatsheep9146 commented Oct 4, 2022

julianocosta89 commented Oct 6, 2022

cartersocha commented Oct 7, 2022 • edited Loading

fatsheep9146 commented Oct 7, 2022

julianocosta89 left a comment

Choose a reason for hiding this comment

julianocosta89 commented Oct 7, 2022

reyang commented Oct 7, 2022

reyang left a comment

Choose a reason for hiding this comment

reyang commented Oct 7, 2022

cartersocha commented Oct 7, 2022

fatsheep9146 commented Aug 23, 2022 •

edited

Loading

fatsheep9146 commented Sep 1, 2022 •

edited

Loading

cartersocha commented Oct 7, 2022 •

edited

Loading