Optimization of the self-monitoring metrics architecture for Go (Golang) plugin modules and unified output of metrics with C++ modules. #1290

linrunqi08 · 2023-12-27T08:29:35Z

golang部分指标基本沿用原定义pkg/pipeline/self_metric.go
pkg/pipeline/context.go 中新增MetricRecord数据结构，概念对齐C++的指标结构
golang插件指标实现，主要看pluginmanager/processor_wrapper.go，现在调用processor插件，会统一经过processor_wrapper
MetricRecord会从processor_wrapper中生成，然后set到context中，具体的processor会在init的时候从context中把MetricRecord拿出来，这里的实现依赖的是单个LogstoreConfig的所有插件是单线程，顺序init的，否则该实现会有问题（现在的实现不需要改接口定义）
plugin_main/plugin_export.go定义了Cgo接口，go插件的指标数据通过Cgo接口一次返回，返回的数据结构为[]map[string]string. core/monitor/MetricExportor.cpp文件需要重点review内存释放逻辑

plugins/processor/encrypt/processor_encrypt.go

pluginmanager/context_imp.go

pluginmanager/self_telemetry_statistics.go

plugins/input/rdb/rdb.go

pluginmanager/flusher_wrapper.go

pluginmanager/processor_wrapper.go

pluginmanager/plugin_runner_helper.go

core/monitor/MetricExportor.cpp

pluginmanager/processor_wrapper.go

pluginmanager/context_imp.go

…e/taiye/add_go_plugin_metric

core/go_pipeline/LogtailPlugin.cpp

plugins/input/mysql/mysql.go

core/go_pipeline/LogtailPlugin.cpp

core/monitor/MetricExportor.cpp

core/pipeline/Pipeline.cpp

pluginmanager/aggregator_wrapper_v1.go

pluginmanager/logstore_config.go

pkg/helper/self_metric_imp.go

pkg/pipeline/context.go

plugin_main/plugin_export.go

core/monitor/LogtaiMetric.cpp

plugins/input/command/VGVzdFNjcmlwdFN0b3JhZ2U=.sh

pluginmanager/logstore_config.go

core/pipeline/Pipeline.cpp

pluginmanager/logstore_config.go

pluginmanager/logstore_config_test.go

linrunqi08 added 4 commits December 27, 2023 11:35

add metric code

0daec5e

add cpp call go

9847679

update call func

30810be

tmp debug

b8559c3

linrunqi08 changed the title ~~Feature/taiye/add go plugin metric~~ unify go plugin metric export Dec 27, 2023

linrunqi08 added 8 commits December 28, 2023 06:07

tmp save

f26e1a8

tmp save

e436d75

add flusher

7c38780

add flusher

33ea6a8

fix cpp interface

9617fb7

refine code

1c7da0f

refine code

1ad21ef

refine code

465b54c

linrunqi08 requested review from yyuuttaaoo and messixukejia December 29, 2023 08:56

linrunqi08 changed the title ~~unify go plugin metric export~~ Optimization of the self-monitoring metrics architecture for Go (Golang) plugin modules and unified output of metrics with C++ modules. Dec 29, 2023

messixukejia requested changes Dec 31, 2023

View reviewed changes

pluginmanager/processor_wrapper.go Outdated Show resolved Hide resolved

pluginmanager/context_imp.go Outdated Show resolved Hide resolved

linrunqi08 added 5 commits January 2, 2024 05:06

fix comments

1ec3dd3

fix comment

c1d90cf

fix comment

d8c2263

Merge branch '1.8' of https://github.com/alibaba/ilogtail into featur…

77ffaaa

…e/taiye/add_go_plugin_metric

fix comments

de51063

linrunqi08 force-pushed the feature/taiye/add_go_plugin_metric branch from 2f01470 to de51063 Compare January 9, 2024 04:15

linrunqi08 added 6 commits January 9, 2024 05:57

fix build

e19a9bd

add node id

09136c2

add child plugin id

9ba425d

fix metric

e104daa

fix unittest

96e8296

fix go lint

2107d4f

linrunqi08 force-pushed the feature/taiye/add_go_plugin_metric branch from fe48d07 to 20480fe Compare January 16, 2024 01:55

messixukejia requested changes Jan 17, 2024

View reviewed changes

yyuuttaaoo reviewed Jan 20, 2024

View reviewed changes

pkg/helper/self_metric_imp.go Show resolved Hide resolved

yyuuttaaoo reviewed Jan 20, 2024

View reviewed changes

pkg/pipeline/context.go Outdated Show resolved Hide resolved

yyuuttaaoo reviewed Jan 22, 2024

View reviewed changes

plugin_main/plugin_export.go Outdated Show resolved Hide resolved

yyuuttaaoo reviewed Jan 22, 2024

View reviewed changes

core/monitor/LogtaiMetric.cpp Outdated Show resolved Hide resolved

add comments

d18453e

yyuuttaaoo reviewed Jan 22, 2024

View reviewed changes

plugins/input/command/VGVzdFNjcmlwdFN0b3JhZ2U=.sh Outdated Show resolved Hide resolved

yyuuttaaoo reviewed Jan 22, 2024

View reviewed changes

pluginmanager/logstore_config.go Outdated Show resolved Hide resolved

pluginmanager/logstore_config.go Outdated Show resolved Hide resolved

linrunqi08 added 8 commits January 22, 2024 06:55

fix comments

0a2d671

fix comments

192337b

fix comments

3d7964b

fix comments

2b401c1

fix comments

c4ecf69

fix lint

a46c523

fix ut

d2aa7d3

fix c++ pluginID

e216f7e

linrunqi08 requested review from messixukejia and yyuuttaaoo January 22, 2024 11:17

messixukejia approved these changes Jan 22, 2024

View reviewed changes

yyuuttaaoo reviewed Jan 23, 2024

View reviewed changes

core/pipeline/Pipeline.cpp Show resolved Hide resolved

pluginmanager/logstore_config.go Outdated Show resolved Hide resolved

pluginmanager/logstore_config_test.go Outdated Show resolved Hide resolved

linrunqi08 added 2 commits January 23, 2024 06:30

fix comments

81e2942

fix ut

fa9eea1

yyuuttaaoo approved these changes Jan 23, 2024

View reviewed changes

linrunqi08 merged commit 476ac70 into 1.8 Jan 23, 2024
17 checks passed

yyuuttaaoo added enhancement Feature enhancement feature request New feature request and removed enhancement Feature enhancement labels Mar 27, 2024

yyuuttaaoo added this to the v1.8 milestone Mar 27, 2024

linrunqi08 deleted the feature/taiye/add_go_plugin_metric branch May 20, 2024 06:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimization of the self-monitoring metrics architecture for Go (Golang) plugin modules and unified output of metrics with C++ modules. #1290

Optimization of the self-monitoring metrics architecture for Go (Golang) plugin modules and unified output of metrics with C++ modules. #1290

linrunqi08 commented Dec 27, 2023 •

edited

Loading

Optimization of the self-monitoring metrics architecture for Go (Golang) plugin modules and unified output of metrics with C++ modules. #1290

Optimization of the self-monitoring metrics architecture for Go (Golang) plugin modules and unified output of metrics with C++ modules. #1290

Conversation

linrunqi08 commented Dec 27, 2023 • edited Loading

linrunqi08 commented Dec 27, 2023 •

edited

Loading