Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[exporter/elasticsearch] Replace go-elasticsearch BulkIndexer with go-docappender #32359

Merged
Merged
Show file tree
Hide file tree
Changes from 41 commits
Commits
Show all changes
50 commits
Select commit Hold shift + click to select a range
1ecda8e
Use go-docappender bulk indexer
carsonip Apr 15, 2024
4889835
Remove go mod replace directive
carsonip Apr 17, 2024
d4caa07
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 17, 2024
e2c4373
Use runtime.NumCPU() as default numWorkers
carsonip Apr 17, 2024
f518aa5
Add changelog
carsonip Apr 17, 2024
aed3960
Describe defaults in readme
carsonip Apr 18, 2024
accd2fb
worker flush should log, not return
carsonip Apr 18, 2024
0784b18
Workers use receive only channel
carsonip Apr 18, 2024
40d8987
Add godoc
carsonip Apr 18, 2024
e4accdc
Make bulkIndexerPool private
carsonip Apr 18, 2024
3ceb029
Grammar
carsonip Apr 18, 2024
0fcddd2
Update changelog
carsonip Apr 18, 2024
dde9f6c
Use go-docappender v2, accept pipeline config
carsonip Apr 18, 2024
3ed5349
Use wg instead of errgroup
carsonip Apr 18, 2024
03a094f
Log error returned by Add and configure client
carsonip Apr 18, 2024
e256d82
Use go-docappender v2.0.1
carsonip Apr 18, 2024
7748722
Fix test loop variable
carsonip Apr 18, 2024
926d817
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 18, 2024
c6ca007
go mod tidy all
carsonip Apr 18, 2024
43a2d53
Add back missing return
carsonip Apr 18, 2024
ad790d7
Check for closed channel without zero check
carsonip Apr 18, 2024
b6642b2
Fix channel closed check
carsonip Apr 18, 2024
274b862
Fix import
carsonip Apr 18, 2024
0ae62df
Use new go-docappender
carsonip Apr 22, 2024
d1e4f4f
Use 429 instead of 500 in integration test
carsonip Apr 22, 2024
cf342f2
Use go-docappender fix to pass integrationtest
carsonip Apr 23, 2024
324efdc
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 23, 2024
057c673
Revert integration test change and wait for rebase
carsonip Apr 23, 2024
7519621
Use upcoming retry.retry_on_status config
carsonip Apr 23, 2024
533331c
Bump go-docappender, remove go mod replace
carsonip Apr 23, 2024
0861fae
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 23, 2024
13cc9f9
Merge branch 'main' into go-docappender-bulkindexer
carsonip Apr 24, 2024
1b2a018
Log document level failures
carsonip Apr 24, 2024
13d35e3
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 3, 2024
6e163a7
Add back missing import
carsonip May 3, 2024
cbf5977
Bump go-docappender
carsonip May 3, 2024
ea7e5b0
go mod tidy
carsonip May 3, 2024
92ee227
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 6, 2024
d9886c1
Update changelog
carsonip May 6, 2024
fd85db1
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 6, 2024
69f8b59
Fix conflict in go.sum
carsonip May 6, 2024
3225676
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
e63d855
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
8bad25c
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
8106bfa
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 8, 2024
336b3c5
Update exporter/elasticsearchexporter/elasticsearch_bulk.go
carsonip May 9, 2024
fd1b4c7
Update exporter/elasticsearchexporter/elasticsearch_bulk.go
carsonip May 9, 2024
43ed76d
Use gzip.NoCompression for readability
carsonip May 9, 2024
2293b53
Merge branch 'main' into go-docappender-bulkindexer
carsonip May 9, 2024
6402cd4
Remove compression level config
carsonip May 9, 2024
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
33 changes: 33 additions & 0 deletions .chloggen/elasticsearchexporter_go-docappender.yaml
Original file line number Diff line number Diff line change
@@ -0,0 +1,33 @@
# Use this changelog template to create an entry for release notes.

# One of 'breaking', 'deprecation', 'new_component', 'enhancement', 'bug_fix'
change_type: enhancement

# The name of the component, or a single word describing the area of concern, (e.g. filelogreceiver)
component: elasticsearchexporter

# A brief description of the change. Surround your text with quotes ("") if it needs to start with a backtick (`).
note: Replace go-elasticsearch BulkIndexer with go-docappender

# Mandatory: One or more tracking issues related to the change. You can use the PR number here if no issue exists.
issues: [32378]

# (Optional) One or more lines of additional information to render under the primary note.
# These lines will be padded with 2 spaces and then inserted directly into the document.
# Use pipe (|) for multiline entries.
subtext: |
Replace go-elasticsearch BulkIndexer with go-docappender bulk indexer, in preparation for future reliability fixes.
As a result of this change, there are minor behavioral differences:
- flush timeout is now enforced on client side
- oversize payload special handling is now removed
- go-docappender uses bulk request filterPath which means bulk response is smaller, less JSON parsing and lower CPU usage
- document level retry debug logging is removed as retries are done transparently

# If your change doesn't affect end users or the exported elements of any package,
# you should instead start your pull request title with [chore] or use the "Skip Changelog" label.
# Optional: The change log or logs in which this entry should be included.
# e.g. '[user]' or '[user, api]'
# Include 'user' if the change is relevant to end users.
# Include 'api' if there is a change to a library API.
# Default: '[user]'
change_logs: [user]
11 changes: 11 additions & 0 deletions cmd/configschema/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -305,6 +305,7 @@ require (
github.com/apache/thrift v0.20.0 // indirect
github.com/ardielle/ardielle-go v1.5.2 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/aws/aws-sdk-go v1.51.32 // indirect
github.com/aws/aws-sdk-go-v2 v1.26.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
Expand Down Expand Up @@ -364,8 +365,13 @@ require (
github.com/eapache/go-resiliency v1.6.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/elastic/elastic-transport-go/v8 v8.5.0 // indirect
github.com/elastic/go-docappender/v2 v2.1.0 // indirect
github.com/elastic/go-elasticsearch/v7 v7.17.10 // indirect
github.com/elastic/go-elasticsearch/v8 v8.13.1 // indirect
github.com/elastic/go-structform v0.0.10 // indirect
github.com/elastic/go-sysinfo v1.7.1 // indirect
github.com/elastic/go-windows v1.0.1 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/envoyproxy/go-control-plane v0.12.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect
Expand Down Expand Up @@ -467,6 +473,7 @@ require (
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/jpillora/backoff v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
Expand Down Expand Up @@ -670,6 +677,9 @@ require (
github.com/yuin/gopher-lua v0.0.0-20220504180219-658193537a64 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.elastic.co/apm/module/apmzap/v2 v2.6.0 // indirect
go.elastic.co/apm/v2 v2.6.0 // indirect
go.elastic.co/fastjson v1.3.0 // indirect
go.etcd.io/bbolt v1.3.9 // indirect
go.mongodb.org/atlas v0.36.0 // indirect
go.mongodb.org/mongo-driver v1.15.0 // indirect
Expand Down Expand Up @@ -757,6 +767,7 @@ require (
gopkg.in/natefinch/lumberjack.v2 v2.2.1 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
gopkg.in/zorkian/go-datadog-api.v2 v2.30.0 // indirect
howett.net/plist v1.0.0 // indirect
k8s.io/api v0.29.3 // indirect
k8s.io/apimachinery v0.29.3 // indirect
k8s.io/client-go v0.29.3 // indirect
Expand Down
32 changes: 32 additions & 0 deletions cmd/configschema/go.sum

Some generated files are not rendered by default. Learn more about how customized files appear on GitHub.

11 changes: 11 additions & 0 deletions cmd/otelcontribcol/go.mod
Original file line number Diff line number Diff line change
Expand Up @@ -372,6 +372,7 @@ require (
github.com/apache/thrift v0.20.0 // indirect
github.com/ardielle/ardielle-go v1.5.2 // indirect
github.com/armon/go-metrics v0.4.1 // indirect
github.com/armon/go-radix v1.0.0 // indirect
github.com/aws/aws-sdk-go v1.51.32 // indirect
github.com/aws/aws-sdk-go-v2 v1.26.1 // indirect
github.com/aws/aws-sdk-go-v2/aws/protocol/eventstream v1.6.2 // indirect
Expand Down Expand Up @@ -431,8 +432,13 @@ require (
github.com/eapache/go-resiliency v1.6.0 // indirect
github.com/eapache/go-xerial-snappy v0.0.0-20230731223053-c322873962e3 // indirect
github.com/eapache/queue v1.1.0 // indirect
github.com/elastic/elastic-transport-go/v8 v8.5.0 // indirect
github.com/elastic/go-docappender/v2 v2.1.0 // indirect
github.com/elastic/go-elasticsearch/v7 v7.17.10 // indirect
github.com/elastic/go-elasticsearch/v8 v8.13.1 // indirect
github.com/elastic/go-structform v0.0.10 // indirect
github.com/elastic/go-sysinfo v1.7.1 // indirect
github.com/elastic/go-windows v1.0.1 // indirect
github.com/emicklei/go-restful/v3 v3.11.0 // indirect
github.com/envoyproxy/go-control-plane v0.12.0 // indirect
github.com/envoyproxy/protoc-gen-validate v1.0.4 // indirect
Expand Down Expand Up @@ -537,6 +543,7 @@ require (
github.com/jcmturner/gokrb5/v8 v8.4.4 // indirect
github.com/jcmturner/rpc/v2 v2.0.3 // indirect
github.com/jmespath/go-jmespath v0.4.0 // indirect
github.com/joeshaw/multierror v0.0.0-20140124173710-69b34d4ec901 // indirect
github.com/josharian/intern v1.0.0 // indirect
github.com/jpillora/backoff v1.0.0 // indirect
github.com/json-iterator/go v1.1.12 // indirect
Expand Down Expand Up @@ -717,6 +724,9 @@ require (
github.com/yuin/gopher-lua v0.0.0-20220504180219-658193537a64 // indirect
github.com/yusufpapurcu/wmi v1.2.4 // indirect
github.com/zeebo/xxh3 v1.0.2 // indirect
go.elastic.co/apm/module/apmzap/v2 v2.6.0 // indirect
go.elastic.co/apm/v2 v2.6.0 // indirect
go.elastic.co/fastjson v1.3.0 // indirect
go.etcd.io/bbolt v1.3.9 // indirect
go.mongodb.org/atlas v0.36.0 // indirect
go.mongodb.org/mongo-driver v1.15.0 // indirect
Expand Down Expand Up @@ -783,6 +793,7 @@ require (
gopkg.in/yaml.v2 v2.4.0 // indirect
gopkg.in/yaml.v3 v3.0.1 // indirect
gopkg.in/zorkian/go-datadog-api.v2 v2.30.0 // indirect
howett.net/plist v1.0.0 // indirect
k8s.io/api v0.29.3 // indirect
k8s.io/apimachinery v0.29.3 // indirect
k8s.io/client-go v0.29.3 // indirect
Expand Down
Loading
Loading