[exporter/elasticsearch] Data should not be dropped with persistent queue enabled when ES cluster is temporarily unavailable #30792

ycombinator · 2024-01-26T07:24:05Z

Component(s)

exporter/elasticsearch

What happened?

Description

If the Elasticsearch exporter is configured with a persistent queue (sending_queue.storage: filestorage) and new data is written to this persistent queue while the Elasticsearch cluster is temporarily unavailable, this data should be indexed into Elasticsearch once the cluster becomes available again.

Steps to Reproduce

Configure the collector like so:

receivers:
  filelog:
    include: /tmp/*.log
    storage: file_storage/filelogreceiver

exporters:
  elasticsearch:
    endpoints: [ "http://localhost:9200" ]
    index: foo
    flush:
      interval: 1s
    sending_queue:
      enabled: true
      storage: file_storage/elasticsearchexporter

extensions:
  file_storage/filelogreceiver:
    directory: /tmp/otelcol/file_storage/filelogreceiver
  file_storage/elasticsearchexporter:
    directory: /tmp/otelcol/file_storage/elasticsearchexporter
 
service:
  extensions: [file_storage/filelogreceiver, file_storage/elasticsearchexporter]
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [elasticsearch]
  telemetry:
    logs:
      level: debug

Start the collector.

Add a log entry to the file.

echo "1 foo bar $RANDOM" >> /tmp/test.log

Query the Elasticsearch cluster to verify that the log entry made it into the index.
Stop the Elasticsearch cluster.

Add a new log entry to the file.

echo "2 foo bar $RANDOM" >> /tmp/test.log

Restart the Elasticsearch cluster.
Query the Elasticsearch cluster to verify that the second log entry made it into the index.

Expected Result

In step 8, there should be two log entries in the index.

Actual Result

In step 8, only the first log entry (added in step 3) is in the index.

Collector version

v0.93.0

Environment information

Environment

OS: (e.g., "Ubuntu 20.04") MacOS
Compiler(if manually compiled): (e.g., "go 14.2") go 1.21.6

OpenTelemetry Collector configuration

receivers:
  filelog:
    include: /tmp/*.log
    storage: file_storage/filelogreceiver

exporters:
  elasticsearch:
    endpoints: [ "http://localhost:9200" ]
    index: foo
    flush:
      interval: 1s
    sending_queue:
      enabled: true
      storage: file_storage/elasticsearchexporter

extensions:
  file_storage/filelogreceiver:
    directory: /tmp/otelcol/file_storage/filelogreceiver
  file_storage/elasticsearchexporter:
    directory: /tmp/otelcol/file_storage/elasticsearchexporter
    
service:
  extensions: [file_storage/filelogreceiver, file_storage/elasticsearchexporter]
  pipelines:
    logs:
      receivers: [filelog]
      exporters: [elasticsearch]
  telemetry:
    logs:
      level: debug

Log output

No response

Additional context

No response

The text was updated successfully, but these errors were encountered:

github-actions · 2024-01-26T07:24:19Z

Pinging code owners:

exporter/elasticsearch: @JaredTan95

See Adding Labels via Comments if you do not have permissions to add labels yourself.

crobert-1 · 2024-01-26T17:00:56Z

Hello @ycombinator, can you share collector logs during this process, preferably with logging set to debug verbosity?

ycombinator · 2024-01-30T20:35:37Z

Hi @crobert-1, here are the collector logs over the course of all the steps, at debug level:

$ ./bin/otelcontribcol_darwin_arm64 --config config.yaml
2024-01-30T12:29:33.548-0800	info	service@v0.93.1-0.20240130182548-89388addcc7f/telemetry.go:59	Setting up own telemetry...
2024-01-30T12:29:33.548-0800	info	service@v0.93.1-0.20240130182548-89388addcc7f/telemetry.go:104	Serving metrics	{"address": ":8888", "level": "Basic"}
2024-01-30T12:29:33.548-0800	debug	extension@v0.93.1-0.20240130182548-89388addcc7f/extension.go:165	Beta component. May change in the future.	{"kind": "extension", "name": "file_storage/filelogreceiver"}
2024-01-30T12:29:33.548-0800	debug	extension@v0.93.1-0.20240130182548-89388addcc7f/extension.go:165	Beta component. May change in the future.	{"kind": "extension", "name": "file_storage/elasticsearchexporter"}
2024-01-30T12:29:33.548-0800	debug	exporter@v0.93.1-0.20240130182548-89388addcc7f/exporter.go:273	Beta component. May change in the future.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
2024-01-30T12:29:33.548-0800	warn	elasticsearchexporter@v0.93.0/factory.go:78	index option are deprecated and replaced with logs_index and traces_index.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
2024-01-30T12:29:33.551-0800	debug	receiver@v0.93.1-0.20240130182548-89388addcc7f/receiver.go:294	Beta component. May change in the future.	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:29:33.551-0800	info	service@v0.93.1-0.20240130182548-89388addcc7f/service.go:140	Starting otelcontribcol...	{"Version": "0.93.0-dev", "NumCPU": 12}
2024-01-30T12:29:33.551-0800	info	extensions/extensions.go:34	Starting extensions...
2024-01-30T12:29:33.551-0800	info	extensions/extensions.go:37	Extension is starting...	{"kind": "extension", "name": "file_storage/elasticsearchexporter"}
2024-01-30T12:29:33.551-0800	info	extensions/extensions.go:52	Extension started.	{"kind": "extension", "name": "file_storage/elasticsearchexporter"}
2024-01-30T12:29:33.551-0800	info	extensions/extensions.go:37	Extension is starting...	{"kind": "extension", "name": "file_storage/filelogreceiver"}
2024-01-30T12:29:33.551-0800	info	extensions/extensions.go:52	Extension started.	{"kind": "extension", "name": "file_storage/filelogreceiver"}
2024-01-30T12:29:33.561-0800	info	internal/persistent_queue.go:143	Initializing new persistent queue	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
2024-01-30T12:29:33.561-0800	debug	internal/persistent_queue.go:405	Checking if there are items left for dispatch by consumers	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
2024-01-30T12:29:33.561-0800	debug	internal/persistent_queue.go:416	No items left for dispatch by consumers	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
2024-01-30T12:29:33.561-0800	info	adapter/receiver.go:45	Starting stanza receiver	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:29:33.569-0800	debug	pipeline/directed.go:59	Starting operator	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:29:33.569-0800	debug	pipeline/directed.go:63	Started operator	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:29:33.569-0800	debug	pipeline/directed.go:59	Starting operator	{"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "file_input", "operator_type": "file_input"}
2024-01-30T12:29:33.569-0800	warn	fileconsumer/file.go:51	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:33.569-0800	debug	pipeline/directed.go:63	Started operator	{"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "file_input", "operator_type": "file_input"}
2024-01-30T12:29:33.569-0800	debug	adapter/converter.go:109	Starting log converter	{"kind": "receiver", "name": "filelog", "data_type": "logs", "worker_count": 3}
2024-01-30T12:29:33.569-0800	info	service@v0.93.1-0.20240130182548-89388addcc7f/service.go:166	Everything is ready. Begin running and processing data.
2024-01-30T12:29:33.569-0800	warn	localhostgate/featuregate.go:63	The default endpoints for all servers in components will change to use localhost instead of 0.0.0.0 in a future version. Use the feature gate to preview the new default.	{"feature gate ID": "component.UseLocalHostAsDefaultHost"}
2024-01-30T12:29:33.771-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:33.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:33.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:33.971-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:33.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:33.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.171-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.371-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.571-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.771-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.971-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:34.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.171-0800	debug	fileconsumer/file.go:131	finding files: no files match the configured criteria	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  []})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  []}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.370-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.371-0800	info	fileconsumer/file.go:268	Started watching file	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer", "path": "/tmp/test.log"}
2024-01-30T12:29:35.568-0800	debug	elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:42	Request roundtrip completed.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch", "path": "/", "method": "GET", "duration": 0.016299, "status": "200 OK"}
2024-01-30T12:29:35.570-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.570-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.712-0800	debug	elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:42	Request roundtrip completed.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch", "path": "/_bulk", "method": "POST", "duration": 0.143966, "status": "200 OK"}
2024-01-30T12:29:35.770-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.770-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:35.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:36.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:37.170-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:37.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.370-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:45.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.772-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.970-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:46.970-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:47.170-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:47.170-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:55.553-0800	error	elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:49	Request failed.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch", "reason": "dial tcp 127.0.0.1:9200: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*clientLogger).LogRoundTrip
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:49
github.com/elastic/go-elasticsearch/v7/estransport.(*Client).logRoundTrip
	github.com/elastic/go-elasticsearch/v7@v7.17.10/estransport/estransport.go:576
github.com/elastic/go-elasticsearch/v7/estransport.(*Client).Perform
	github.com/elastic/go-elasticsearch/v7@v7.17.10/estransport/estransport.go:372
github.com/elastic/go-elasticsearch/v7.(*Client).Perform
	github.com/elastic/go-elasticsearch/v7@v7.17.10/elasticsearch.go:304
github.com/elastic/go-elasticsearch/v7/esapi.BulkRequest.Do
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esapi/api.bulk.go:188
github.com/elastic/go-elasticsearch/v7/esutil.(*worker).flush
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:553
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:321
2024-01-30T12:29:55.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:55.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:55.651-0800	error	elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:49	Request failed.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch", "reason": "dial tcp 127.0.0.1:9200: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*clientLogger).LogRoundTrip
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:49
github.com/elastic/go-elasticsearch/v7/estransport.(*Client).logRoundTrip
	github.com/elastic/go-elasticsearch/v7@v7.17.10/estransport/estransport.go:576
github.com/elastic/go-elasticsearch/v7/estransport.(*Client).Perform
	github.com/elastic/go-elasticsearch/v7@v7.17.10/estransport/estransport.go:372
github.com/elastic/go-elasticsearch/v7.(*Client).Perform
	github.com/elastic/go-elasticsearch/v7@v7.17.10/elasticsearch.go:304
github.com/elastic/go-elasticsearch/v7/esapi.BulkRequest.Do
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esapi/api.bulk.go:188
github.com/elastic/go-elasticsearch/v7/esutil.(*worker).flush
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:553
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:321
2024-01-30T12:29:55.743-0800	error	elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:49	Request failed.	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch", "reason": "dial tcp 127.0.0.1:9200: connect: connection refused"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.(*clientLogger).LogRoundTrip
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:49
github.com/elastic/go-elasticsearch/v7/estransport.(*Client).logRoundTrip
	github.com/elastic/go-elasticsearch/v7@v7.17.10/estransport/estransport.go:576
github.com/elastic/go-elasticsearch/v7/estransport.(*Client).Perform
	github.com/elastic/go-elasticsearch/v7@v7.17.10/estransport/estransport.go:372
github.com/elastic/go-elasticsearch/v7.(*Client).Perform
	github.com/elastic/go-elasticsearch/v7@v7.17.10/elasticsearch.go:304
github.com/elastic/go-elasticsearch/v7/esapi.BulkRequest.Do
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esapi/api.bulk.go:188
github.com/elastic/go-elasticsearch/v7/esutil.(*worker).flush
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:553
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:321
2024-01-30T12:29:55.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:55.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:55.919-0800	error	elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:150	Bulk indexer error: flush: dial tcp 127.0.0.1:9200: connect: connection refused	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:150
github.com/elastic/go-elasticsearch/v7/esutil.(*worker).flush
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:557
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:321
2024-01-30T12:29:55.919-0800	error	elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:150	Bulk indexer error: flush: dial tcp 127.0.0.1:9200: connect: connection refused	{"kind": "exporter", "data_type": "logs", "name": "elasticsearch"}
github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter.newBulkIndexer.func1
	github.com/open-telemetry/opentelemetry-collector-contrib/exporter/elasticsearchexporter@v0.93.0/elasticsearch_bulk.go:150
github.com/elastic/go-elasticsearch/v7/esutil.(*bulkIndexer).init.func1
	github.com/elastic/go-elasticsearch/v7@v7.17.10/esutil/bulk_indexer.go:324
2024-01-30T12:29:55.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:55.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.170-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.170-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:56.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:57.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:57.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:57.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:29:57.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:05.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:05.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:05.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:05.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:05.970-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:05.970-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.970-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:06.970-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:07.170-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:07.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:07.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:07.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:15.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:15.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:15.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:15.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.370-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.370-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.771-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.771-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.971-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:16.971-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:17.171-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:17.171-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:17.371-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:17.371-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:17.571-0800	debug	fileconsumer/file.go:135	matched files%!(EXTRA zapcore.Field={paths 1 0  [/tmp/test.log]})	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
2024-01-30T12:30:17.571-0800	debug	fileconsumer/file.go:167	Consuming files{paths 1 0  [/tmp/test.log]}	{"kind": "receiver", "name": "filelog", "data_type": "logs", "component": "fileconsumer"}
^C2024-01-30T12:30:20.487-0800	info	otelcol@v0.93.1-0.20240130182548-89388addcc7f/collector.go:258	Received signal from OS	{"signal": "interrupt"}
2024-01-30T12:30:20.487-0800	info	service@v0.93.1-0.20240130182548-89388addcc7f/service.go:181	Starting shutdown...
2024-01-30T12:30:20.487-0800	info	adapter/receiver.go:140	Stopping stanza receiver	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:30:20.487-0800	debug	pipeline/directed.go:74	Stopping operator	{"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "file_input", "operator_type": "file_input"}
2024-01-30T12:30:20.488-0800	debug	pipeline/directed.go:78	Stopped operator	{"kind": "receiver", "name": "filelog", "data_type": "logs", "operator_id": "file_input", "operator_type": "file_input"}
2024-01-30T12:30:20.488-0800	debug	pipeline/directed.go:74	Stopping operator	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:30:20.488-0800	debug	pipeline/directed.go:78	Stopped operator	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:30:20.488-0800	debug	adapter/receiver.go:120	Converter channel got closed	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:30:20.488-0800	debug	adapter/receiver.go:115	Consumer loop stopped	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:30:20.488-0800	debug	adapter/receiver.go:90	Receive loop stopped	{"kind": "receiver", "name": "filelog", "data_type": "logs"}
2024-01-30T12:30:20.488-0800	info	extensions/extensions.go:59	Stopping extensions...
2024-01-30T12:30:20.488-0800	info	service@v0.93.1-0.20240130182548-89388addcc7f/service.go:195	Shutdown complete.

hvaghani221 · 2024-02-06T09:28:44Z

The sending queue doesn't retry when the upload fails. To enable retry_on_failure , 2 changes are required:

Add exporterhelper.WithRetry along with exporterhelper.WithQueue in factory method

opentelemetry-collector-contrib/exporter/elasticsearchexporter/factory.go

Lines 88 to 95 in 0dc5c1f

    
           return exporterhelper.NewLogsExporter( 
        
           	ctx, 
        
           	set, 
        
           	cfg, 
        
           	logsExporter.pushLogsData, 
        
           	exporterhelper.WithShutdown(logsExporter.Shutdown), 
        
           	exporterhelper.WithQueue(cf.QueueSettings), 
        
           )

Return the consumererror.Logs error from pushLogData/pushTraceData when upload fails. The consumererror signals the retry helper to retry the data.

The second step will be a little tricky to implement since the bulk indexer delays uploading data until the buffer is full. So, the exporter cannot immediately know if the upload fails.

github-actions · 2024-04-08T03:29:50Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/elasticsearch: @JaredTan95 @ycombinator

See Adding Labels via Comments if you do not have permissions to add labels yourself.

carsonip · 2024-04-16T17:12:12Z

I'm working on a few PRs to improve the reliability of Elasticsearch exporter in cases like ES downtime and collector crashes. The plan is that #32377 resolves this very issue.

github-actions · 2024-06-17T03:32:37Z

This issue has been inactive for 60 days. It will be closed in 60 days if there is no activity. To ping code owners by adding a component label, see Adding Labels via Comments, or if you are unsure of which component this issue relates to, please ping @open-telemetry/collector-contrib-triagers. If this issue is still relevant, please ping the code owners or leave a comment explaining why it is still relevant. Otherwise, please close it.

Pinging code owners:

exporter/elasticsearch: @JaredTan95 @ycombinator @carsonip

See Adding Labels via Comments if you do not have permissions to add labels yourself.

carsonip · 2024-06-17T09:25:46Z

#32377 is still WIP.

carsonip · 2024-08-14T14:33:29Z

This is completed via #32377 (PR: #34238). Requires opt-in for the new behavior.

This issue can be closed.

ycombinator added bug Something isn't working needs triage New item requiring triage labels Jan 26, 2024

github-actions bot added the exporter/elasticsearch label Jan 26, 2024

github-actions bot mentioned this issue Jan 30, 2024

Weekly Report: 2024-01-23 - 2024-01-30 #30848

Closed

github-actions bot mentioned this issue Feb 6, 2024

Weekly Report: 2024-01-30 - 2024-02-06 #31055

Closed

crobert-1 added enhancement New feature or request and removed bug Something isn't working needs triage New item requiring triage labels Feb 6, 2024

lahsivjar mentioned this issue Mar 20, 2024

[exporter/elasticsearch] Add reliability tests #31848

Merged

ycombinator mentioned this issue Apr 2, 2024

REQUEST: New membership for ycombinator open-telemetry/community#2031

Closed

6 tasks

github-actions bot added the Stale label Apr 8, 2024

crobert-1 removed the Stale label Apr 8, 2024

This was referenced Apr 15, 2024

[exporter/elasticsearch] Use batch sender exporter helper for reliability #32377

Closed

[exporter/elasticsearch] Replace go-elasticsearch with go-docappender BulkIndexer to prepare for reliability fixes #32378

Closed

lahsivjar mentioned this issue Apr 18, 2024

[exporter/elasticsearch] Add micro/macro benchmarks to reason about improvement PRs #32504

Closed

github-actions bot added the Stale label Jun 17, 2024

crobert-1 removed the Stale label Jun 17, 2024

mashhurs mentioned this issue Jul 15, 2024

Add SHA512 converter. #34050

Merged

crobert-1 closed this as completed Aug 14, 2024

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

[exporter/elasticsearch] Data should not be dropped with persistent queue enabled when ES cluster is temporarily unavailable #30792

[exporter/elasticsearch] Data should not be dropped with persistent queue enabled when ES cluster is temporarily unavailable #30792

ycombinator commented Jan 26, 2024 •

edited

Loading

github-actions bot commented Jan 26, 2024

crobert-1 commented Jan 26, 2024

ycombinator commented Jan 30, 2024

hvaghani221 commented Feb 6, 2024

github-actions bot commented Apr 8, 2024

carsonip commented Apr 16, 2024

github-actions bot commented Jun 17, 2024

carsonip commented Jun 17, 2024

carsonip commented Aug 14, 2024 •

edited

Loading

[exporter/elasticsearch] Data should not be dropped with persistent queue enabled when ES cluster is temporarily unavailable #30792

[exporter/elasticsearch] Data should not be dropped with persistent queue enabled when ES cluster is temporarily unavailable #30792

Comments

ycombinator commented Jan 26, 2024 • edited Loading

Component(s)

What happened?

Description

Steps to Reproduce

Expected Result

Actual Result

Collector version

Environment information

Environment

OpenTelemetry Collector configuration

Log output

Additional context

github-actions bot commented Jan 26, 2024

crobert-1 commented Jan 26, 2024

ycombinator commented Jan 30, 2024

hvaghani221 commented Feb 6, 2024

github-actions bot commented Apr 8, 2024

carsonip commented Apr 16, 2024

github-actions bot commented Jun 17, 2024

carsonip commented Jun 17, 2024

carsonip commented Aug 14, 2024 • edited Loading

ycombinator commented Jan 26, 2024 •

edited

Loading

carsonip commented Aug 14, 2024 •

edited

Loading