Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Update tools and benchmark datasets to read compressed json #174

Merged
merged 18 commits into from
Jun 12, 2023

Conversation

moh-osman3
Copy link
Contributor

@moh-osman3 moh-osman3 commented Jun 8, 2023

What does this PR do?

This PR

  • updates the [logs|traces|metrics] datasets to read files that are in a compressed json format. Files are read line by line. This logic is similar to how the otel contrib filereceiver component will read compressed json.
  • updates the [logs|traces|metrics] test data generating tools to write compressed json format. Telemetry is written one at a time followed by a newline. This logic is similar to how the otel contrib fileexporter will write compressed json.

@moh-osman3 moh-osman3 changed the title Using compressed reader for benchmark data Update tools and benchmark datasets to read compressed json Jun 9, 2023
@moh-osman3 moh-osman3 marked this pull request as ready for review June 9, 2023 09:20
Copy link
Contributor

@lquerel lquerel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

First of all, thanks for adding the support for the JSON format. But I'd like to keep the option to generate and read metrics/logs/traces data in a binary format. The main reason is that I have a bunch of data already used in various benchmarks and I need to keep them in order to reproduce them.

The JSON+ZSTD should be integrated as an additional supported format and not as a replacement for the binary format.

pkg/benchmark/dataset/real_logs_dataset.go Outdated Show resolved Hide resolved
tools/logs_benchmark/main.go Outdated Show resolved Hide resolved
tools/logs_benchmark/main.go Outdated Show resolved Hide resolved
tools/logs_gen/main.go Outdated Show resolved Hide resolved
@moh-osman3 moh-osman3 requested a review from lquerel June 12, 2023 20:36
@lquerel lquerel enabled auto-merge (squash) June 12, 2023 21:21
@lquerel
Copy link
Contributor

lquerel commented Jun 12, 2023

@moh-osman3 there is an issue in the build process preventing us to merge this PR. Could you fix it and I will merge this PR. Thanks.

auto-merge was automatically disabled June 12, 2023 23:09

Head branch was pushed to by a user without write access

@moh-osman3
Copy link
Contributor Author

@moh-osman3 there is an issue in the build process preventing us to merge this PR. Could you fix it and I will merge this PR. Thanks.

@lquerel Ahh forgot to update one of the test files with the updated function signature. I think it should be fixed now. Thank you!

@lquerel lquerel merged commit 223ff57 into f5:main Jun 12, 2023
@lquerel
Copy link
Contributor

lquerel commented Jun 12, 2023

@moh-osman3 thanks for this PR.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants