Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Obfuscation processor and example #191

Merged
merged 13 commits into from
Jun 29, 2023
Merged

Obfuscation processor and example #191

merged 13 commits into from
Jun 29, 2023

Conversation

jmacd
Copy link
Collaborator

@jmacd jmacd commented Jun 28, 2023

This PR was originally authored by @moh-osman3. With him out of town, I've updated it with several commits in my fork, so this is a replacement PR.

This PR:

  • adds Lightstep's generatorreceiver to create synthetic telemetry in a collector pipeline
  • adds obfuscation processor that will obfuscate all attributes with string values when recording telemetry

Simple Test

$ cd collector/examples/synthesize
$ go run ../../cmd/otelarrowcol --config ./record.yaml

This writes zstd compressed and obfuscated traces to an output file (e.g cmd/otelarrowcol/recorded_traces.json)

This can be replayed by running

$ cd collector/examples
$ go run ../../cmd/otelarrowcol --config ./replay.yaml

This reads the data in the recorded file and rewrites it in an uncompressed form

You can also read in the compressed file recorded_traces.json with the the benchmark tools e.g.

$ go run ./tools/trace_benchmark --format json collector/cmd/otelarrowcol/recorded_traces.json

(cc @moh-osman3 @lquerel)

@jmacd jmacd requested a review from lquerel June 28, 2023 21:26
@jmacd
Copy link
Collaborator Author

jmacd commented Jun 28, 2023

PTAL @lquerel This is working as-is and I found that the obfuscation processor is not obfuscating 100% of fields. With this being less important than unblocking you to try synthetic data, I thought we should merge this. #192

Copy link
Contributor

@lquerel lquerel left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

2 minor comments.
Otherwise, LGTM.

collector/examples/synthesize/README.md Outdated Show resolved Hide resolved
cpy.PutStr(o.encryptString(k), o.encryptString(value.Str()))
default:
// TODO: This does not cover all string values, needs
// to be updated for StringSlice and KVList types at
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes slices in general as it's possible to have slice of kvlist containing string, or more complex constructs.

Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Yes, I'd like to delay until @moh-osman3 returns.

#192

Other fields include metric description and unit, and I suspect there may be more.

processors:
batch:
obfuscation:
encrypt_key: "some-32-byte-long-key-to-be-safe"
Copy link
Collaborator Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

As a possible TODO --
@moh-osman3 I had thrown away the key when obfuscating, to be sure it's not stored and used to de-obfuscate to be even-safer than safe, I guess.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants