Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Truncate long sql queries to prevent oversized events to APM server #842

Merged
merged 9 commits into from
Jun 4, 2020
Merged

Truncate long sql queries to prevent oversized events to APM server #842

merged 9 commits into from
Jun 4, 2020

Conversation

basepi
Copy link
Contributor

@basepi basepi commented May 29, 2020

Fixes #827

@basepi
Copy link
Contributor Author

basepi commented May 29, 2020

@beniwohli I'm unsure of how to test this. We don't test the CursorProxy class directly, but rather rely on tests against the instrumentations which inherit from it. Should I just try to write a test against sqlite and any one of the other instrumentations to cover both code paths? I also need to figure out a good extremely-long query to use in these tests...

Any thoughts?

@apmmachine
Copy link
Contributor

apmmachine commented May 29, 2020 •
edited
Loading

💚 Build Succeeded

Pipeline View Test View Changes Artifacts preview

Expand to view the summary

Build stats

  • Build Cause: [Pull request #842 updated]

  • Start Time: 2020-06-04T18:55:21.010+0000

  • Duration: 26 min 26 sec

Test stats 🧪

Test Results
Failed 0
Passed 9233
Skipped 6796
Total 16029

beniwohli
beniwohli reviewed Jun 2, 2020
View reviewed changes
Copy link
Contributor

@beniwohli beniwohli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Regarding testing, you could use mock to change the value of KEYWORD_MAX_LENGTH to a very low value, so we don't need a super long query. Or just do a "SELECT xyz from test_table WHERE xyz = %s" % ("x" * 10010). I think testing with SQLite only is fine.

elasticapm/instrumentation/packages/sqlite.py Outdated Show resolved Hide resolved
@basepi basepi marked this pull request as ready for review June 2, 2020 21:41
@basepi
Copy link
Contributor Author

basepi commented Jun 2, 2020

Excellent advice, thanks @beniwohli. This is ready for review.

@basepi basepi requested a review from beniwohli June 3, 2020 22:27
beniwohli
beniwohli approved these changes Jun 4, 2020
View reviewed changes
Copy link
Contributor

@beniwohli beniwohli left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM, nice work!

@basepi basepi merged commit 27bdc2e into elastic:master Jun 4, 2020
romulorosa pushed a commit to romulorosa/apm-agent-python that referenced this pull request Oct 15, 2020
@basepi @romulorosa
Truncate long sql queries to prevent oversized events to APM server (e…
a7b46c5 
…lastic#842)

* Truncate long sql queries to prevent oversized events to APM server

* Truncate sql to 10000 characters instead of keyword length

* Add a test

* Add to changelog

* Add encoding for sqlite_tests.py

* Use existing `shorten()` instead of `truncate()`
romulorosa pushed a commit to romulorosa/apm-agent-python that referenced this pull request Oct 15, 2020
@basepi @romulorosa
Truncate long sql queries to prevent oversized events to APM server (e…
e7ed468 
…lastic#842)

* Truncate long sql queries to prevent oversized events to APM server

* Truncate sql to 10000 characters instead of keyword length

* Add a test

* Add to changelog

* Add encoding for sqlite_tests.py

* Use existing `shorten()` instead of `truncate()`
beniwohli pushed a commit to beniwohli/apm-agent-python that referenced this pull request Sep 14, 2021
@basepi @beniwohli
Truncate long sql queries to prevent oversized events to APM server (e…
8b6811a 
…lastic#842)

* Truncate long sql queries to prevent oversized events to APM server

* Truncate sql to 10000 characters instead of keyword length

* Add a test

* Add to changelog

* Add encoding for sqlite_tests.py

* Use existing `shorten()` instead of `truncate()`
stj added a commit to stj/apm-agent-python that referenced this pull request Nov 24, 2021
@stj
Shorten long SQL query to prevent oversized event
a640893 
APM server allows events with a max size of 10000 characters. Shorten
queries traced in asyncpg to max allowed.

See: elastic#842
@stj stj mentioned this pull request Nov 24, 2021
Merged
beniwohli added a commit that referenced this pull request Nov 29, 2021
@stj @beniwohli
Shorten long SQL query to prevent oversized event (#1416)
6a4656e 
* Shorten long SQL query to prevent oversized event

APM server allows events with a max size of 10000 characters. Shorten
queries traced in asyncpg to max allowed.

See: #842

* Fix copied SQL

* fix test

* instrument for test

* move changelog entry to "Unreleased" and fix PR number

Co-authored-by: Benjamin Wohlwend <beni@elastic.co>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@beniwohli beniwohli beniwohli approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

APM agent crashes if the SQL query is too large
3 participants
@basepi @apmmachine @beniwohli