Add asyncpg instrumentation in Elastic APM #889

odimko · 2020-07-17T13:08:58Z

What does this pull request do?

Adds support for the asyncpg database interface library for Python and PostgreSQL by implementing corresponding instrumentation in the APM agent.

Related issues

#755

Closes issues

#755

cla-checker-service · 2020-07-17T13:09:01Z

💚 CLA has been signed

apmmachine · 2020-07-17T13:12:28Z

💚 Build Succeeded

Expand to view the summary

Build stats

Build Cause: [Started by user Benjamin Wohlwend, Replayed #38]
Start Time: 2020-08-27T12:29:11.542+0000
Duration: 29 min 1 sec

Test stats 🧪

Test	Results
Failed	0
Passed	10927
Skipped	8506
Total	19433

Steps errors

Expand to view the steps failures

Name: Shell Script
- Description: ./tests/scripts/docker/run_tests.sh pypy-3 django-2.0
- Duration: 3 min 56 sec
- Start Time: 2020-08-27T12:36:06.636+0000
- log

basepi · 2020-07-17T19:40:18Z

Heya @odimko! Thanks for this!

Please sign the CLA when you get a chance. Do you think you'll be able to add some tests here as well?

odimko · 2020-07-21T06:20:56Z

hey there @basepi! Thanks for checking out my PR. it's not ready yet, so more commits will come (I need to update the tests).

basepi · 2020-07-21T14:16:39Z

Nope, shows signed. Sometimes it just lags a bit. Thanks, this looks awesome!

…hon into add_support_asyncpg

odimko · 2020-07-23T10:53:39Z

hey @basepi! Could you let me know how I can run CI tests as I am currently getting this error:
githubPrCheckApproved: The PR is not allowed to run in the CI yet. (Only users with write permissions can do so.), thus, I can't check my tests.

odimko · 2020-07-30T19:15:56Z

hi @basepi :) I was wondering if you have had some time to have a look at my comment above :)

sethmlarson · 2020-07-30T19:23:00Z

@odimko FYI @basepi has been on vacation for the past week and will review when he returns 🛬

odimko · 2020-07-30T19:57:31Z

thanks for letting me know @sethmlarson 🥇

beniwohli

@odimko this is great, thanks! And sorry for the long turnaround on the review, both @basepi and me were on PTO

elasticapm/instrumentation/register.py

elasticapm/instrumentation/packages/asyncio/asyncpg.py

.ci/.jenkins_framework.yml

odimko · 2020-08-03T13:22:35Z

@odimko this is great, thanks! And sorry for the long turnaround on the review, both @basepi and me were on PTO

hey @beniwohli and @basepi! I have implemented all the requested changes. Please have a look at the PR if it looks good now and let me know if I should improve anything else :)

beniwohli · 2020-08-03T14:21:48Z

@odimko great work on the updates. The tests currently fails because asyncpg isn't able to parse the DSN. Turns out that for asyncpg, the DSN has a different format than psycopg2, using a URL format:

https://magicstack.github.io/asyncpg/current/api/index.html#asyncpg.connection.connect

You should be able to adapt the dsn() function in your asyncpg tests easily.

…hon into add_support_asyncpg

odimko · 2020-08-25T14:52:55Z

@beniwohli and @basepi, hey there :)
please have a look at the latest changes to the PR, I've switched to use the yield approach as suggested (since we are moving towards Python 3.5 deprecation). Is there anything else left to do?

all tests pass successfully except for linting. any suggestions on how I can run it? otherwise, I would appreciate if someone could run it so that the last flailing check will pass :)

beniwohli

This looks great! Thanks a lot for your contribution and sorry about the sometimes slow answer times!

docs/supported-technologies.asciidoc

odimko · 2020-08-27T21:40:12Z

This looks great! Thanks a lot for your contribution and sorry about the sometimes slow answer times!

No problem at all! Thanks for supporting with it 🥇

@beniwohli, may I ask you when it will be released?

basepi · 2020-09-01T16:02:26Z

We don't currently have a release date planned, but I expect it will be Soon (tm) since we have a couple of useful features that are currently unreleased (and since we no longer need to include the removal of py3.5 in the release).

odimko · 2020-09-01T19:49:34Z

We don't currently have a release date planned, but I expect it will be Soon (tm) since we have a couple of useful features that are currently unreleased (and since we no longer need to include the removal of py3.5 in the release).

got it, thanks for explaining!

odimko · 2020-09-21T08:45:06Z

Good morning @basepi and @beniwohli! just checking in to check what is your plan regarding releasing the next version of the APM package. could you please let me know?

basepi · 2020-09-22T16:19:59Z

Very soon. We'd really like to get #927 in, but it needs some test fixes.

odimko · 2020-09-22T20:36:23Z

perfect, thanks for the update!

basepi · 2020-09-23T18:10:29Z

It's out! https://github.com/elastic/apm-agent-python/releases/tag/v5.9.0

odimko · 2020-09-24T08:15:47Z

It's out! https://github.com/elastic/apm-agent-python/releases/tag/v5.9.0

awesome! thanks a lot!

Issue elastic#889 added instrumentation for asyncpg, though only `execute` and `executemany` were instrumented. This extends the instrumentation to include `fetch`, `fetchval` and `fetchrow` methods.

Issue #889 added instrumentation for asyncpg, though only `execute` and `executemany` were instrumented. This extends the instrumentation to include `fetch`, `fetchval` and `fetchrow` methods.

* Add a new file * Add AsyncPGInstrumentation to register.py * Fix a typo in register.py * Use correct function for registering * Update the docs + add a template for the tests * Update variable names in the tests * Add asyncpg.sh for the tests * Remove unused docstrings * Improve tests * Remove callproc test * PR suggestions * FIx tests: use URI instead of explicit parameters * FIx tests: implement further PR suggestions * FIx tests: implement more PR suggestions * Use Connection.execute and Connection.executemany instead of Connection._do_execute * Remove unnecessary requirements * Fix a type in the tests * Use yield and proper awaiting in the tests * Use yield and proper awaiting in the tests + fix a typo * Fix tests * Additional fixes to tests * fix typo Co-authored-by: Colton Myers <colton.myers@gmail.com> Co-authored-by: Benjamin Wohlwend <bw@piquadrat.ch>

Issue elastic#889 added instrumentation for asyncpg, though only `execute` and `executemany` were instrumented. This extends the instrumentation to include `fetch`, `fetchval` and `fetchrow` methods.

* Add a new file * Add AsyncPGInstrumentation to register.py * Fix a typo in register.py * Use correct function for registering * Update the docs + add a template for the tests * Update variable names in the tests * Add asyncpg.sh for the tests * Remove unused docstrings * Improve tests * Remove callproc test * PR suggestions * FIx tests: use URI instead of explicit parameters * FIx tests: implement further PR suggestions * FIx tests: implement more PR suggestions * Use Connection.execute and Connection.executemany instead of Connection._do_execute * Remove unnecessary requirements * Fix a type in the tests * Use yield and proper awaiting in the tests * Use yield and proper awaiting in the tests + fix a typo * Fix tests * Additional fixes to tests * fix typo Co-authored-by: Colton Myers <colton.myers@gmail.com> Co-authored-by: Benjamin Wohlwend <bw@piquadrat.ch>

Issue elastic#889 added instrumentation for asyncpg, though only `execute` and `executemany` were instrumented. This extends the instrumentation to include `fetch`, `fetchval` and `fetchrow` methods.

* Add a new file * Add AsyncPGInstrumentation to register.py * Fix a typo in register.py * Use correct function for registering * Update the docs + add a template for the tests * Update variable names in the tests * Add asyncpg.sh for the tests * Remove unused docstrings * Improve tests * Remove callproc test * PR suggestions * FIx tests: use URI instead of explicit parameters * FIx tests: implement further PR suggestions * FIx tests: implement more PR suggestions * Use Connection.execute and Connection.executemany instead of Connection._do_execute * Remove unnecessary requirements * Fix a type in the tests * Use yield and proper awaiting in the tests * Use yield and proper awaiting in the tests + fix a typo * Fix tests * Additional fixes to tests * fix typo Co-authored-by: Colton Myers <colton.myers@gmail.com> Co-authored-by: Benjamin Wohlwend <bw@piquadrat.ch>

Issue elastic#889 added instrumentation for asyncpg, though only `execute` and `executemany` were instrumented. This extends the instrumentation to include `fetch`, `fetchval` and `fetchrow` methods.

Add a new file

f2bc6a7

odimko changed the title ~~Add a new file~~ Add asyncpg instrumentation in Elastic APM Jul 17, 2020

odimko added 2 commits July 20, 2020 16:26

Add AsyncPGInstrumentation to register.py

f3fdf3c

Fix a typo in register.py

a783ee4

odimko added 2 commits July 21, 2020 15:40

Use correct function for registering

5b6eb24

Update the docs + add a template for the tests

536f0c2

odimko marked this pull request as ready for review July 21, 2020 14:15

odimko and others added 7 commits July 21, 2020 17:23

Update variable names in the tests

708ae00

Add asyncpg.sh for the tests

671612c

Remove unused docstrings

a48c00c

Improve tests

6c0571c

Merge branch 'master' into add_support_asyncpg

1e5f45a

Remove callproc test

9dcd53c

Merge branch 'add_support_asyncpg' of github.com:odimko/apm-agent-pyt…

62d5aac

…hon into add_support_asyncpg

beniwohli suggested changes Aug 3, 2020

View reviewed changes

elasticapm/instrumentation/register.py Outdated Show resolved Hide resolved

elasticapm/instrumentation/packages/asyncio/asyncpg.py Outdated Show resolved Hide resolved

.ci/.jenkins_framework.yml Show resolved Hide resolved

odimko and others added 2 commits August 3, 2020 14:55

Merge branch 'master' into add_support_asyncpg

1932a10

PR suggestions

1eefe82

beniwohli approved these changes Aug 3, 2020

View reviewed changes

FIx tests: use URI instead of explicit parameters

c6b71e9

odimko added 3 commits August 25, 2020 17:40

Use yield and proper awaiting in the tests

dbdae63

Merge branch 'add_support_asyncpg' of github.com:odimko/apm-agent-pyt…

fc2c113

…hon into add_support_asyncpg

Use yield and proper awaiting in the tests + fix a typo

264a7c8

odimko and others added 3 commits August 25, 2020 18:13

Fix tests

ab56993

Additional fixes to tests

662cfe7

Merge branch 'master' into add_support_asyncpg

8b50ca6

basepi requested a review from beniwohli August 26, 2020 19:33

beniwohli approved these changes Aug 27, 2020

View reviewed changes

docs/supported-technologies.asciidoc Outdated Show resolved Hide resolved

fix typo

8593f09

beniwohli merged commit 4bef64b into elastic:master Aug 27, 2020

odimko deleted the add_support_asyncpg branch August 27, 2020 21:39

basepi mentioned this pull request Sep 2, 2020

[Feature] Add instrumentation for asyncpg #755

Closed

stj mentioned this pull request Oct 5, 2020

Extend asyncpg instrumentation in Elastic APM #939

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add asyncpg instrumentation in Elastic APM #889

Add asyncpg instrumentation in Elastic APM #889

odimko commented Jul 17, 2020 •

edited

Loading

cla-checker-service bot commented Jul 17, 2020 •

edited

Loading

apmmachine commented Jul 17, 2020 •

edited

Loading

Build stats

Test stats 🧪

basepi commented Jul 17, 2020

odimko commented Jul 21, 2020 •

edited

Loading

basepi commented Jul 21, 2020

odimko commented Jul 23, 2020

odimko commented Jul 30, 2020

sethmlarson commented Jul 30, 2020 •

edited

Loading

odimko commented Jul 30, 2020

beniwohli left a comment

odimko commented Aug 3, 2020

beniwohli commented Aug 3, 2020

odimko commented Aug 25, 2020 •

edited

Loading

beniwohli left a comment

odimko commented Aug 27, 2020 •

edited

Loading

basepi commented Sep 1, 2020

odimko commented Sep 1, 2020

odimko commented Sep 21, 2020

basepi commented Sep 22, 2020

odimko commented Sep 22, 2020

basepi commented Sep 23, 2020

odimko commented Sep 24, 2020

Add asyncpg instrumentation in Elastic APM #889

Add asyncpg instrumentation in Elastic APM #889

Conversation

odimko commented Jul 17, 2020 • edited Loading

What does this pull request do?

Related issues

Closes issues

cla-checker-service bot commented Jul 17, 2020 • edited Loading

apmmachine commented Jul 17, 2020 • edited Loading

💚 Build Succeeded

Build stats

Test stats 🧪

Steps errors

basepi commented Jul 17, 2020

odimko commented Jul 21, 2020 • edited Loading

basepi commented Jul 21, 2020

odimko commented Jul 23, 2020

odimko commented Jul 30, 2020

sethmlarson commented Jul 30, 2020 • edited Loading

odimko commented Jul 30, 2020

beniwohli left a comment

Choose a reason for hiding this comment

odimko commented Aug 3, 2020

beniwohli commented Aug 3, 2020

odimko commented Aug 25, 2020 • edited Loading

beniwohli left a comment

Choose a reason for hiding this comment

odimko commented Aug 27, 2020 • edited Loading

basepi commented Sep 1, 2020

odimko commented Sep 1, 2020

odimko commented Sep 21, 2020

basepi commented Sep 22, 2020

odimko commented Sep 22, 2020

basepi commented Sep 23, 2020

odimko commented Sep 24, 2020

odimko commented Jul 17, 2020 •

edited

Loading

cla-checker-service bot commented Jul 17, 2020 •

edited

Loading

apmmachine commented Jul 17, 2020 •

edited

Loading

odimko commented Jul 21, 2020 •

edited

Loading

sethmlarson commented Jul 30, 2020 •

edited

Loading

odimko commented Aug 25, 2020 •

edited

Loading

odimko commented Aug 27, 2020 •

edited

Loading