Add query logging callbacks and context manager #1043
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
elprans
reviewed
Jul 6, 2023
asyncpg/connection.py
Outdated
| record = LoggedQuery(query, args, timeout, elapsed) | ||
| for cb in self._query_loggers: | ||
| if cb.is_async: | ||
| self._loop.create_task(cb.cb(con_ref, record)) |
There was a problem hiding this comment.
Passing a con_ref is probably unnecessary?
There was a problem hiding this comment.
I likely wouldn't use it, so happy to remove it, but I put it there so you could potentially log queries by host.
There was a problem hiding this comment.
A concern here is potentially retaining references to free-d connections. Other callbacks take it, of course, but that's an API decision I've come to regret. Perhaps we can pass connection's addr and params instead?
elprans
reviewed
Jul 6, 2023
asyncpg/connection.py
Outdated
| @@ -314,7 +339,11 @@ async def execute(self, query: str, *args, timeout: float=None) -> str: | |||
| self._check_open() | |||
|
|
|||
| if not args: | |||
| return await self._protocol.query(query, timeout) | |||
| start = time.monotonic() | |||
There was a problem hiding this comment.
I think a context manager would probably be nicer/reduce duplication.
There was a problem hiding this comment.
Agreed, change incoming.
elprans
reviewed
Jul 7, 2023
asyncpg/connection.py
Outdated
Comment on lines
1751
to
1759
| with utils.timer() as t: | ||
| result, stmt = await self._do_execute( | ||
| query, | ||
| executor, | ||
| timeout, | ||
| record_class=record_class, | ||
| ignore_custom_codec=ignore_custom_codec, | ||
| ) | ||
| self._log_query(query, args, timeout, t.elapsed) |
There was a problem hiding this comment.
You could go further and pull self._log_query into the context manager and that way you'll be able to log query errors too, e.g.
Suggested change
| with utils.timer() as t: | |
| result, stmt = await self._do_execute( | |
| query, | |
| executor, | |
| timeout, | |
| record_class=record_class, | |
| ignore_custom_codec=ignore_custom_codec, | |
| ) | |
| self._log_query(query, args, timeout, t.elapsed) | |
| with self._time_and_log(query, args, timeout): | |
| result, stmt = await self._do_execute( | |
| query, | |
| executor, | |
| timeout, | |
| record_class=record_class, | |
| ignore_custom_codec=ignore_custom_codec, | |
| ) |
There was a problem hiding this comment.
Great idea, I feel silly for not considering error logging as part of this.
elprans
reviewed
Jul 7, 2023
asyncpg/connection.py
Outdated
| @@ -2327,6 +2414,12 @@ class _ConnectionProxy: | |||
| __slots__ = () | |||
|
|
|||
|
|
|||
| LoggedQuery = collections.namedtuple( | |||
| 'LoggedQuery', | |||
| ['query', 'args', 'timeout', 'elapsed', 'exception', 'addr', 'params']) | |||
There was a problem hiding this comment.
Suggested change
| ['query', 'args', 'timeout', 'elapsed', 'exception', 'addr', 'params']) | |
| ['query', 'args', 'timeout', 'elapsed', 'exception', 'conn_addr', 'conn_params']) |
elprans
approved these changes
Oct 9, 2023
This was referenced Oct 9, 2023
Closed
elprans
added a commit
that referenced
this pull request
Nov 5, 2023
Minor fixes and improvements. Improvements ============ * Python 3.12 and PostgreSQL 16 support (#1084) (by @elprans in deea86c) * Add support for tuple-format custom codecs on composite types (#1061) (by @elprans in 922fcd1) * Support `target_session_attrs` in URL format, add tests (#1073) (by @elprans in 7cb4e70) * Infinity numeric support (#1067) (by @krokoziabla in 0c3bf60 for #1020) * Add support for the `WHERE` clause in `copy_to` methods (#941) (by @kaylynn234 in b7ffab6) * Add query logging callbacks and context manager (#1043) (by @dcwatson in b2697ff) Fixes ===== * When prepared statements are disabled, avoid relying on them harder (#1065) (by @elprans in cbf64e1) * Handle environments with HOME set to a not-a-directory (#1063) (by @elprans in af922bc) * Fix handling of non-ASCII passwords (#1062) (by @elprans in 89d5bd0) * Disable JIT while doing type introspection (#1082) (by @elprans in f21ebf6) * Remove connection parameter caching in `Pool` (#1053) (by @ermakov-oleg in 4ddb039) * Switch to Python 3.12-style `wait_for` (#1086) (by @elprans in 4bdd8a7) * Update automatic PostGIS type conversion for Shapely 2.0 (#1085) (by @ChimneySwift in 8b45beb) * Use the `timeout` context manager in the connection path (#1087) (by @elprans in 313b2b2) * Small fix for documentation on using SSL in Connection (#995) (by @ScottFred in ccc7baf) * Use cleanup_ctx in pool usage doc (#878) (by @ir4y in 70c8bd8) * Close cursor portals once the iterator is exhausted (#1088) (by @elprans in ca9f03b) * Cut BaseProtocol circular reference on close. (#1049) (by @pteromys in 93a6f79) * Allow passing hosts as tuples to `connect()` (in addition to lists) (#1021) (by @lezram in d7faaff) Other ===== * Drop support for Python 3.7 (#1064) (by @bryanforbes in 87ab143)
Merged
elprans
added a commit
that referenced
this pull request
Nov 5, 2023
Minor fixes and improvements. Improvements ============ * Python 3.12 and PostgreSQL 16 support (#1084) (by @elprans in deea86c) * Add support for tuple-format custom codecs on composite types (#1061) (by @elprans in 922fcd1) * Support `target_session_attrs` in URL format, add tests (#1073) (by @elprans in 7cb4e70) * Infinity numeric support (#1067) (by @krokoziabla in 0c3bf60 for #1020) * Add support for the `WHERE` clause in `copy_to` methods (#941) (by @kaylynn234 in b7ffab6) * Add query logging callbacks and context manager (#1043) (by @dcwatson in b2697ff) Fixes ===== * When prepared statements are disabled, avoid relying on them harder (#1065) (by @elprans in cbf64e1) * Handle environments with HOME set to a not-a-directory (#1063) (by @elprans in af922bc) * Fix handling of non-ASCII passwords (#1062) (by @elprans in 89d5bd0) * Disable JIT while doing type introspection (#1082) (by @elprans in f21ebf6) * Remove connection parameter caching in `Pool` (#1053) (by @ermakov-oleg in 4ddb039) * Switch to Python 3.12-style `wait_for` (#1086) (by @elprans in 4bdd8a7) * Update automatic PostGIS type conversion for Shapely 2.0 (#1085) (by @ChimneySwift in 8b45beb) * Use the `timeout` context manager in the connection path (#1087) (by @elprans in 313b2b2) * Small fix for documentation on using SSL in Connection (#995) (by @ScottFred in ccc7baf) * Use cleanup_ctx in pool usage doc (#878) (by @ir4y in 70c8bd8) * Close cursor portals once the iterator is exhausted (#1088) (by @elprans in ca9f03b) * Cut BaseProtocol circular reference on close. (#1049) (by @pteromys in 93a6f79) * Allow passing hosts as tuples to `connect()` (in addition to lists) (#1021) (by @lezram in d7faaff) Other ===== * Drop support for Python 3.7 (#1064) (by @bryanforbes in 87ab143)
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Another take on query logging (#629), with a similar approach to existing listeners. Consider this a WIP - I'm happy to write documentation and more tests if this approach is preferable to #630.