This repository has been archived by the owner on Apr 26, 2024. It is now read-only.
Don't hammer the database for destination retry timings every ~5mins #10036
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
They were both being run on workers, so the seperation was pointless.
erikjohnston
force-pushed
the
erikj/destination_retry_schemes
branch
from
a641edb
to
1b1b595
Compare
clokep
approved these changes
May 21, 2021
synapse/util/retryutils.py
Outdated
Comment on lines
86
to
89
| retry_last_ts, retry_interval = ( | ||
| retry_timings["retry_last_ts"], | ||
| retry_timings["retry_interval"], | ||
| retry_timings.retry_last_ts, | ||
| retry_timings.retry_interval, | ||
| ) |
There was a problem hiding this comment.
I know this code was here before, but it seems simpler to write it as:
retry_last_ts = retry_timings.retry_last_ts
retry_interval = retry_timings.retry_intervalI guess this is trying to match the code from above, but it seems more confusing to me.
| @@ -145,7 +147,11 @@ async def set_received_txn_response( | |||
| desc="set_received_txn_response", | |||
| ) | |||
|
|
|||
| async def get_destination_retry_timings(self, destination): | |||
| @cached(max_entries=10000) | |||
There was a problem hiding this comment.
10k destinations seems high, but maybe it isn't? (Did you just grab a random number or did you check the previous size of the cache?)
There was a problem hiding this comment.
The flip side was that 1000 felt a bit small. I mostly just pulled it out of thin air, and it matches some other "small" caches.
There was a problem hiding this comment.
Ah, yeah I guess 1000 doesn't feel big enough. 👍 Was mostly curious where it came from. Thin air seems OK to me.
Co-authored-by: Patrick Cloke <clokep@users.noreply.github.com>
netbsd-srcmastr
pushed a commit
to NetBSD/pkgsrc
that referenced
this pull request
Jun 5, 2021
Synapse 1.35.1 (2021-06-03) =========================== Bugfixes -------- - Fix a bug introduced in v1.35.0 where invite-only rooms would be shown to all users in a space, regardless of if the user had access to it. ([\#10109](matrix-org/synapse#10109)) Synapse 1.35.0 (2021-06-01) =========================== Note that [the tag](https://github.com/matrix-org/synapse/releases/tag/v1.35.0rc3) and [docker images](https://hub.docker.com/layers/matrixdotorg/synapse/v1.35.0rc3/images/sha256-34ccc87bd99a17e2cbc0902e678b5937d16bdc1991ead097eee6096481ecf2c4?context=explore) for `v1.35.0rc3` were incorrectly built. If you are experiencing issues with either, it is recommended to upgrade to the equivalent tag or docker image for the `v1.35.0` release. Deprecations and Removals ------------------------- - The core Synapse development team plan to drop support for the [unstable API of MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option, in August 2021. Client authors should ensure that their clients are updated to use the stable API (which has been supported since Synapse 1.30) well before that time, to give their users time to upgrade. ([\#10101](matrix-org/synapse#10101)) Bugfixes -------- - Fixed a bug causing replication requests to fail when receiving a lot of events via federation. Introduced in v1.33.0. ([\#10082](matrix-org/synapse#10082)) - Fix HTTP response size limit to allow joining very large rooms over federation. Introduced in v1.33.0. ([\#10093](matrix-org/synapse#10093)) Internal Changes ---------------- - Log method and path when dropping request due to size limit. ([\#10091](matrix-org/synapse#10091)) Synapse 1.35.0rc2 (2021-05-27) ============================== Bugfixes -------- - Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](matrix-org/synapse#10079)) Synapse 1.35.0rc1 (2021-05-25) ============================== Features -------- - Add experimental support to allow a user who could join a restricted room to view it in the spaces summary. ([\#9922](matrix-org/synapse#9922), [\#10007](matrix-org/synapse#10007), [\#10038](matrix-org/synapse#10038)) - Reduce memory usage when joining very large rooms over federation. ([\#9958](matrix-org/synapse#9958)) - Add a configuration option which allows enabling opentracing by user id. ([\#9978](matrix-org/synapse#9978)) - Enable experimental support for [MSC2946](matrix-org/matrix-spec-proposals#2946) (spaces summary API) and [MSC3083](matrix-org/matrix-spec-proposals#3083) (restricted join rules) by default. ([\#10011](matrix-org/synapse#10011)) Bugfixes -------- - Fix a bug introduced in v1.26.0 which meant that `synapse_port_db` would not correctly initialise some postgres sequences, requiring manual updates afterwards. ([\#9991](matrix-org/synapse#9991)) - Fix `synctl`'s `--no-daemonize` parameter to work correctly with worker processes. ([\#9995](matrix-org/synapse#9995)) - Fix a validation bug introduced in v1.34.0 in the ordering of spaces in the space summary API. ([\#10002](matrix-org/synapse#10002)) - Fixed deletion of new presence stream states from database. ([\#10014](matrix-org/synapse#10014), [\#10033](matrix-org/synapse#10033)) - Fixed a bug with very high resolution image uploads throwing internal server errors. ([\#10029](matrix-org/synapse#10029)) Updates to the Docker image --------------------------- - Fix bug introduced in Synapse 1.33.0 which caused a `Permission denied: '/homeserver.log'` error when starting Synapse with the generated log configuration. Contributed by Sergio Miguéns Iglesias. ([\#10045](matrix-org/synapse#10045)) Improved Documentation ---------------------- - Add hardened systemd files as proposed in [#9760](matrix-org/synapse#9760) and added them to `contrib/`. Change the docs to reflect the presence of these files. ([\#9803](matrix-org/synapse#9803)) - Clarify documentation around SSO mapping providers generating unique IDs and localparts. ([\#9980](matrix-org/synapse#9980)) - Updates to the PostgreSQL documentation (`postgres.md`). ([\#9988](matrix-org/synapse#9988), [\#9989](matrix-org/synapse#9989)) - Fix broken link in user directory documentation. Contributed by @junquera. ([\#10016](matrix-org/synapse#10016)) - Add missing room state entry to the table of contents of room admin API. ([\#10043](matrix-org/synapse#10043)) Deprecations and Removals ------------------------- - Removed support for the deprecated `tls_fingerprints` configuration setting. Contributed by Jerin J Titus. ([\#9280](matrix-org/synapse#9280)) Internal Changes ---------------- - Allow sending full presence to users via workers other than the one that called `ModuleApi.send_local_online_presence_to`. ([\#9823](matrix-org/synapse#9823)) - Update comments in the space summary handler. ([\#9974](matrix-org/synapse#9974)) - Minor enhancements to the `@cachedList` descriptor. ([\#9975](matrix-org/synapse#9975)) - Split multipart email sending into a dedicated handler. ([\#9977](matrix-org/synapse#9977)) - Run `black` on files in the `scripts` directory. ([\#9981](matrix-org/synapse#9981)) - Add missing type hints to `synapse.util` module. ([\#9982](matrix-org/synapse#9982)) - Simplify a few helper functions. ([\#9984](matrix-org/synapse#9984), [\#9985](matrix-org/synapse#9985), [\#9986](matrix-org/synapse#9986)) - Remove unnecessary property from SQLBaseStore. ([\#9987](matrix-org/synapse#9987)) - Remove `keylen` param on `LruCache`. ([\#9993](matrix-org/synapse#9993)) - Update the Grafana dashboard in `contrib/`. ([\#10001](matrix-org/synapse#10001)) - Add a batching queue implementation. ([\#10017](matrix-org/synapse#10017)) - Reduce memory usage when verifying signatures on large numbers of events at once. ([\#10018](matrix-org/synapse#10018)) - Properly invalidate caches for destination retry timings every (instead of expiring entries every 5 minutes). ([\#10036](matrix-org/synapse#10036)) - Fix running complement tests with Synapse workers. ([\#10039](matrix-org/synapse#10039)) - Fix typo in `get_state_ids_for_event` docstring where the return type was incorrect. ([\#10050](matrix-org/synapse#10050))
babolivier
added a commit
to matrix-org/synapse-dinsic
that referenced
this pull request
Sep 1, 2021
Synapse 1.35.0 (2021-06-01) =========================== Note that [the tag](https://github.com/matrix-org/synapse/releases/tag/v1.35.0rc3) and [docker images](https://hub.docker.com/layers/matrixdotorg/synapse/v1.35.0rc3/images/sha256-34ccc87bd99a17e2cbc0902e678b5937d16bdc1991ead097eee6096481ecf2c4?context=explore) for `v1.35.0rc3` were incorrectly built. If you are experiencing issues with either, it is recommended to upgrade to the equivalent tag or docker image for the `v1.35.0` release. Deprecations and Removals ------------------------- - The core Synapse development team plan to drop support for the [unstable API of MSC2858](https://github.com/matrix-org/matrix-doc/blob/master/proposals/2858-Multiple-SSO-Identity-Providers.md#unstable-prefix), including the undocumented `experimental.msc2858_enabled` config option, in August 2021. Client authors should ensure that their clients are updated to use the stable API (which has been supported since Synapse 1.30) well before that time, to give their users time to upgrade. ([\#10101](matrix-org/synapse#10101)) Bugfixes -------- - Fixed a bug causing replication requests to fail when receiving a lot of events via federation. Introduced in v1.33.0. ([\#10082](matrix-org/synapse#10082)) - Fix HTTP response size limit to allow joining very large rooms over federation. Introduced in v1.33.0. ([\#10093](matrix-org/synapse#10093)) Internal Changes ---------------- - Log method and path when dropping request due to size limit. ([\#10091](matrix-org/synapse#10091)) Synapse 1.35.0rc2 (2021-05-27) ============================== Bugfixes -------- - Fix a bug introduced in v1.35.0rc1 when calling the spaces summary API via a GET request. ([\#10079](matrix-org/synapse#10079)) Synapse 1.35.0rc1 (2021-05-25) ============================== Features -------- - Add experimental support to allow a user who could join a restricted room to view it in the spaces summary. ([\#9922](matrix-org/synapse#9922), [\#10007](matrix-org/synapse#10007), [\#10038](matrix-org/synapse#10038)) - Reduce memory usage when joining very large rooms over federation. ([\#9958](matrix-org/synapse#9958)) - Add a configuration option which allows enabling opentracing by user id. ([\#9978](matrix-org/synapse#9978)) - Enable experimental support for [MSC2946](matrix-org/matrix-spec-proposals#2946) (spaces summary API) and [MSC3083](matrix-org/matrix-spec-proposals#3083) (restricted join rules) by default. ([\#10011](matrix-org/synapse#10011)) Bugfixes -------- - Fix a bug introduced in v1.26.0 which meant that `synapse_port_db` would not correctly initialise some postgres sequences, requiring manual updates afterwards. ([\#9991](matrix-org/synapse#9991)) - Fix `synctl`'s `--no-daemonize` parameter to work correctly with worker processes. ([\#9995](matrix-org/synapse#9995)) - Fix a validation bug introduced in v1.34.0 in the ordering of spaces in the space summary API. ([\#10002](matrix-org/synapse#10002)) - Fixed deletion of new presence stream states from database. ([\#10014](matrix-org/synapse#10014), [\#10033](matrix-org/synapse#10033)) - Fixed a bug with very high resolution image uploads throwing internal server errors. ([\#10029](matrix-org/synapse#10029)) Updates to the Docker image --------------------------- - Fix bug introduced in Synapse 1.33.0 which caused a `Permission denied: '/homeserver.log'` error when starting Synapse with the generated log configuration. Contributed by Sergio Miguéns Iglesias. ([\#10045](matrix-org/synapse#10045)) Improved Documentation ---------------------- - Add hardened systemd files as proposed in [#9760](matrix-org/synapse#9760) and added them to `contrib/`. Change the docs to reflect the presence of these files. ([\#9803](matrix-org/synapse#9803)) - Clarify documentation around SSO mapping providers generating unique IDs and localparts. ([\#9980](matrix-org/synapse#9980)) - Updates to the PostgreSQL documentation (`postgres.md`). ([\#9988](matrix-org/synapse#9988), [\#9989](matrix-org/synapse#9989)) - Fix broken link in user directory documentation. Contributed by @junquera. ([\#10016](matrix-org/synapse#10016)) - Add missing room state entry to the table of contents of room admin API. ([\#10043](matrix-org/synapse#10043)) Deprecations and Removals ------------------------- - Removed support for the deprecated `tls_fingerprints` configuration setting. Contributed by Jerin J Titus. ([\#9280](matrix-org/synapse#9280)) Internal Changes ---------------- - Allow sending full presence to users via workers other than the one that called `ModuleApi.send_local_online_presence_to`. ([\#9823](matrix-org/synapse#9823)) - Update comments in the space summary handler. ([\#9974](matrix-org/synapse#9974)) - Minor enhancements to the `@cachedList` descriptor. ([\#9975](matrix-org/synapse#9975)) - Split multipart email sending into a dedicated handler. ([\#9977](matrix-org/synapse#9977)) - Run `black` on files in the `scripts` directory. ([\#9981](matrix-org/synapse#9981)) - Add missing type hints to `synapse.util` module. ([\#9982](matrix-org/synapse#9982)) - Simplify a few helper functions. ([\#9984](matrix-org/synapse#9984), [\#9985](matrix-org/synapse#9985), [\#9986](matrix-org/synapse#9986)) - Remove unnecessary property from SQLBaseStore. ([\#9987](matrix-org/synapse#9987)) - Remove `keylen` param on `LruCache`. ([\#9993](matrix-org/synapse#9993)) - Update the Grafana dashboard in `contrib/`. ([\#10001](matrix-org/synapse#10001)) - Add a batching queue implementation. ([\#10017](matrix-org/synapse#10017)) - Reduce memory usage when verifying signatures on large numbers of events at once. ([\#10018](matrix-org/synapse#10018)) - Properly invalidate caches for destination retry timings every (instead of expiring entries every 5 minutes). ([\#10036](matrix-org/synapse#10036)) - Fix running complement tests with Synapse workers. ([\#10039](matrix-org/synapse#10039)) - Fix typo in `get_state_ids_for_event` docstring where the return type was incorrect. ([\#10050](matrix-org/synapse#10050))
Sign up for free
to subscribe to this conversation on GitHub.
Already have an account?
Sign in.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
We used an expiring cache to store the retry timings for remote servers as at the time we didn't have a reliable way of invalidating caches from workers, so we just expired the caches every 5mins. This has the disadvantage that every 5mins the server will do 100s of queries to get the retry timings of all servers its sending too.
Now that we have a reliable way of invalidating the caches we should use a standard cache.