Optimise get_rooms_for_user (drop with_stream_ordering) #13787

Fizzadar · 2022-09-13T09:13:55Z

More adventures on my quest to kill JOINs - bit more involved here: the get_rooms_for_user_with_stream_ordering function only has one use where the stream ordering is use - everywhere else this data is ignored. The only remaining call is in sync to cover the case where a join happens after the sync handler gets the current token (original change: 8cb44da).

As it stands this PR should work as-is - sync processes will hit the original function and other paths will utilize the new methods. I suspect this would yield benefit in a workers setup but probably a small negative impact for monolith installs. What I'd really like to do is remove the method entirely, some thoughts:

The current call is within the get_rooms_for_user_at method in the sync handler. If I understand correctly the aim here is to get the users rooms ensuring they were joined at the current token, rather than "now" which will be slightly ahead. I think it could be solved by doing:

Call get_rooms_for_user before we get the current token
Get the current token
Call get_membership_changes_for_user to pull any joins between 1 & 2, we already need to pull the membership events anyway during other parts of sync (would need to change this to be passed between methods)

Update: I have made a POC of the below here: beeper/synapse#11. This probably makes sense to pull into this PR once reviewed?

Signed off by Nick @ Beeper (@Fizzadar).

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file.
Pull request includes a sign off
Code style is correct
(run the linters)

This avoids the join on `events` to get stream ordering that is mostly unused.

None of the calls to this function ever use the stream ordering component returned, so can just use the simplified `get_rooms_for_users`.

Fizzadar · 2022-09-22T09:41:46Z

Note to reviewer: I have made a POC of the proposed removal of the stream ordering call described above here: beeper/synapse#11. This probably makes sense to pull into this PR once reviewed?

erikjohnston · 2022-09-23T10:28:21Z

@Fizzadar sorry for the delay on reviewing this, but I'm actually finding it quite hard to reason about the performance impact/characteristics of this change.

The changes to the sync code look quite heavyweight (which is a hot path), do you have any sense of how much the performance impact of that is?

Fizzadar · 2022-09-23T10:49:57Z

@erikjohnston no worries - the main improvement from the database query is the removal of the join to the events table to get the room list for a user(s) (this new query vs the current one). The ordering field is not actually used anywhere except one path in sync (will come back to that) so there's no need for the expensive join. The sync changes (c98bbfc) just switches to the new methods that don't include stream order.

That's the scope of this PR as it stands - I've taken the idea further in beeper/synapse#11 and replaced the use of rooms-with-stream-ordering entirely by using the membership change events instead. Logic is mostly copied over from the existing get_users_for_room_at method but doesn't require the stream ordering. That PR also further optimises the membership event lookup by doing that just once during sync.

Both should be reviewable commit-by-commit; let me know if that does/doesn't make any sense, also happy to hop on a call to discuss if that'd be easier :)

erikjohnston

I think I've convinced myself that this is safe perf-wise, especially with the other PR to get rid of get_rooms_for_users_with_stream_ordering entirely.

changelog.d/13787.misc

erikjohnston · 2022-09-27T07:39:59Z

synapse/storage/databases/main/roommember.py

+                c.type = 'm.room.member'
+                AND c.state_key = ?
+                AND c.membership = ?
+        """


We may as well just use simple_select_onecol here?

erikjohnston · 2022-09-27T07:40:58Z

synapse/storage/databases/main/roommember.py

+            WHERE
+                c.type = 'm.room.member'
+                AND c.membership = ?
+                AND {clause}


Ditto, I think we can use simple_select_many_batch to simplify things.

Co-authored-by: Erik Johnston <erikj@jki.re>

Fizzadar · 2022-09-28T09:00:16Z

@erikjohnston I have updated the function calls - would you prefer me to merge in the other PR here as well or keep the two separate? (Have updated comments on other PR too).

EDIT: will not merge develop changes in just yet pending above

erikjohnston · 2022-09-28T10:02:37Z

@erikjohnston I have updated the function calls - would you prefer me to merge in the other PR here as well or keep the two separate? (Have updated comments on other PR too).

EDIT: will not merge develop changes in just yet pending above

Let's keep the PRs separate. But can you merge in develop or something, looks like GH has just decided not to run CI on a bunch of PRs this morning, and I can't see a way of kicking it from my side 😡

# Conflicts: # synapse/storage/databases/main/events.py # synapse/storage/databases/main/roommember.py

The cached list (sensibly) requires that every input value has a corresponding output.

Fizzadar · 2022-09-28T15:54:30Z

Should be good to go now, bar a complement flake! Once this is in I'll fixup the other branch and PR here properly. Thank you for the review(s) @erikjohnston!

erikjohnston

Woop!

@lukasdenk

Synapse 1.69.0rc1 (2022-10-04) ============================== Please note that legacy Prometheus metric names are now deprecated and will be removed in Synapse 1.73.0. Server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.69/upgrade.html#upgrading-to-v1690) for more details. Features -------- - Allow application services to set the `origin_server_ts` of a state event by providing the query parameter `ts` in [`PUT /_matrix/client/r0/rooms/{roomId}/state/{eventType}/{stateKey}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3roomsroomidstateeventtypestatekey), per [MSC3316](matrix-org/matrix-spec-proposals#3316). Contributed by @lukasdenk. ([\#11866](#11866)) - Allow server admins to require a manual approval process before new accounts can be used (using [MSC3866](matrix-org/matrix-spec-proposals#3866)). ([\#13556](#13556)) - Exponentially backoff from backfilling the same event over and over. ([\#13635](#13635), [\#13936](#13936)) - Add cache invalidation across workers to module API. ([\#13667](#13667), [\#13947](#13947)) - Experimental implementation of [MSC3882](matrix-org/matrix-spec-proposals#3882) to allow an existing device/session to generate a login token for use on a new device/session. ([\#13722](#13722), [\#13868](#13868)) - Experimental support for thread-specific receipts ([MSC3771](matrix-org/matrix-spec-proposals#3771)). ([\#13782](#13782), [\#13893](#13893), [\#13932](#13932), [\#13937](#13937), [\#13939](#13939)) - Add experimental support for [MSC3881: Remotely toggle push notifications for another client](matrix-org/matrix-spec-proposals#3881). ([\#13799](#13799), [\#13831](#13831), [\#13860](#13860)) - Keep track when an event pulled over federation fails its signature check so we can intelligently back-off in the future. ([\#13815](#13815)) - Improve validation for the unspecced, internal-only `_matrix/client/unstable/add_threepid/msisdn/submit_token` endpoint. ([\#13832](#13832)) - Faster remote room joins: record _when_ we first partial-join to a room. ([\#13892](#13892)) - Support a `dir` parameter on the `/relations` endpoint per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\#13920](#13920)) - Ask mail servers receiving emails from Synapse to not send automatic replies (e.g. out-of-office responses). ([\#13957](#13957)) Bugfixes -------- - Send push notifications for invites received over federation. ([\#13719](#13719), [\#14014](#14014)) - Fix a long-standing bug where typing events would be accepted from remote servers not present in a room. Also fix a bug where incoming typing events would cause other incoming events to get stuck during a fast join. ([\#13830](#13830)) - Fix a bug introduced in Synapse v1.53.0 where the experimental implementation of [MSC3715](matrix-org/matrix-spec-proposals#3715) would give incorrect results when paginating forward. ([\#13840](#13840)) - Fix access token leak to logs from proxy agent. ([\#13855](#13855)) - Fix `have_seen_event` cache not being invalidated after we persist an event which causes inefficiency effects like extra `/state` federation calls. ([\#13863](#13863)) - Faster room joins: Fix a bug introduced in 1.66.0 where an error would be logged when syncing after joining a room. ([\#13872](#13872)) - Fix a bug introduced in 1.66.0 where some required fields in the pushrules sent to clients were not present anymore. Contributed by Nico. ([\#13904](#13904)) - Fix packaging to include `Cargo.lock` in `sdist`. ([\#13909](#13909)) - Fix a long-standing bug where device updates could cause delays sending out to-device messages over federation. ([\#13922](#13922)) - Fix a bug introduced in v1.68.0 where Synapse would require `setuptools_rust` at runtime, even though the package is only required at build time. ([\#13952](#13952)) - Fix a long-standing bug where `POST /_matrix/client/v3/keys/query` requests could result in excessively large SQL queries. ([\#13956](#13956)) - Fix a performance regression in the `get_users_in_room` database query. Introduced in v1.67.0. ([\#13972](#13972)) - Fix a bug introduced in v1.68.0 bug where Rust extension wasn't built in `release` mode when using `poetry install`. ([\#14009](#14009)) - Do not return an unspecified `original_event` field when using the stable `/relations` endpoint. Introduced in Synapse v1.57.0. ([\#14025](#14025)) - Correctly handle a race with device lists when a remote user leaves during a partial join. ([\#13885](#13885)) - Correctly handle sending local device list updates to remote servers during a partial join. ([\#13934](#13934)) Improved Documentation ---------------------- - Add `worker_main_http_uri` for the worker generator bash script. ([\#13772](#13772)) - Update URL for the NixOS module for Synapse. ([\#13818](#13818)) - Fix a mistake in sso_mapping_providers.md: `map_user_attributes` is expected to return `display_name`, not `displayname`. ([\#13836](#13836)) - Fix a cross-link from the registration admin API to the `registration_shared_secret` configuration documentation. ([\#13870](#13870)) - Update the man page for the `hash_password` script to correct the default number of bcrypt rounds performed. ([\#13911](#13911), [\#13930](#13930)) - Emphasize the right reasons when to use `(room_id, event_id)` in a database schema. ([\#13915](#13915)) - Add instruction to contributing guide for running unit tests in parallel. Contributed by @ashfame. ([\#13928](#13928)) - Clarify that the `auto_join_rooms` config option can also be used with Space aliases. ([\#13931](#13931)) - Add some cross references to worker documentation. ([\#13974](#13974)) - Linkify urls in config documentation. ([\#14003](#14003)) Deprecations and Removals ------------------------- - Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. ([\#13843](#13843)) - Announce that legacy metric names are deprecated, will be turned off by default in Synapse v1.71.0 and removed altogether in Synapse v1.73.0. See the upgrade notes for more information. ([\#14024](#14024)) Internal Changes ---------------- - Speed up creation of DM rooms. ([\#13487](#13487), [\#13800](#13800)) - Port push rules to using Rust. ([\#13768](#13768), [\#13838](#13838), [\#13889](#13889)) - Optimise get rooms for user calls. Contributed by Nick @ Beeper (@Fizzadar). ([\#13787](#13787)) - Update the script which makes full schema dumps. ([\#13792](#13792)) - Use shared methods for cache invalidation when persisting events, remove duplicate codepaths. Contributed by Nick @ Beeper (@Fizzadar). ([\#13796](#13796)) - Improve the `synapse.api.auth.Auth` mock used in unit tests. ([\#13809](#13809)) - Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. ([\#13823](#13823)) - Carry IdP Session IDs through user-mapping sessions. ([\#13839](#13839)) - Fix the release script not publishing binary wheels. ([\#13850](#13850)) - Raise issue if complement fails with latest deps. ([\#13859](#13859)) - Correct the comments in the complement dockerfile. ([\#13867](#13867)) - Create a new snapshot of the database schema. ([\#13873](#13873)) - Faster room joins: Send device list updates to most servers in rooms with partial state. ([\#13874](#13874), [\#14013](#14013)) - Add comments to the Prometheus recording rules to make it clear which set of rules you need for Grafana or Prometheus Console. ([\#13876](#13876)) - Only pull relevant backfill points from the database based on the current depth and limit (instead of all) every time we want to `/backfill`. ([\#13879](#13879)) - Faster room joins: Avoid waiting for full state when processing `/keys/changes` requests. ([\#13888](#13888)) - Improve backfill robustness by trying more servers when we get a `4xx` error back. ([\#13890](#13890)) - Fix mypy errors with canonicaljson 1.6.3. ([\#13905](#13905)) - Faster remote room joins: correctly handle remote device list updates during a partial join. ([\#13913](#13913)) - Complement image: propagate SIGTERM to all workers. ([\#13914](#13914)) - Update an innaccurate comment in Synapse's upsert database helper. ([\#13924](#13924)) - Update mypy (0.950 -> 0.981) and mypy-zope (0.3.7 -> 0.3.11). ([\#13925](#13925), [\#13993](#13993)) - Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating users to copy over during a room upgrade. ([\#13960](#13960)) - Refactor language in user directory `_track_user_joined_room` code to make it more clear that we use both local and remote users. ([\#13966](#13966)) - Revert catch-all exceptions being recorded as event pull attempt failures (only handle what we know about). ([\#13969](#13969)) - Speed up calculating push actions in large rooms. ([\#13973](#13973), [\#13992](#13992)) - Enable update notifications from Github's dependabot. ([\#13976](#13976)) - Prototype a workflow to automatically add changelogs to dependabot PRs. ([\#13998](#13998), [\#14011](#14011), [\#14017](#14017), [\#14021](#14021), [\#14027](#14027)) - Fix type annotations to be compatible with new annotations in development versions of twisted. ([\#14012](#14012)) - Clear out stale entries in `event_push_actions_staging` table. ([\#14020](#14020)) - Bump versions of GitHub actions. ([\#13978](#13978), [\#13979](#13979), [\#13980](#13980), [\#13982](#13982), [\#14015](#14015), [\#14019](#14019), [\#14022](#14022), [\#14023](#14023))

…3787) # Conflicts: # synapse/storage/_base.py # synapse/storage/databases/main/roommember.py

@lukasdenk

NOTE: this is absolutely *not* safe for Beeper usage as-is. I have merged all of the Python code in but all our customizations to the base rules and push rule evaluator are not yet present in the new Rust module. This will fail tests as-is and future commits will re-apply our changes in Rust. Synapse 1.69.0 (2022-10-17) =========================== Please note that legacy Prometheus metric names are now deprecated and will be removed in Synapse 1.73.0. Server administrators should update their dashboards and alerting rules to avoid using the deprecated metric names. See the [upgrade notes](https://matrix-org.github.io/synapse/v1.69/upgrade.html#upgrading-to-v1690) for more details. No significant changes since 1.69.0rc4. Synapse 1.69.0rc4 (2022-10-14) ============================== Bugfixes -------- - Fix poor performance of the `event_push_backfill_thread_id` background update, which was introduced in Synapse 1.68.0rc1. ([\matrix-org#14172](matrix-org#14172), [\matrix-org#14181](matrix-org#14181)) Updates to the Docker image --------------------------- - Fix docker build OOMing in CI for arm64 builds. ([\matrix-org#14173](matrix-org#14173)) Synapse 1.69.0rc3 (2022-10-12) ============================== Bugfixes -------- - Fix an issue with Docker images causing the Rust dependencies to not be pinned correctly. Introduced in v1.68.0 ([\matrix-org#14129](matrix-org#14129)) - Fix a bug introduced in Synapse 1.69.0rc1 which would cause registration replication requests to fail if the worker sending the request is not running Synapse 1.69. ([\matrix-org#14135](matrix-org#14135)) - Fix error in background update when rotating existing notifications. Introduced in v1.69.0rc2. ([\matrix-org#14138](matrix-org#14138)) Internal Changes ---------------- - Rename the `url_preview` extra to `url-preview`, for compatability with poetry-core 1.3.0 and [PEP 685](https://peps.python.org/pep-0685/). From-source installations using this extra will need to install using the new name. ([\matrix-org#14085](matrix-org#14085)) Synapse 1.69.0rc2 (2022-10-06) ============================== Deprecations and Removals ------------------------- - Deprecate the `generate_short_term_login_token` method in favor of an async `create_login_token` method in the Module API. ([\matrix-org#13842](matrix-org#13842)) Internal Changes ---------------- - Ensure Synapse v1.69 works with upcoming database changes in v1.70. ([\matrix-org#14045](matrix-org#14045)) - Fix a bug introduced in Synapse v1.68.0 where messages could not be sent in rooms with non-integer `notifications` power level. ([\matrix-org#14073](matrix-org#14073)) - Temporarily pin build-system requirements to workaround an incompatibility with poetry-core 1.3.0. This will be reverted before the v1.69.0 release proper, see [\matrix-org#14079](matrix-org#14079). ([\matrix-org#14080](matrix-org#14080)) Synapse 1.69.0rc1 (2022-10-04) ============================== Features -------- - Allow application services to set the `origin_server_ts` of a state event by providing the query parameter `ts` in [`PUT /_matrix/client/r0/rooms/{roomId}/state/{eventType}/{stateKey}`](https://spec.matrix.org/v1.4/client-server-api/#put_matrixclientv3roomsroomidstateeventtypestatekey), per [MSC3316](matrix-org/matrix-spec-proposals#3316). Contributed by @lukasdenk. ([\matrix-org#11866](matrix-org#11866)) - Allow server admins to require a manual approval process before new accounts can be used (using [MSC3866](matrix-org/matrix-spec-proposals#3866)). ([\matrix-org#13556](matrix-org#13556)) - Exponentially backoff from backfilling the same event over and over. ([\matrix-org#13635](matrix-org#13635), [\matrix-org#13936](matrix-org#13936)) - Add cache invalidation across workers to module API. ([\matrix-org#13667](matrix-org#13667), [\matrix-org#13947](matrix-org#13947)) - Experimental implementation of [MSC3882](matrix-org/matrix-spec-proposals#3882) to allow an existing device/session to generate a login token for use on a new device/session. ([\matrix-org#13722](matrix-org#13722), [\matrix-org#13868](matrix-org#13868)) - Experimental support for thread-specific receipts ([MSC3771](matrix-org/matrix-spec-proposals#3771)). ([\matrix-org#13782](matrix-org#13782), [\matrix-org#13893](matrix-org#13893), [\matrix-org#13932](matrix-org#13932), [\matrix-org#13937](matrix-org#13937), [\matrix-org#13939](matrix-org#13939)) - Add experimental support for [MSC3881: Remotely toggle push notifications for another client](matrix-org/matrix-spec-proposals#3881). ([\matrix-org#13799](matrix-org#13799), [\matrix-org#13831](matrix-org#13831), [\matrix-org#13860](matrix-org#13860)) - Keep track when an event pulled over federation fails its signature check so we can intelligently back-off in the future. ([\matrix-org#13815](matrix-org#13815)) - Improve validation for the unspecced, internal-only `_matrix/client/unstable/add_threepid/msisdn/submit_token` endpoint. ([\matrix-org#13832](matrix-org#13832)) - Faster remote room joins: record _when_ we first partial-join to a room. ([\matrix-org#13892](matrix-org#13892)) - Support a `dir` parameter on the `/relations` endpoint per [MSC3715](matrix-org/matrix-spec-proposals#3715). ([\matrix-org#13920](matrix-org#13920)) - Ask mail servers receiving emails from Synapse to not send automatic replies (e.g. out-of-office responses). ([\matrix-org#13957](matrix-org#13957)) Bugfixes -------- - Send push notifications for invites received over federation. ([\matrix-org#13719](matrix-org#13719), [\matrix-org#14014](matrix-org#14014)) - Fix a long-standing bug where typing events would be accepted from remote servers not present in a room. Also fix a bug where incoming typing events would cause other incoming events to get stuck during a fast join. ([\matrix-org#13830](matrix-org#13830)) - Fix a bug introduced in Synapse v1.53.0 where the experimental implementation of [MSC3715](matrix-org/matrix-spec-proposals#3715) would give incorrect results when paginating forward. ([\matrix-org#13840](matrix-org#13840)) - Fix access token leak to logs from proxy agent. ([\matrix-org#13855](matrix-org#13855)) - Fix `have_seen_event` cache not being invalidated after we persist an event which causes inefficiency effects like extra `/state` federation calls. ([\matrix-org#13863](matrix-org#13863)) - Faster room joins: Fix a bug introduced in 1.66.0 where an error would be logged when syncing after joining a room. ([\matrix-org#13872](matrix-org#13872)) - Fix a bug introduced in 1.66.0 where some required fields in the pushrules sent to clients were not present anymore. Contributed by Nico. ([\matrix-org#13904](matrix-org#13904)) - Fix packaging to include `Cargo.lock` in `sdist`. ([\matrix-org#13909](matrix-org#13909)) - Fix a long-standing bug where device updates could cause delays sending out to-device messages over federation. ([\matrix-org#13922](matrix-org#13922)) - Fix a bug introduced in v1.68.0 where Synapse would require `setuptools_rust` at runtime, even though the package is only required at build time. ([\matrix-org#13952](matrix-org#13952)) - Fix a long-standing bug where `POST /_matrix/client/v3/keys/query` requests could result in excessively large SQL queries. ([\matrix-org#13956](matrix-org#13956)) - Fix a performance regression in the `get_users_in_room` database query. Introduced in v1.67.0. ([\matrix-org#13972](matrix-org#13972)) - Fix a bug introduced in v1.68.0 bug where Rust extension wasn't built in `release` mode when using `poetry install`. ([\matrix-org#14009](matrix-org#14009)) - Do not return an unspecified `original_event` field when using the stable `/relations` endpoint. Introduced in Synapse v1.57.0. ([\matrix-org#14025](matrix-org#14025)) - Correctly handle a race with device lists when a remote user leaves during a partial join. ([\matrix-org#13885](matrix-org#13885)) - Correctly handle sending local device list updates to remote servers during a partial join. ([\matrix-org#13934](matrix-org#13934)) Improved Documentation ---------------------- - Add `worker_main_http_uri` for the worker generator bash script. ([\matrix-org#13772](matrix-org#13772)) - Update URL for the NixOS module for Synapse. ([\matrix-org#13818](matrix-org#13818)) - Fix a mistake in sso_mapping_providers.md: `map_user_attributes` is expected to return `display_name`, not `displayname`. ([\matrix-org#13836](matrix-org#13836)) - Fix a cross-link from the registration admin API to the `registration_shared_secret` configuration documentation. ([\matrix-org#13870](matrix-org#13870)) - Update the man page for the `hash_password` script to correct the default number of bcrypt rounds performed. ([\matrix-org#13911](matrix-org#13911), [\matrix-org#13930](matrix-org#13930)) - Emphasize the right reasons when to use `(room_id, event_id)` in a database schema. ([\matrix-org#13915](matrix-org#13915)) - Add instruction to contributing guide for running unit tests in parallel. Contributed by @ashfame. ([\matrix-org#13928](matrix-org#13928)) - Clarify that the `auto_join_rooms` config option can also be used with Space aliases. ([\matrix-org#13931](matrix-org#13931)) - Add some cross references to worker documentation. ([\matrix-org#13974](matrix-org#13974)) - Linkify urls in config documentation. ([\matrix-org#14003](matrix-org#14003)) Deprecations and Removals ------------------------- - Remove the `complete_sso_login` method from the Module API which was deprecated in Synapse 1.13.0. ([\matrix-org#13843](matrix-org#13843)) - Announce that legacy metric names are deprecated, will be turned off by default in Synapse v1.71.0 and removed altogether in Synapse v1.73.0. See the upgrade notes for more information. ([\matrix-org#14024](matrix-org#14024)) Internal Changes ---------------- - Speed up creation of DM rooms. ([\matrix-org#13487](matrix-org#13487), [\matrix-org#13800](matrix-org#13800)) - Port push rules to using Rust. ([\matrix-org#13768](matrix-org#13768), [\matrix-org#13838](matrix-org#13838), [\matrix-org#13889](matrix-org#13889)) - Optimise get rooms for user calls. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13787](matrix-org#13787)) - Update the script which makes full schema dumps. ([\matrix-org#13792](matrix-org#13792)) - Use shared methods for cache invalidation when persisting events, remove duplicate codepaths. Contributed by Nick @ Beeper (@Fizzadar). ([\matrix-org#13796](matrix-org#13796)) - Improve the `synapse.api.auth.Auth` mock used in unit tests. ([\matrix-org#13809](matrix-org#13809)) - Faster Remote Room Joins: tell remote homeservers that we are unable to authorise them if they query a room which has partial state on our server. ([\matrix-org#13823](matrix-org#13823)) - Carry IdP Session IDs through user-mapping sessions. ([\matrix-org#13839](matrix-org#13839)) - Fix the release script not publishing binary wheels. ([\matrix-org#13850](matrix-org#13850)) - Raise issue if complement fails with latest deps. ([\matrix-org#13859](matrix-org#13859)) - Correct the comments in the complement dockerfile. ([\matrix-org#13867](matrix-org#13867)) - Create a new snapshot of the database schema. ([\matrix-org#13873](matrix-org#13873)) - Faster room joins: Send device list updates to most servers in rooms with partial state. ([\matrix-org#13874](matrix-org#13874), [\matrix-org#14013](matrix-org#14013)) - Add comments to the Prometheus recording rules to make it clear which set of rules you need for Grafana or Prometheus Console. ([\matrix-org#13876](matrix-org#13876)) - Only pull relevant backfill points from the database based on the current depth and limit (instead of all) every time we want to `/backfill`. ([\matrix-org#13879](matrix-org#13879)) - Faster room joins: Avoid waiting for full state when processing `/keys/changes` requests. ([\matrix-org#13888](matrix-org#13888)) - Improve backfill robustness by trying more servers when we get a `4xx` error back. ([\matrix-org#13890](matrix-org#13890)) - Fix mypy errors with canonicaljson 1.6.3. ([\matrix-org#13905](matrix-org#13905)) - Faster remote room joins: correctly handle remote device list updates during a partial join. ([\matrix-org#13913](matrix-org#13913)) - Complement image: propagate SIGTERM to all workers. ([\matrix-org#13914](matrix-org#13914)) - Update an innaccurate comment in Synapse's upsert database helper. ([\matrix-org#13924](matrix-org#13924)) - Update mypy (0.950 -> 0.981) and mypy-zope (0.3.7 -> 0.3.11). ([\matrix-org#13925](matrix-org#13925), [\matrix-org#13993](matrix-org#13993)) - Use dedicated `get_local_users_in_room(room_id)` function to find local users when calculating users to copy over during a room upgrade. ([\matrix-org#13960](matrix-org#13960)) - Refactor language in user directory `_track_user_joined_room` code to make it more clear that we use both local and remote users. ([\matrix-org#13966](matrix-org#13966)) - Revert catch-all exceptions being recorded as event pull attempt failures (only handle what we know about). ([\matrix-org#13969](matrix-org#13969)) - Speed up calculating push actions in large rooms. ([\matrix-org#13973](matrix-org#13973), [\matrix-org#13992](matrix-org#13992)) - Enable update notifications from Github's dependabot. ([\matrix-org#13976](matrix-org#13976)) - Prototype a workflow to automatically add changelogs to dependabot PRs. ([\matrix-org#13998](matrix-org#13998), [\matrix-org#14011](matrix-org#14011), [\matrix-org#14017](matrix-org#14017), [\matrix-org#14021](matrix-org#14021), [\matrix-org#14027](matrix-org#14027)) - Fix type annotations to be compatible with new annotations in development versions of twisted. ([\matrix-org#14012](matrix-org#14012)) - Clear out stale entries in `event_push_actions_staging` table. ([\matrix-org#14020](matrix-org#14020)) - Bump versions of GitHub actions. ([\matrix-org#13978](matrix-org#13978), [\matrix-org#13979](matrix-org#13979), [\matrix-org#13980](matrix-org#13980), [\matrix-org#13982](matrix-org#13982), [\matrix-org#14015](matrix-org#14015), [\matrix-org#14019](matrix-org#14019), [\matrix-org#14022](matrix-org#14022), [\matrix-org#14023](matrix-org#14023)) # -----BEGIN PGP SIGNATURE----- # # iQFEBAABCgAuFiEEBTGR3/RnAzBGUif3pULk7RsPrAkFAmNNMOMQHGVyaWtAbWF0 # cml4Lm9yZwAKCRClQuTtGw+sCVnjB/9jpJRVnicteEpDfVX9iLo2qfIfcO/GhUJK # pJhv4yuY9whAldvJpmNw2f9tfUbAMcvrjlFvNrjihWmXcAGFazC6i3fNBjPgZW2e # Sxsuuy8xc9X/OqH2EUpHtNZQX3FfSbdBS93Z62ZO3R8tEbCQvjw6FXBdjjjf5uLO # y5Lsx94+41FJYOhs1Kt4fN92B9WMACR6e/O1YcsDjIXsoZI3uqO1h8filbQIZee7 # DTATE7eIPtShs2Ezaaeuc7tZGVDyPvgWIbuxuT6OGx20zmuChYJgIcVaD1me4UzJ # i9bVigtpYN0eUxuWnjLf7YC6Ys/Y9wZ7/lhdgaBwdbQKEJdpi+S4 # =JWaO # -----END PGP SIGNATURE----- # gpg: Signature made Mon Oct 17 11:39:31 2022 BST # gpg: using RSA key 053191DFF4670330465227F7A542E4ED1B0FAC09 # gpg: issuer "erik@matrix.org" # gpg: Can't check signature: No public key # Conflicts: # docker/Dockerfile # pyproject.toml # synapse/_scripts/update_synapse_database.py # synapse/handlers/message.py # synapse/handlers/receipts.py # synapse/logging/context.py # synapse/push/baserules.py # synapse/push/bulk_push_rule_evaluator.py # synapse/push/push_rule_evaluator.py # synapse/replication/http/send_event.py # synapse/rest/admin/users.py # synapse/rest/client/read_marker.py # synapse/rest/client/receipts.py # synapse/rest/client/room.py # synapse/storage/_base.py # synapse/storage/databases/main/__init__.py # synapse/storage/databases/main/cache.py # synapse/storage/databases/main/events.py # synapse/storage/databases/main/receipts.py # tests/push/test_push_rule_evaluator.py

#11915 introduced the `@cached` `is_interested_in_room` method in Synapse 1.55.0, which depends upon `get_aliases_for_room`. Add a missing cache invalidation callback so that the `is_interested_in_room` cache is invalidated when `get_aliases_for_room` is invalidated. #13787 made `get_rooms_for_user` `@cached`. Add a missing cache invalidation callback so that the `is_interested_in_presence` cache is invalidated when `get_rooms_for_user` is invalidated. Signed-off-by: Sean Quah <seanq@matrix.org>

Fizzadar added 6 commits September 13, 2022 09:26

Reimplement get_rooms_for_user and get_rooms_for_users

1008106

This avoids the join on `events` to get stream ordering that is mostly unused.

Replace calls to get_rooms_for_users_with_stream_ordering

c98bbfc

None of the calls to this function ever use the stream ordering component returned, so can just use the simplified `get_rooms_for_users`.

Drop unused get_rooms_for_users_with_stream_ordering method

7210fa1

Drop simplified cache in sync tests

7719d9c

Linting fixes

b745518

Fix call to get_immediate

661e7cf

Fizzadar requested a review from a team as a code owner September 13, 2022 09:13

Fizzadar added 2 commits September 13, 2022 10:26

Add placeholder changelog file

3df4579

Fix formatting

0497004

Fizzadar force-pushed the optimise-get-rooms-for-user branch from b9c4f7d to 0497004 Compare September 13, 2022 09:27

Don't overwrite variable

7e127f8

Fizzadar mentioned this pull request Sep 21, 2022

Optimise get rooms for user sync part 2 beeper/synapse-legacy-fork#11

Closed

4 tasks

Merge branch 'develop' into optimise-get-rooms-for-user

b97ae86

erikjohnston reviewed Sep 27, 2022

View reviewed changes

Fizzadar and others added 3 commits September 28, 2022 08:45

Remove blank changelog line

7512017

Co-authored-by: Erik Johnston <erikj@jki.re>

Use simple_select_many_batch in get_rooms_for_users

21c8e8d

Use simple_select_onecol in get_rooms_for_user

9907aaa

Fizzadar requested a review from erikjohnston September 28, 2022 09:00

erikjohnston approved these changes Sep 28, 2022

View reviewed changes

Fizzadar added 4 commits September 28, 2022 13:59

Merge branch 'develop' into optimise-get-rooms-for-user

ff3ae7e

# Conflicts: # synapse/storage/databases/main/events.py # synapse/storage/databases/main/roommember.py

Fixed merge formatting

4da7073

Fix current_state_events column name

bd70af9

Always ensure we return a value for every user in get_rooms_for_users

7b0e311

The cached list (sensibly) requires that every input value has a corresponding output.

Fizzadar force-pushed the optimise-get-rooms-for-user branch from 740d42f to 7b0e311 Compare September 28, 2022 14:54

Fizzadar requested a review from erikjohnston September 28, 2022 15:54

erikjohnston approved these changes Sep 29, 2022

View reviewed changes

erikjohnston enabled auto-merge (squash) September 29, 2022 13:30

erikjohnston merged commit a466164 into matrix-org:develop Sep 29, 2022

Fizzadar mentioned this pull request Sep 30, 2022

Remove get rooms for user with stream ordering #13991

Merged

4 tasks

Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Oct 4, 2022

Optimise get_rooms_for_user (drop with_stream_ordering) (matrix-org#1…

a0fc5fe

…3787) # Conflicts: # synapse/storage/_base.py # synapse/storage/databases/main/roommember.py

Fizzadar added a commit to beeper/synapse-legacy-fork that referenced this pull request Oct 4, 2022

Optimise get_rooms_for_user (drop with_stream_ordering) (matrix-org#1…

74f3088

…3787) # Conflicts: # synapse/storage/_base.py # synapse/storage/databases/main/roommember.py

squahtx mentioned this pull request Dec 12, 2022

Fix missing cache invalidation in application service code #14670

Merged

Fizzadar deleted the optimise-get-rooms-for-user branch January 24, 2023 10:52

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Optimise get_rooms_for_user (drop with_stream_ordering) #13787

Optimise get_rooms_for_user (drop with_stream_ordering) #13787

Fizzadar commented Sep 13, 2022 •

edited

Loading

Fizzadar commented Sep 22, 2022

erikjohnston commented Sep 23, 2022

Fizzadar commented Sep 23, 2022 •

edited

Loading

erikjohnston left a comment

erikjohnston Sep 27, 2022

Fizzadar Sep 28, 2022

erikjohnston Sep 27, 2022

Fizzadar Sep 28, 2022

Fizzadar commented Sep 28, 2022 •

edited

Loading

erikjohnston commented Sep 28, 2022

Fizzadar commented Sep 28, 2022

erikjohnston left a comment

Optimise get_rooms_for_user (drop with_stream_ordering) #13787

Optimise get_rooms_for_user (drop with_stream_ordering) #13787

Conversation

Fizzadar commented Sep 13, 2022 • edited Loading

Pull Request Checklist

Fizzadar commented Sep 22, 2022

erikjohnston commented Sep 23, 2022

Fizzadar commented Sep 23, 2022 • edited Loading

erikjohnston left a comment

Choose a reason for hiding this comment

erikjohnston Sep 27, 2022

Choose a reason for hiding this comment

Fizzadar Sep 28, 2022

Choose a reason for hiding this comment

erikjohnston Sep 27, 2022

Choose a reason for hiding this comment

Fizzadar Sep 28, 2022

Choose a reason for hiding this comment

Fizzadar commented Sep 28, 2022 • edited Loading

erikjohnston commented Sep 28, 2022

Fizzadar commented Sep 28, 2022

erikjohnston left a comment

Choose a reason for hiding this comment

Fizzadar commented Sep 13, 2022 •

edited

Loading

Fizzadar commented Sep 23, 2022 •

edited

Loading

Fizzadar commented Sep 28, 2022 •

edited

Loading