More speedups/fixes to creating batched events #15195

H-Shay · 2023-03-02T20:51:13Z

These changes come from a discussion with Erik and are follow-ups to the batching persisting event work, speeding it up/making it more efficient. Best reviewable commit-by-commit, and I can split these out into separate PRs if others think it necessary.

…in db anyway)

synapse/event_auth.py

synapse/handlers/event_auth.py

synapse/event_auth.py

synapse/events/snapshot.py

synapse/handlers/room.py

clokep · 2023-03-07T14:57:33Z

changelog.d/15195.misc

@@ -0,0 +1 @@
+Misc speedups to creating and authing events.


Suggested change

Misc speedups to creating and authing events.

Improve performance of creating and authenticating events.

synapse/events/snapshot.py

clokep

LGTM if CI is happy.

H-Shay · 2023-03-07T21:54:01Z

Merged #15195 into main, thanks for the review.

Synapse 1.80.0 (2023-03-28) =========================== No significant changes since 1.80.0rc2. Synapse 1.80.0rc2 (2023-03-22) ============================== Bugfixes -------- - Fix a bug in which the [`POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3roomsroomidreporteventid) endpoint would return the wrong error if the user did not have permission to view the event. This aligns Synapse's implementation with [MSC2249](matrix-org/matrix-spec-proposals#2249). ([\#15298](matrix-org/synapse#15298), [\#15300](matrix-org/synapse#15300)) - Fix a bug introduced in Synapse 1.75.0rc1 where the [SQLite port_db script](https://matrix-org.github.io/synapse/latest/postgres.html#porting-from-sqlite) would fail to open the SQLite database. ([\#15301](matrix-org/synapse#15301)) Synapse 1.80.0rc1 (2023-03-21) ============================== Features -------- - Stabilise support for [MSC3966](matrix-org/matrix-spec-proposals#3966): `event_property_contains` push condition. ([\#15187](matrix-org/synapse#15187)) - Implement [MSC2659](matrix-org/matrix-spec-proposals#2659): application service ping endpoint. Contributed by Tulir @ Beeper. ([\#15249](matrix-org/synapse#15249)) - Allow loading `/register/available` endpoint on workers. ([\#15268](matrix-org/synapse#15268)) - Improve performance of creating and authenticating events. ([\#15195](matrix-org/synapse#15195)) - Add topic and name events to group of events that are batch persisted when creating a room. ([\#15229](matrix-org/synapse#15229)) Bugfixes -------- - Fix a long-standing bug in which the user directory would assume any remote membership state events represent a profile change. ([\#14755](matrix-org/synapse#14755), [\#14756](matrix-org/synapse#14756)) - Implement [MSC3873](matrix-org/matrix-spec-proposals#3873) to fix a long-standing bug where properties with dots were handled ambiguously in push rules. ([\#15190](matrix-org/synapse#15190)) - Faster joins: Fix a bug introduced in Synapse 1.66 where spurious "Failed to find memberships ..." errors would be logged. ([\#15232](matrix-org/synapse#15232)) - Fix a long-standing error when sending message into deleted room. ([\#15235](matrix-org/synapse#15235)) Updates to the Docker image --------------------------- - Ensure the Dockerfile builds on platforms that don't have a `cryptography` wheel. ([\#15239](matrix-org/synapse#15239)) - Mirror images to the GitHub Container Registry (`ghcr.io/matrix-org/synapse`). ([\#15281](matrix-org/synapse#15281), [\#15282](matrix-org/synapse#15282)) Improved Documentation ---------------------- - Add a missing endpoint to the workers documentation. ([\#15223](matrix-org/synapse#15223)) Internal Changes ---------------- - Add additional functionality to declaring worker types when starting Complement in worker mode. ([\#14921](matrix-org/synapse#14921)) - Add `Synapse-Trace-Id` to `access-control-expose-headers` header. ([\#14974](matrix-org/synapse#14974)) - Make the `HttpTransactionCache` use the `Requester` in addition of the just the `Request` to build the transaction key. ([\#15200](matrix-org/synapse#15200)) - Improve log lines when purging rooms. ([\#15222](matrix-org/synapse#15222)) - Improve type hints. ([\#15230](matrix-org/synapse#15230), [\#15231](matrix-org/synapse#15231), [\#15238](matrix-org/synapse#15238)) - Move various module API callback registration methods to a dedicated class. ([\#15237](matrix-org/synapse#15237)) - Configure GitHub Actions for merge queues. ([\#15244](matrix-org/synapse#15244)) - Add schema comments about the `destinations` and `destination_rooms` tables. ([\#15247](matrix-org/synapse#15247)) - Skip processing of auto-join room behaviour if there are no auto-join rooms configured. ([\#15262](matrix-org/synapse#15262)) - Remove unused store method `_set_destination_retry_timings_emulated`. ([\#15266](matrix-org/synapse#15266)) - Reorganize URL preview code. ([\#15269](matrix-org/synapse#15269)) - Clean-up direct TCP replication code. ([\#15272](matrix-org/synapse#15272), [\#15274](matrix-org/synapse#15274)) - Make `configure_workers_and_start` script used in Complement tests compatible with older versions of Python. ([\#15275](matrix-org/synapse#15275)) - Add a `/versions` flag for [MSC3952](matrix-org/matrix-spec-proposals#3952). ([\#15293](matrix-org/synapse#15293)) - Bump hiredis from 2.2.1 to 2.2.2. ([\#15252](matrix-org/synapse#15252)) - Bump serde from 1.0.152 to 1.0.155. ([\#15253](matrix-org/synapse#15253)) - Bump pysaml2 from 7.2.1 to 7.3.1. ([\#15254](matrix-org/synapse#15254)) - Bump msgpack from 1.0.4 to 1.0.5. ([\#15255](matrix-org/synapse#15255)) - Bump gitpython from 3.1.30 to 3.1.31. ([\#15256](matrix-org/synapse#15256)) - Bump cryptography from 39.0.1 to 39.0.2. ([\#15257](matrix-org/synapse#15257)) - Bump pydantic from 1.10.4 to 1.10.6. ([\#15286](matrix-org/synapse#15286)) - Bump serde from 1.0.155 to 1.0.157. ([\#15287](matrix-org/synapse#15287)) - Bump anyhow from 1.0.69 to 1.0.70. ([\#15288](matrix-org/synapse#15288)) - Bump txredisapi from 1.4.7 to 1.4.9. ([\#15289](matrix-org/synapse#15289)) - Bump pygithub from 1.57 to 1.58.1. ([\#15290](matrix-org/synapse#15290)) - Bump types-requests from 2.28.11.12 to 2.28.11.15. ([\#15291](matrix-org/synapse#15291)) Synapse 1.79.0 (2023-03-14) =========================== No significant changes since 1.79.0rc2. Synapse 1.79.0rc2 (2023-03-13) ============================== Bugfixes -------- - Fix a bug introduced in Synapse 1.79.0rc1 where attempting to register a `on_remove_user_third_party_identifier` module API callback would be a no-op. ([\#15227](matrix-org/synapse#15227)) - Fix a rare bug introduced in Synapse 1.73 where events could remain unsent to other homeservers after a faster-join to a room. ([\#15248](matrix-org/synapse#15248)) Internal Changes ---------------- - Refactor `filter_events_for_server`. ([\#15240](matrix-org/synapse#15240)) Synapse 1.79.0rc1 (2023-03-07) ============================== Features -------- - Add two new Third Party Rules module API callbacks: [`on_add_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_add_user_third_party_identifier) and [`on_remove_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_remove_user_third_party_identifier). ([\#15044](matrix-org/synapse#15044)) - Experimental support for [MSC3967](matrix-org/matrix-spec-proposals#3967) to not require UIA for setting up cross-signing on first use. ([\#15077](matrix-org/synapse#15077)) - Add media information to the command line [user data export tool](https://matrix-org.github.io/synapse/v1.79/usage/administration/admin_faq.html#how-can-i-export-user-data). ([\#15107](matrix-org/synapse#15107)) - Add an [admin API](https://matrix-org.github.io/synapse/latest/usage/administration/admin_api/index.html) to delete a [specific event report](https://spec.matrix.org/v1.6/client-server-api/#reporting-content). ([\#15116](matrix-org/synapse#15116)) - Add support for knocking to workers. ([\#15133](matrix-org/synapse#15133)) - Allow use of the `/filter` Client-Server APIs on workers. ([\#15134](matrix-org/synapse#15134)) - Update support for [MSC2677](matrix-org/matrix-spec-proposals#2677): remove support for server-side aggregation of reactions. ([\#15172](matrix-org/synapse#15172)) - Stabilise support for [MSC3758](matrix-org/matrix-spec-proposals#3758): `event_property_is` push condition. ([\#15185](matrix-org/synapse#15185)) Bugfixes -------- - Fix a bug introduced in Synapse 1.75 that caused experimental support for deleting account data to raise an internal server error while using an account data writer worker. ([\#14869](matrix-org/synapse#14869)) - Fix a long-standing bug where Synapse handled an unspecced field on push rules. ([\#15088](matrix-org/synapse#15088)) - Fix a long-standing bug where a URL preview would break if the discovered oEmbed failed to download. ([\#15092](matrix-org/synapse#15092)) - Fix a long-standing bug where an initial sync would not respond to changes to the list of ignored users if there was an initial sync cached. ([\#15163](matrix-org/synapse#15163)) - Add the `transaction_id` in the events included in many endpoints' responses. ([\#15174](matrix-org/synapse#15174)) - Fix a bug introduced in Synapse 1.78.0 where requests to claim dehydrated devices would fail with a `405` error. ([\#15180](matrix-org/synapse#15180)) - Stop applying edits when bundling aggregations, per [MSC3925](matrix-org/matrix-spec-proposals#3925). ([\#15193](matrix-org/synapse#15193)) - Fix a long-standing bug where the user directory search was not case-insensitive for accented characters. ([\#15143](matrix-org/synapse#15143)) Updates to the Docker image --------------------------- - Improve startup logging in the with-workers Docker image. ([\#15186](matrix-org/synapse#15186)) Improved Documentation ---------------------- - Document how to use caches in a module. ([\#14026](matrix-org/synapse#14026)) - Clarify which worker processes the ThirdPartyRules' [`on_new_event`](https://matrix-org.github.io/synapse/v1.78/modules/third_party_rules_callbacks.html#on_new_event) module API callback runs on. ([\#15071](matrix-org/synapse#15071)) - Document using [Shibboleth](https://www.shibboleth.net/) as an OpenID Provider. ([\#15112](matrix-org/synapse#15112)) - Correct reference to `federation_verify_certificates` in configuration documentation. ([\#15139](matrix-org/synapse#15139)) - Correct small documentation errors in some `MatrixFederationHttpClient` methods. ([\#15148](matrix-org/synapse#15148)) - Correct the description of the behavior of `registration_shared_secret_path` on startup. ([\#15168](matrix-org/synapse#15168)) Deprecations and Removals ------------------------- - Deprecate the `on_threepid_bind` module callback, to be replaced by [`on_add_user_third_party_identifier`](https://matrix-org.github.io/synapse/v1.79/modules/third_party_rules_callbacks.html#on_add_user_third_party_identifier). See [upgrade notes](https://github.com/matrix-org/synapse/blob/release-v1.79/docs/upgrade.md#upgrading-to-v1790). ([\#15044](matrix-org/synapse#15044)) - Remove the unspecced `room_alias` field from the [`/createRoom`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3createroom) response. ([\#15093](matrix-org/synapse#15093)) - Remove the unspecced `PUT` on the `/knock/{roomIdOrAlias}` endpoint. ([\#15189](matrix-org/synapse#15189)) - Remove the undocumented and unspecced `type` parameter to the `/thumbnail` endpoint. ([\#15137](matrix-org/synapse#15137)) - Remove unspecced and buggy `PUT` method on the unstable `/rooms/<room_id>/batch_send` endpoint. ([\#15199](matrix-org/synapse#15199)) Internal Changes ---------------- - Run the integration test suites with the asyncio reactor enabled in CI. ([\#14101](matrix-org/synapse#14101)) - Batch up storing state groups when creating a new room. ([\#14918](matrix-org/synapse#14918)) - Update [MSC3952](matrix-org/matrix-spec-proposals#3952) support based on changes to the MSC. ([\#15051](matrix-org/synapse#15051)) - Refactor writing json data in `FileExfiltrationWriter`. ([\#15095](matrix-org/synapse#15095)) - Tighten the login ratelimit defaults. ([\#15135](matrix-org/synapse#15135)) - Fix a typo in an experimental config setting. ([\#15138](matrix-org/synapse#15138)) - Refactor the media modules. ([\#15146](matrix-org/synapse#15146), [\#15175](matrix-org/synapse#15175)) - Improve type hints. ([\#15164](matrix-org/synapse#15164)) - Move `get_event_report` and `get_event_reports_paginate` from `RoomStore` to `RoomWorkerStore`. ([\#15165](matrix-org/synapse#15165)) - Remove dangling reference to being a reference implementation in docstring. ([\#15167](matrix-org/synapse#15167)) - Add an option to force a rebuild of the "editable" complement image. ([\#15184](matrix-org/synapse#15184)) - Use nightly rustfmt in CI. ([\#15188](matrix-org/synapse#15188)) - Add a `get_next_txn` method to `StreamIdGenerator` to match `MultiWriterIdGenerator`. ([\#15191](matrix-org/synapse#15191)) - Combine `AbstractStreamIdTracker` and `AbstractStreamIdGenerator`. ([\#15192](matrix-org/synapse#15192)) - Automatically fix errors with `ruff`. ([\#15194](matrix-org/synapse#15194)) - Refactor database transaction for query users' devices to reduce database pool contention. ([\#15215](matrix-org/synapse#15215)) - Correct `test_icu_word_boundary_punctuation` so that it passes with the ICU versions available in Alpine and macOS. ([\#15177](matrix-org/synapse#15177)) <details><summary>Locked dependency updates</summary> - Bump actions/checkout from 2 to 3. ([\#15155](matrix-org/synapse#15155)) - Bump black from 22.12.0 to 23.1.0. ([\#15103](matrix-org/synapse#15103)) - Bump dawidd6/action-download-artifact from 2.25.0 to 2.26.0. ([\#15152](matrix-org/synapse#15152)) - Bump docker/login-action from 1 to 2. ([\#15154](matrix-org/synapse#15154)) - Bump matrix-org/backend-meta from 1 to 2. ([\#15156](matrix-org/synapse#15156)) - Bump ruff from 0.0.237 to 0.0.252. ([\#15159](matrix-org/synapse#15159)) - Bump serde_json from 1.0.93 to 1.0.94. ([\#15214](matrix-org/synapse#15214)) - Bump types-commonmark from 0.9.2.1 to 0.9.2.2. ([\#15209](matrix-org/synapse#15209)) - Bump types-opentracing from 2.4.10.1 to 2.4.10.3. ([\#15158](matrix-org/synapse#15158)) - Bump types-pillow from 9.4.0.13 to 9.4.0.17. ([\#15211](matrix-org/synapse#15211)) - Bump types-psycopg2 from 2.9.21.4 to 2.9.21.8. ([\#15210](matrix-org/synapse#15210)) - Bump types-pyopenssl from 22.1.0.2 to 23.0.0.4. ([\#15213](matrix-org/synapse#15213)) - Bump types-setuptools from 67.3.0.1 to 67.4.0.3. ([\#15160](matrix-org/synapse#15160)) - Bump types-setuptools from 67.4.0.3 to 67.5.0.0. ([\#15212](matrix-org/synapse#15212)) - Bump typing-extensions from 4.4.0 to 4.5.0. ([\#15157](matrix-org/synapse#15157)) </details>

Synapse 1.80.0 (2023-03-28) =========================== No significant changes since 1.80.0rc2. Synapse 1.80.0rc2 (2023-03-22) ============================== Bugfixes -------- - Fix a bug in which the [`POST /_matrix/client/v3/rooms/{roomId}/report/{eventId}`](https://spec.matrix.org/v1.6/client-server-api/#post_matrixclientv3roomsroomidreporteventid) endpoint would return the wrong error if the user did not have permission to view the event. This aligns Synapse's implementation with [MSC2249](matrix-org/matrix-spec-proposals#2249). ([\matrix-org#15298](matrix-org#15298), [\matrix-org#15300](matrix-org#15300)) - Fix a bug introduced in Synapse 1.75.0rc1 where the [SQLite port_db script](https://matrix-org.github.io/synapse/latest/postgres.html#porting-from-sqlite) would fail to open the SQLite database. ([\matrix-org#15301](matrix-org#15301)) Synapse 1.80.0rc1 (2023-03-21) ============================== Features -------- - Stabilise support for [MSC3966](matrix-org/matrix-spec-proposals#3966): `event_property_contains` push condition. ([\matrix-org#15187](matrix-org#15187)) - Implement [MSC2659](matrix-org/matrix-spec-proposals#2659): application service ping endpoint. Contributed by Tulir @ Beeper. ([\matrix-org#15249](matrix-org#15249)) - Allow loading `/register/available` endpoint on workers. ([\matrix-org#15268](matrix-org#15268)) - Improve performance of creating and authenticating events. ([\matrix-org#15195](matrix-org#15195)) - Add topic and name events to group of events that are batch persisted when creating a room. ([\matrix-org#15229](matrix-org#15229)) Bugfixes -------- - Fix a long-standing bug in which the user directory would assume any remote membership state events represent a profile change. ([\matrix-org#14755](matrix-org#14755), [\matrix-org#14756](matrix-org#14756)) - Implement [MSC3873](matrix-org/matrix-spec-proposals#3873) to fix a long-standing bug where properties with dots were handled ambiguously in push rules. ([\matrix-org#15190](matrix-org#15190)) - Faster joins: Fix a bug introduced in Synapse 1.66 where spurious "Failed to find memberships ..." errors would be logged. ([\matrix-org#15232](matrix-org#15232)) - Fix a long-standing error when sending message into deleted room. ([\matrix-org#15235](matrix-org#15235)) Updates to the Docker image --------------------------- - Ensure the Dockerfile builds on platforms that don't have a `cryptography` wheel. ([\matrix-org#15239](matrix-org#15239)) - Mirror images to the GitHub Container Registry (`ghcr.io/matrix-org/synapse`). ([\matrix-org#15281](matrix-org#15281), [\matrix-org#15282](matrix-org#15282)) Improved Documentation ---------------------- - Add a missing endpoint to the workers documentation. ([\matrix-org#15223](matrix-org#15223)) Internal Changes ---------------- - Add additional functionality to declaring worker types when starting Complement in worker mode. ([\matrix-org#14921](matrix-org#14921)) - Add `Synapse-Trace-Id` to `access-control-expose-headers` header. ([\matrix-org#14974](matrix-org#14974)) - Make the `HttpTransactionCache` use the `Requester` in addition of the just the `Request` to build the transaction key. ([\matrix-org#15200](matrix-org#15200)) - Improve log lines when purging rooms. ([\matrix-org#15222](matrix-org#15222)) - Improve type hints. ([\matrix-org#15230](matrix-org#15230), [\matrix-org#15231](matrix-org#15231), [\matrix-org#15238](matrix-org#15238)) - Move various module API callback registration methods to a dedicated class. ([\matrix-org#15237](matrix-org#15237)) - Configure GitHub Actions for merge queues. ([\matrix-org#15244](matrix-org#15244)) - Add schema comments about the `destinations` and `destination_rooms` tables. ([\matrix-org#15247](matrix-org#15247)) - Skip processing of auto-join room behaviour if there are no auto-join rooms configured. ([\matrix-org#15262](matrix-org#15262)) - Remove unused store method `_set_destination_retry_timings_emulated`. ([\matrix-org#15266](matrix-org#15266)) - Reorganize URL preview code. ([\matrix-org#15269](matrix-org#15269)) - Clean-up direct TCP replication code. ([\matrix-org#15272](matrix-org#15272), [\matrix-org#15274](matrix-org#15274)) - Make `configure_workers_and_start` script used in Complement tests compatible with older versions of Python. ([\matrix-org#15275](matrix-org#15275)) - Add a `/versions` flag for [MSC3952](matrix-org/matrix-spec-proposals#3952). ([\matrix-org#15293](matrix-org#15293)) - Bump hiredis from 2.2.1 to 2.2.2. ([\matrix-org#15252](matrix-org#15252)) - Bump serde from 1.0.152 to 1.0.155. ([\matrix-org#15253](matrix-org#15253)) - Bump pysaml2 from 7.2.1 to 7.3.1. ([\matrix-org#15254](matrix-org#15254)) - Bump msgpack from 1.0.4 to 1.0.5. ([\matrix-org#15255](matrix-org#15255)) - Bump gitpython from 3.1.30 to 3.1.31. ([\matrix-org#15256](matrix-org#15256)) - Bump cryptography from 39.0.1 to 39.0.2. ([\matrix-org#15257](matrix-org#15257)) - Bump pydantic from 1.10.4 to 1.10.6. ([\matrix-org#15286](matrix-org#15286)) - Bump serde from 1.0.155 to 1.0.157. ([\matrix-org#15287](matrix-org#15287)) - Bump anyhow from 1.0.69 to 1.0.70. ([\matrix-org#15288](matrix-org#15288)) - Bump txredisapi from 1.4.7 to 1.4.9. ([\matrix-org#15289](matrix-org#15289)) - Bump pygithub from 1.57 to 1.58.1. ([\matrix-org#15290](matrix-org#15290)) - Bump types-requests from 2.28.11.12 to 2.28.11.15. ([\matrix-org#15291](matrix-org#15291))

H-Shay added 4 commits March 2, 2023 12:39

don't try and fetch auth events if we already have them (they're not …

dd1d796

…in db anyway)

add a state_map_before_event field to EventContext

05e7bdb

copy dict rather than mutating it

423f13c

newsfragement

81df09b

H-Shay requested a review from a team as a code owner March 2, 2023 20:51

H-Shay changed the title ~~More speedups to creating batched events~~ More speedups/fixes to creating batched events Mar 2, 2023

clokep reviewed Mar 2, 2023

View reviewed changes

synapse/event_auth.py Outdated Show resolved Hide resolved

clokep reviewed Mar 2, 2023

View reviewed changes

synapse/handlers/event_auth.py Outdated Show resolved Hide resolved

requested cahnges

db30a9f

clokep reviewed Mar 3, 2023

View reviewed changes

synapse/event_auth.py Show resolved Hide resolved

clokep reviewed Mar 3, 2023

View reviewed changes

synapse/events/snapshot.py Outdated Show resolved Hide resolved

clokep reviewed Mar 3, 2023

View reviewed changes

synapse/handlers/room.py Show resolved Hide resolved

clarifying comments

cfaf33f

H-Shay requested a review from clokep March 6, 2023 20:10

clokep suggested changes Mar 7, 2023

View reviewed changes

H-Shay added 2 commits March 7, 2023 09:27

requested changes

5fe44f0

improve changelog message

d3d9d73

clokep approved these changes Mar 7, 2023

View reviewed changes

H-Shay merged commit a368d30 into develop Mar 7, 2023

H-Shay deleted the shay/batching_fixes branch March 7, 2023 21:54

H-Shay self-assigned this Mar 9, 2023

ggogel mentioned this pull request Feb 27, 2024

Prevent locking up while processing batched_auth_events element-hq/synapse#16968

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

More speedups/fixes to creating batched events #15195

More speedups/fixes to creating batched events #15195

H-Shay commented Mar 2, 2023

clokep Mar 7, 2023

clokep left a comment

H-Shay commented Mar 7, 2023

	Misc speedups to creating and authing events.
	Improve performance of creating and authenticating events.

More speedups/fixes to creating batched events #15195

More speedups/fixes to creating batched events #15195

Conversation

H-Shay commented Mar 2, 2023

clokep Mar 7, 2023

Choose a reason for hiding this comment

clokep left a comment

Choose a reason for hiding this comment

H-Shay commented Mar 7, 2023