Add presence federation stream #9819

erikjohnston · 2021-04-15T11:03:18Z

This is to allow moving presence off master. It turned out that making the federation send queue stuff work on arbitrary workers was fiddly (since not all workers have HTTP replication listeners), and so we replace it with a custom stream just for sending ad hoc presence states over federation.

The new stream is only used for sending out federation pings and when a new remote host joins a room. Normal presence updates are replicated via the existing presence stream. We just stream which user's presence we should send to which destinations, rather than the full presence states (as the remotes should already have that data via the presence stream).

Currently the presence federation stream always stores the last 5 minutes of data in memory, so that other workers can query recent data if they restart. The idea here is that a) this is just ephemeral data anyway, so safe to drop and b) there shouldn't be that much data going through anyway. It shouldn't be hard to add back in a periodic ACK to allow us to clear out the data.

The federation send queue is now unused, and a future PR will remove it (I can add it to this PR if that is preferable, I'm just aware that its already large).

Reviewable commit by commit.

Update: This is now based on #9828

synapse/federation/send_queue.py

synapse/handlers/presence.py

erikjohnston · 2021-04-16T11:06:16Z

I've split out a bunch of stuff into #9828, which this PR is now based on.

ShadowJonathan · 2021-04-19T10:17:59Z

synapse/handlers/presence.py

+        self._presence_writer = hs.config.worker.worker_app is None
+
+        # The federation sender if this instance is a federation sender.
+        self._federation = None


Why is it called _federation if its a federation sender?

Small nitpick, resolve at will, I'm just pointing at the naming

Ah wait, i see, it is an interface to send to federation through

What is the actual type of this variable, though? Is there an interface type that describes this behaviour? (I'd like it to be annotated with it)

It's an Optional[FederationSender], aiui.

erikjohnston · 2021-04-19T10:52:16Z

synapse/handlers/presence.py

-                    self._federation.send_presence_to_destinations(states, destinations)
+                    self._federation_queue.send_presence_to_destinations(
+                        states, destinations
+                    )


@richvdh To continue the conversation from #9828 (comment), this is a similar shape as in maybe_send_presence_to_interested_destinations, except it uses the federation queue instead of the federation sender. I don't know whether it is worth trying to merge them, but I think that gets a bit convoluted?

erikjohnston · 2021-04-19T11:02:12Z

synapse/handlers/presence.py

@@ -127,10 +130,10 @@ def __init__(self, hs: "HomeServer"):
        self.state = hs.get_state_handler()

        self._federation = None
-        if hs.should_send_federation() or not hs.config.worker_app:
+        if hs.should_send_federation():
            self._federation = hs.get_federation_sender()


When we remove the federation send queue I plan on making hs.get_federation_sender() return an optional federation sender, which will simplify this even more.

by "federation send queue" you mean the FederationRemoteSendQueue ? Don't we need that for device list updates too?

I do mean that yes. The device list updates carefully call the queue, but the queue just no ops. I have a branch that rips the send queue out entirely and everything still works fine.

richvdh

seems... plausible.

synapse/handlers/presence.py

richvdh · 2021-04-19T14:42:58Z

synapse/handlers/presence.py

+        self._presence_writer = hs.config.worker.worker_app is None
+
+        # The federation sender if this instance is a federation sender.
+        self._federation = None


It's an Optional[FederationSender], aiui.

synapse/handlers/presence.py

synapse/replication/tcp/streams/_base.py

tests/handlers/test_presence.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

synapse/handlers/presence.py

richvdh

lgtm!

synapse/handlers/presence.py

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

…eration_stream

@rkfg

Synapse 1.33.0rc1 (2021-04-28) ============================== Features -------- - Update experimental support for [MSC3083](matrix-org/matrix-spec-proposals#3083): restricting room access via group membership. ([\#9800](#9800), [\#9814](#9814)) - Add experimental support for handling presence on a worker. ([\#9819](#9819), [\#9820](#9820), [\#9828](#9828), [\#9850](#9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](#9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](#9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](#9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](#9802)) - Limit the size of HTTP responses read over federation. ([\#9833](#9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](#9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](#9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](#9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](#9162)) - Apply `pyupgrade` across the codebase. ([\#9786](#9786)) - Move some replication processing out of `generic_worker`. ([\#9796](#9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](#9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](#9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](#9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](#9821)) - Small speed up for joining large remote rooms. ([\#9825](#9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](#9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](#9845)) - Limit length of accepted email addresses. ([\#9855](#9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](#9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](#9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](#9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](#9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](#9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](#9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](#9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](#9879)) - Small performance improvement around handling new local presence updates. ([\#9887](#9887))

Only affects workers. Introduced in #9819. Fixes #9899.

@rkfg

Synapse 1.33.0 (2021-05-05) =========================== Features -------- - Build Debian packages for Ubuntu 21.04 (Hirsute Hippo). ([\#9909](matrix-org/synapse#9909)) Synapse 1.33.0rc2 (2021-04-29) ============================== Bugfixes -------- - Fix tight loop when handling presence replication when using workers. Introduced in v1.33.0rc1. ([\#9900](matrix-org/synapse#9900)) Synapse 1.33.0rc1 (2021-04-28) ============================== Features -------- - Update experimental support for [MSC3083](matrix-org/matrix-spec-proposals#3083): restricting room access via group membership. ([\#9800](matrix-org/synapse#9800), [\#9814](matrix-org/synapse#9814)) - Add experimental support for handling presence on a worker. ([\#9819](matrix-org/synapse#9819), [\#9820](matrix-org/synapse#9820), [\#9828](matrix-org/synapse#9828), [\#9850](matrix-org/synapse#9850)) - Return a new template when an user attempts to renew their account multiple times with the same token, stating that their account is set to expire. This replaces the invalid token template that would previously be shown in this case. This change concerns the optional account validity feature. ([\#9832](matrix-org/synapse#9832)) Bugfixes -------- - Fixes the OIDC SSO flow when using a `public_baseurl` value including a non-root URL path. ([\#9726](matrix-org/synapse#9726)) - Fix thumbnail generation for some sites with non-standard content types. Contributed by @rkfg. ([\#9788](matrix-org/synapse#9788)) - Add some sanity checks to identity server passed to 3PID bind/unbind endpoints. ([\#9802](matrix-org/synapse#9802)) - Limit the size of HTTP responses read over federation. ([\#9833](matrix-org/synapse#9833)) - Fix a bug which could cause Synapse to get stuck in a loop of resyncing device lists. ([\#9867](matrix-org/synapse#9867)) - Fix a long-standing bug where errors from federation did not propagate to the client. ([\#9868](matrix-org/synapse#9868)) Improved Documentation ---------------------- - Add a note to the docker docs mentioning that we mirror upstream's supported Docker platforms. ([\#9801](matrix-org/synapse#9801)) Internal Changes ---------------- - Add a dockerfile for running Synapse in worker-mode under Complement. ([\#9162](matrix-org/synapse#9162)) - Apply `pyupgrade` across the codebase. ([\#9786](matrix-org/synapse#9786)) - Move some replication processing out of `generic_worker`. ([\#9796](matrix-org/synapse#9796)) - Replace `HomeServer.get_config()` with inline references. ([\#9815](matrix-org/synapse#9815)) - Rename some handlers and config modules to not duplicate the top-level module. ([\#9816](matrix-org/synapse#9816)) - Fix a long-standing bug which caused `max_upload_size` to not be correctly enforced. ([\#9817](matrix-org/synapse#9817)) - Reduce CPU usage of the user directory by reusing existing calculated room membership. ([\#9821](matrix-org/synapse#9821)) - Small speed up for joining large remote rooms. ([\#9825](matrix-org/synapse#9825)) - Introduce flake8-bugbear to the test suite and fix some of its lint violations. ([\#9838](matrix-org/synapse#9838)) - Only store the raw data in the in-memory caches, rather than objects that include references to e.g. the data stores. ([\#9845](matrix-org/synapse#9845)) - Limit length of accepted email addresses. ([\#9855](matrix-org/synapse#9855)) - Remove redundant `synapse.types.Collection` type definition. ([\#9856](matrix-org/synapse#9856)) - Handle recently added rate limits correctly when using `--no-rate-limit` with the demo scripts. ([\#9858](matrix-org/synapse#9858)) - Disable invite rate-limiting by default when running the unit tests. ([\#9871](matrix-org/synapse#9871)) - Pass a reactor into `SynapseSite` to make testing easier. ([\#9874](matrix-org/synapse#9874)) - Make `DomainSpecificString` an `attrs` class. ([\#9875](matrix-org/synapse#9875)) - Add type hints to `synapse.api.auth` and `synapse.api.auth_blocking` modules. ([\#9876](matrix-org/synapse#9876)) - Remove redundant `_PushHTTPChannel` test class. ([\#9878](matrix-org/synapse#9878)) - Remove backwards-compatibility code for Python versions < 3.6. ([\#9879](matrix-org/synapse#9879)) - Small performance improvement around handling new local presence updates. ([\#9887](matrix-org/synapse#9887))

erikjohnston force-pushed the erikj/presence_federation_stream branch from 346bdec to 5c63b65 Compare April 15, 2021 11:03

erikjohnston requested a review from a team April 15, 2021 11:08

erikjohnston mentioned this pull request Apr 15, 2021

Split presence out of master #9820

Merged

erikjohnston force-pushed the erikj/presence_federation_stream branch 2 times, most recently from 7dece83 to 59d6a01 Compare April 15, 2021 14:17

anoadragon453 mentioned this pull request Apr 15, 2021

Use a database table to hold the users that should have full presence sent to them, instead of something in-memory #9823

Merged

richvdh suggested changes Apr 15, 2021

View reviewed changes

synapse/federation/send_queue.py Outdated Show resolved Hide resolved

synapse/handlers/presence.py Outdated Show resolved Hide resolved

richvdh reviewed Apr 15, 2021

View reviewed changes

synapse/handlers/presence.py Outdated Show resolved Hide resolved

erikjohnston mentioned this pull request Apr 16, 2021

Don't send normal presence updates over federation replication stream #9828

Merged

erikjohnston force-pushed the erikj/presence_federation_stream branch from 59d6a01 to 79bdd8e Compare April 16, 2021 11:05

erikjohnston force-pushed the erikj/presence_federation_stream branch from 79bdd8e to bf93f68 Compare April 16, 2021 11:11

Pass all replication rows to presence handler

25cb535

erikjohnston force-pushed the erikj/presence_federation_stream branch 6 times, most recently from fa4253a to ebb157b Compare April 19, 2021 10:04

erikjohnston added 4 commits April 19, 2021 11:06

Add a presence federation replication stream

885c3d9

Newsfile

14b263d

Actually fix tests

63c8516

Clean up 'self._federation' var

ee9a5ce

erikjohnston force-pushed the erikj/presence_federation_stream branch from ebb157b to ee9a5ce Compare April 19, 2021 10:06

ShadowJonathan reviewed Apr 19, 2021

View reviewed changes

erikjohnston commented Apr 19, 2021

View reviewed changes

erikjohnston requested a review from richvdh April 19, 2021 11:01

erikjohnston commented Apr 19, 2021

View reviewed changes

richvdh reviewed Apr 19, 2021

View reviewed changes

Update tests/handlers/test_presence.py

28ba928

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

erikjohnston and others added 4 commits April 19, 2021 16:57

Update synapse/handlers/presence.py

2571eeb

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Update synapse/handlers/presence.py

e4ec78b

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Add some comments about how PresenceFederationQueue works on workers

d429733

Fix comment

a20a442

erikjohnston requested a review from richvdh April 19, 2021 16:48

richvdh reviewed Apr 19, 2021

View reviewed changes

synapse/handlers/presence.py Outdated Show resolved Hide resolved

synapse/handlers/presence.py Show resolved Hide resolved

richvdh approved these changes Apr 19, 2021

View reviewed changes

synapse/handlers/presence.py Outdated Show resolved Hide resolved

erikjohnston and others added 2 commits April 20, 2021 13:39

Apply suggestions from code review

091f5da

Co-authored-by: Richard van der Hoff <1389908+richvdh@users.noreply.github.com>

Merge remote-tracking branch 'origin/develop' into erikj/presence_fed…

e36cce5

…eration_stream

erikjohnston merged commit de0d088 into develop Apr 20, 2021

erikjohnston deleted the erikj/presence_federation_stream branch April 20, 2021 13:11

fredo mentioned this pull request Apr 26, 2021

Prepare RC for mainnet release raiden-network/raiden-service-bundle#265

Closed

6 tasks

erikjohnston added a commit that referenced this pull request Apr 28, 2021

Fix tight loop handling presence replication.

468831f

Only affects workers. Introduced in #9819. Fixes #9899.

erikjohnston mentioned this pull request Apr 28, 2021

Fix tight loop handling presence replication. #9900

Merged

erikjohnston added a commit that referenced this pull request Apr 28, 2021

Fix tight loop handling presence replication. (#9900)

e4ab867

Only affects workers. Introduced in #9819. Fixes #9899.

This was referenced Jun 11, 2021

Fix sending presence over federation when using workers #10163

Merged

Fix bug when running presence off master #10149

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add presence federation stream #9819

Add presence federation stream #9819

erikjohnston commented Apr 15, 2021 •

edited

Loading

erikjohnston commented Apr 16, 2021

ShadowJonathan Apr 19, 2021

ShadowJonathan Apr 19, 2021

richvdh Apr 19, 2021

erikjohnston Apr 19, 2021

erikjohnston Apr 19, 2021

richvdh Apr 19, 2021

erikjohnston Apr 19, 2021

richvdh left a comment

richvdh Apr 19, 2021

richvdh left a comment

Add presence federation stream #9819

Add presence federation stream #9819

Conversation

erikjohnston commented Apr 15, 2021 • edited Loading

erikjohnston commented Apr 16, 2021

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

richvdh left a comment

Choose a reason for hiding this comment

erikjohnston commented Apr 15, 2021 •

edited

Loading