Merge branch 'develop' of github.com:matrix-org/synapse into anoa/pas…

…sword_reset_confirmation

* 'develop' of github.com:matrix-org/synapse: (39 commits)
  Fix errors when updating the user directory with invalid data (#8223)
  Explain better what GDPR-erased means (#8189)
  Convert additional databases to async/await part 3 (#8201)
  Convert appservice code to async/await. (#8207)
  Rename `_get_e2e_device_keys_txn` (#8222)
  Convert additional databases to async/await (#8199)
  Convert the well known resolver to async (#8214)
  Convert additional databases to async/await part 2 (#8200)
  Make MultiWriterIDGenerator work for streams that use negative stream IDs (#8203)
  Do not install setuptools 50.0. (#8212)
  Move and rename `get_devices_with_keys_by_user` (#8204)
  Rename `get_e2e_device_keys` to better reflect its purpose (#8205)
  Add a comment about _LimitedHostnameResolver
  Convert ReadWriteLock to async/await. (#8202)
  Fix incorrect return signature
  Fix `wait_for_stream_position` for multiple waiters. (#8196)
  Convert `event_push_actions`, `registration`, and `roommember` datastores to async (#8197)
  Only return devices with keys from `/federation/v1/user/devices/` (#8198)
  Convert state and stream stores and related code to async (#8194)
  Ensure that the OpenID Connect remote ID is a string. (#8190)
  ...

CHANGES.md

@@ -12,6 +12,12 @@ from Synapse as most users have updated their client. Further context can be
 found at [\#6766](https://github.com/matrix-org/synapse/issues/6766).
 
 
+Synapse 1.19.1 (2020-08-27)
+===========================
+
+No significant changes.
+
+
 Synapse 1.19.1rc1 (2020-08-25)
 ==============================
 

INSTALL.md

@@ -73,7 +73,7 @@ mkdir -p ~/synapse
 virtualenv -p python3 ~/synapse/env
 source ~/synapse/env/bin/activate
 pip install --upgrade pip
-pip install --upgrade setuptools
+pip install --upgrade setuptools!=50.0  # setuptools==50.0 fails on some older Python versions
 pip install matrix-synapse
 ```
 

changelog.d/7757.misc

@@ -0,0 +1 @@
+Reduce run times of some unit tests by advancing the reactor a fewer number of times.

changelog.d/8144.docker

@@ -0,0 +1 @@
+Fix builds of the Docker image on non-x86 platforms.

changelog.d/8166.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8168.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8173.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8174.misc

@@ -0,0 +1 @@
+Remove unused `is_guest` parameter from, and add safeguard to, `MessageHandler.get_room_data`.

changelog.d/8175.misc

@@ -0,0 +1 @@
+Standardize the mypy configuration.

changelog.d/8176.feature

@@ -0,0 +1 @@
+Add support for shadow-banning users (ignoring any message send requests).

changelog.d/8179.misc

@@ -0,0 +1 @@
+Add functions to `MultiWriterIdGen` used by events stream.

changelog.d/8181.misc

@@ -0,0 +1 @@
+Remove unused `is_guest` parameter from, and add safeguard to, `MessageHandler.get_room_data`.

changelog.d/8182.misc

@@ -0,0 +1 @@
+Refactor some of `LoginRestServlet`'s helper methods, and move them to `AuthHandler` for easier reuse.

changelog.d/8183.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.state`.

changelog.d/8187.misc

@@ -0,0 +1 @@
+Add type hints to `synapse.storage.database`.

changelog.d/8189.doc

@@ -0,0 +1 @@
+Explain better what GDPR-erased means when deactivating a user.

changelog.d/8190.bugfix

@@ -0,0 +1 @@
+Fix logging in via OpenID Connect with a provider that uses integer user IDs.

changelog.d/8191.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8192.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8193.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8194.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8195.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8196.misc

@@ -0,0 +1 @@
+Fix `wait_for_stream_position` to allow multiple waiters on same stream ID.

changelog.d/8197.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8198.feature

@@ -0,0 +1 @@
+Optimise `/federation/v1/user/devices/` API by only returning devices with encryption keys.

changelog.d/8199.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8200.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8201.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8202.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8203.misc

@@ -0,0 +1 @@
+Make `MultiWriterIDGenerator` work for streams that use negative values.

changelog.d/8204.misc

@@ -0,0 +1 @@
+Refactor queries for device keys and cross-signatures.

changelog.d/8205.misc

@@ -0,0 +1 @@
+ Refactor queries for device keys and cross-signatures.

changelog.d/8207.misc

@@ -0,0 +1 @@
+Convert various parts of the codebase to async/await.

changelog.d/8212.bugfix

@@ -0,0 +1 @@
+Do not install setuptools 50.0. It can lead to a broken configuration on some older Python versions.

changelog.d/8214.misc

@@ -0,0 +1 @@
+ Convert various parts of the codebase to async/await.

changelog.d/8222.misc

@@ -0,0 +1 @@
+Refactor queries for device keys and cross-signatures.

changelog.d/8223.bugfix

@@ -0,0 +1 @@
+Fixes a longstanding bug where user directory updates could break when unexpected profile data was included in events.

debian/changelog

@@ -1,3 +1,15 @@
+matrix-synapse-py3 (1.19.0ubuntu1) UNRELEASED; urgency=medium
+
+  * Use Type=notify in systemd service
+
+ -- Dexter Chua <dec41@srcf.net>  Wed, 26 Aug 2020 12:41:36 +0000
+
+matrix-synapse-py3 (1.19.1) stable; urgency=medium
+
+  * New synapse release 1.19.1.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 27 Aug 2020 10:50:19 +0100
+
 matrix-synapse-py3 (1.19.0) stable; urgency=medium
 
   [ Synapse Packaging team ]

debian/matrix-synapse.service

@@ -2,7 +2,7 @@
 Description=Synapse Matrix homeserver
 
 [Service]
-Type=simple
+Type=notify
 User=matrix-synapse
 WorkingDirectory=/var/lib/matrix-synapse
 EnvironmentFile=/etc/default/matrix-synapse

docker/Dockerfile

@@ -19,11 +19,16 @@ ARG PYTHON_VERSION=3.7
 FROM docker.io/python:${PYTHON_VERSION}-slim as builder
 
 # install the OS build deps
-
-
 RUN apt-get update && apt-get install -y \
     build-essential \
+    libffi-dev \
+    libjpeg-dev \
     libpq-dev \
+    libssl-dev \
+    libwebp-dev \
+    libxml++2.6-dev \
+    libxslt1-dev \
+    zlib1g-dev \
  && rm -rf /var/lib/apt/lists/*
 
 # Build dependencies that are not available as wheels, to speed up rebuilds
@@ -56,9 +61,11 @@ FROM docker.io/python:${PYTHON_VERSION}-slim
 
 RUN apt-get update && apt-get install -y \
     curl \
+    gosu \
+    libjpeg62-turbo \
     libpq5 \
+    libwebp6 \
     xmlsec1 \
-    gosu \
  && rm -rf /var/lib/apt/lists/*
 
 COPY --from=builder /install /usr/local

docs/admin_api/user_admin_api.rst

@@ -214,9 +214,11 @@ Deactivate Account
 
 This API deactivates an account. It removes active access tokens, resets the
 password, and deletes third-party IDs (to prevent the user requesting a
-password reset). It can also mark the user as GDPR-erased (stopping their data
-from distributed further, and deleting it entirely if there are no other
-references to it).
+password reset).
+
+It can also mark the user as GDPR-erased. This means messages sent by the
+user will still be visible by anyone that was in the room when these messages
+were sent, but hidden from users joining the room afterwards.
 
 The api is::
 

mypy.ini

@@ -6,6 +6,56 @@ check_untyped_defs = True
 show_error_codes = True
 show_traceback = True
 mypy_path = stubs
+files =
+  synapse/api,
+  synapse/appservice,
+  synapse/config,
+  synapse/event_auth.py,
+  synapse/events/builder.py,
+  synapse/events/spamcheck.py,
+  synapse/federation,
+  synapse/handlers/auth.py,
+  synapse/handlers/cas_handler.py,
+  synapse/handlers/directory.py,
+  synapse/handlers/federation.py,
+  synapse/handlers/identity.py,
+  synapse/handlers/message.py,
+  synapse/handlers/oidc_handler.py,
+  synapse/handlers/presence.py,
+  synapse/handlers/room.py,
+  synapse/handlers/room_member.py,
+  synapse/handlers/room_member_worker.py,
+  synapse/handlers/saml_handler.py,
+  synapse/handlers/sync.py,
+  synapse/handlers/ui_auth,
+  synapse/http/federation/well_known_resolver.py,
+  synapse/http/server.py,
+  synapse/http/site.py,
+  synapse/logging/,
+  synapse/metrics,
+  synapse/module_api,
+  synapse/notifier.py,
+  synapse/push/pusherpool.py,
+  synapse/push/push_rule_evaluator.py,
+  synapse/replication,
+  synapse/rest,
+  synapse/server.py,
+  synapse/server_notices,
+  synapse/spam_checker_api,
+  synapse/state,
+  synapse/storage/databases/main/ui_auth.py,
+  synapse/storage/database.py,
+  synapse/storage/engines,
+  synapse/storage/state.py,
+  synapse/storage/util,
+  synapse/streams,
+  synapse/types.py,
+  synapse/util/caches/stream_change_cache.py,
+  synapse/util/metrics.py,
+  tests/replication,
+  tests/test_utils,
+  tests/rest/client/v2_alpha/test_auth.py,
+  tests/util/test_stream_change_cache.py
 
 [mypy-pymacaroons.*]
 ignore_missing_imports = True

synapse/__init__.py

@@ -48,7 +48,7 @@
 except ImportError:
     pass
 
-__version__ = "1.19.1rc1"
+__version__ = "1.19.1"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/app/_base.py

@@ -334,6 +334,13 @@ def install_dns_limiter(reactor, max_dns_requests_in_flight=100):
     This is to workaround https://twistedmatrix.com/trac/ticket/9620, where we
     can run out of file descriptors and infinite loop if we attempt to do too
     many DNS queries at once
+
+    XXX: I'm confused by this. reactor.nameResolver does not use twisted.names unless
+    you explicitly install twisted.names as the resolver; rather it uses a GAIResolver
+    backed by the reactor's default threadpool (which is limited to 10 threads). So
+    (a) I don't understand why twisted ticket 9620 is relevant, and (b) I don't
+    understand why we would run out of FDs if we did too many lookups at once.
+    -- richvdh 2020/08/29
     """
     new_resolver = _LimitedHostnameResolver(
         reactor.nameResolver, max_dns_requests_in_flight

synapse/appservice/__init__.py

@@ -14,11 +14,16 @@
 # limitations under the License.
 import logging
 import re
+from typing import TYPE_CHECKING
 
 from synapse.api.constants import EventTypes
+from synapse.appservice.api import ApplicationServiceApi
 from synapse.types import GroupID, get_domain_from_id
 from synapse.util.caches.descriptors import cached
 
+if TYPE_CHECKING:
+    from synapse.storage.databases.main import DataStore
+
 logger = logging.getLogger(__name__)
 
 
@@ -35,30 +40,28 @@ def __init__(self, service, id, events):
         self.id = id
         self.events = events
 
-    def send(self, as_api):
+    async def send(self, as_api: ApplicationServiceApi) -> bool:
         """Sends this transaction using the provided AS API interface.
 
         Args:
-            as_api(ApplicationServiceApi): The API to use to send.
+            as_api: The API to use to send.
         Returns:
-            An Awaitable which resolves to True if the transaction was sent.
+            True if the transaction was sent.
         """
-        return as_api.push_bulk(
+        return await as_api.push_bulk(
             service=self.service, events=self.events, txn_id=self.id
         )
 
-    def complete(self, store):
+    async def complete(self, store: "DataStore") -> None:
         """Completes this transaction as successful.
 
         Marks this transaction ID on the application service and removes the
         transaction contents from the database.
 
         Args:
             store: The database store to operate on.
-        Returns:
-            A Deferred which resolves to True if the transaction was completed.
         """
-        return store.complete_appservice_txn(service=self.service, txn_id=self.id)
+        await store.complete_appservice_txn(service=self.service, txn_id=self.id)
 
 
 class ApplicationService(object):

synapse/appservice/api.py

@@ -14,18 +14,20 @@
 # limitations under the License.
 import logging
 import urllib
+from typing import TYPE_CHECKING, Optional
 
 from prometheus_client import Counter
 
-from twisted.internet import defer
-
 from synapse.api.constants import EventTypes, ThirdPartyEntityKind
 from synapse.api.errors import CodeMessageException
 from synapse.events.utils import serialize_event
 from synapse.http.client import SimpleHttpClient
-from synapse.types import ThirdPartyInstanceID
+from synapse.types import JsonDict, ThirdPartyInstanceID
 from synapse.util.caches.response_cache import ResponseCache
 
+if TYPE_CHECKING:
+    from synapse.appservice import ApplicationService
+
 logger = logging.getLogger(__name__)
 
 sent_transactions_counter = Counter(
@@ -163,19 +165,20 @@ async def query_3pe(self, service, kind, protocol, fields):
             logger.warning("query_3pe to %s threw exception %s", uri, ex)
             return []
 
-    def get_3pe_protocol(self, service, protocol):
+    async def get_3pe_protocol(
+        self, service: "ApplicationService", protocol: str
+    ) -> Optional[JsonDict]:
         if service.url is None:
             return {}
 
-        @defer.inlineCallbacks
-        def _get():
+        async def _get() -> Optional[JsonDict]:
             uri = "%s%s/thirdparty/protocol/%s" % (
                 service.url,
                 APP_SERVICE_PREFIX,
                 urllib.parse.quote(protocol),
             )
             try:
-                info = yield defer.ensureDeferred(self.get_json(uri, {}))
+                info = await self.get_json(uri, {})
 
                 if not _is_valid_3pe_metadata(info):
                     logger.warning(
@@ -196,7 +199,7 @@ def _get():
                 return None
 
         key = (service.id, protocol)
-        return self.protocol_meta_cache.wrap(key, _get)
+        return await self.protocol_meta_cache.wrap(key, _get)
 
     async def push_bulk(self, service, events, txn_id=None):
         if service.url is None: