Merge commit '5cdca53aa' into dinsic

* commit '5cdca53aa':
  Merge different Resource implementation classes (#7732)
  Fix inconsistent handling of upper and lower cases of email addresses. (#7021)
  Allow YAML config file to contain None (#7779)
  Fix a typo.
  Move 1.15.2 after 1.16.0rc2.
  1.16.0rc2
  Remove an extraneous space.
  Add links to the fixes.
  Fix tense in the release notes.
  Hack to add push priority to push notifications (#7765)
  Add early returns to `_check_for_soft_fail` (#7769)
  Use symbolic names for replication stream names (#7768)
  Type checking for `FederationHandler` (#7770)
  Fix new metric where we used ms instead of seconds (#7771)
  Fix incorrect error message when database CTYPE was set incorrectly. (#7760)
  Pin link in CHANGES.md
  Fixes to CHANGES.md

CHANGES.md

@@ -1,31 +1,73 @@
+Synapse 1.16.0rc2 (2020-07-02)
+==============================
+
+Synapse 1.16.0rc2 includes the security fixes released with Synapse 1.15.2.
+Please see [below](#synapse-1152-2020-07-02) for more details.
+
+Improved Documentation
+----------------------
+
+- Update postgres image in example `docker-compose.yaml` to tag `12-alpine`. ([\#7696](https://github.com/matrix-org/synapse/issues/7696))
+
+
+Internal Changes
+----------------
+
+- Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7771](https://github.com/matrix-org/synapse/issues/7771))
+
+
+Synapse 1.15.2 (2020-07-02)
+===========================
+
+Due to the two security issues highlighted below, server administrators are
+encouraged to update Synapse. We are not aware of these vulnerabilities being
+exploited in the wild.
+
+Security advisory
+-----------------
+
+* A malicious homeserver could force Synapse to reset the state in a room to a
+  small subset of the correct state. This affects all Synapse deployments which
+  federate with untrusted servers. ([96e9afe6](https://github.com/matrix-org/synapse/commit/96e9afe62500310977dc3cbc99a8d16d3d2fa15c))
+* HTML pages served via Synapse were vulnerable to clickjacking attacks. This
+  predominantly affects homeservers with single-sign-on enabled, but all server
+  administrators are encouraged to upgrade. ([ea26e9a9](https://github.com/matrix-org/synapse/commit/ea26e9a98b0541fc886a1cb826a38352b7599dbe))
+
+  This was reported by [Quentin Gliech](https://sandhose.fr/).
+
+
 Synapse 1.16.0rc1 (2020-07-01)
 ==============================
 
+Note that this release deprecates the `m.login.jwt` login method, renaming it
+to `org.matrix.login.jwt`, as `m.login.jwt` is not part of the Matrix spec.
+Otherwise the behaviour is identical. Synapse will accept both names for now,
+but this may change in a future release.
+
 Features
 --------
 
 - Add an option to enable encryption by default for new rooms. ([\#7639](https://github.com/matrix-org/synapse/issues/7639))
-- Add support for running multiple media repository workers. See [docs/workers.md](docs/workers.md) for instructions. ([\#7706](https://github.com/matrix-org/synapse/issues/7706))
+- Add support for running multiple media repository workers. See [docs/workers.md](https://github.com/matrix-org/synapse/blob/release-v1.16.0/docs/workers.md) for instructions. ([\#7706](https://github.com/matrix-org/synapse/issues/7706))
 - Media can now be marked as safe from quarantined. ([\#7718](https://github.com/matrix-org/synapse/issues/7718))
-- Add unread messages count to sync responses. ([\#7761](https://github.com/matrix-org/synapse/issues/7761))
 - Expand the configuration options for auto-join rooms. ([\#7763](https://github.com/matrix-org/synapse/issues/7763))
 
 
 Bugfixes
 --------
 
 - Remove `user_id` from the response to `GET /_matrix/client/r0/presence/{userId}/status` to match the specification. ([\#7606](https://github.com/matrix-org/synapse/issues/7606))
-- In working mode, ensure that replicated data has not already been received. ([\#7648](https://github.com/matrix-org/synapse/issues/7648))
+- In worker mode, ensure that replicated data has not already been received. ([\#7648](https://github.com/matrix-org/synapse/issues/7648))
 - Fix intermittent exception during startup, introduced in Synapse 1.14.0. ([\#7663](https://github.com/matrix-org/synapse/issues/7663))
 - Include a user-agent for federation and well-known requests. ([\#7677](https://github.com/matrix-org/synapse/issues/7677))
-- Accept the proper field (`phone`) for the `m.id.phone` identifier type. The legacy field of `number` is still accepted as a fallback. Bug introduced in v0.20.0-rc1. ([\#7687](https://github.com/matrix-org/synapse/issues/7687))
-- Fix "Starting db txn 'get_completed_ui_auth_stages' from sentinel context" warning. The bug was introduced in 1.13.0rc1. ([\#7688](https://github.com/matrix-org/synapse/issues/7688))
-- Compare the URI and method during user interactive authentication (instead of the URI twice). Bug introduced in 1.13.0rc1. ([\#7689](https://github.com/matrix-org/synapse/issues/7689))
+- Accept the proper field (`phone`) for the `m.id.phone` identifier type. The legacy field of `number` is still accepted as a fallback. Bug introduced in v0.20.0. ([\#7687](https://github.com/matrix-org/synapse/issues/7687))
+- Fix "Starting db txn 'get_completed_ui_auth_stages' from sentinel context" warning. The bug was introduced in 1.13.0. ([\#7688](https://github.com/matrix-org/synapse/issues/7688))
+- Compare the URI and method during user interactive authentication (instead of the URI twice). Bug introduced in 1.13.0. ([\#7689](https://github.com/matrix-org/synapse/issues/7689))
 - Fix a long standing bug where the response to the `GET room_keys/version` endpoint had the incorrect type for the `etag` field. ([\#7691](https://github.com/matrix-org/synapse/issues/7691))
 - Fix logged error during device resync in opentracing. Broke in v1.14.0. ([\#7698](https://github.com/matrix-org/synapse/issues/7698))
 - Do not break push rule evaluation when receiving an event with a non-string body. This is a long-standing bug. ([\#7701](https://github.com/matrix-org/synapse/issues/7701))
 - Fixs a long standing bug which resulted in an exception: "TypeError: argument of type 'ObservableDeferred' is not iterable". ([\#7708](https://github.com/matrix-org/synapse/issues/7708))
-- The `synapse_port_db` script no longer fails when the `ui_auth_sessions` table is non-empty. This bug has existed since v1.13.0rc1. ([\#7711](https://github.com/matrix-org/synapse/issues/7711))
+- The `synapse_port_db` script no longer fails when the `ui_auth_sessions` table is non-empty. This bug has existed since v1.13.0. ([\#7711](https://github.com/matrix-org/synapse/issues/7711))
 - Synapse will now fetch media from the proper specified URL (using the r0 prefix instead of the unspecified v1). ([\#7714](https://github.com/matrix-org/synapse/issues/7714))
 - Fix the tables ignored by `synapse_port_db` to be in sync the current database schema. ([\#7717](https://github.com/matrix-org/synapse/issues/7717))
 - Fix missing `Content-Length` on HTTP responses from the metrics handler. ([\#7730](https://github.com/matrix-org/synapse/issues/7730))
@@ -51,7 +93,7 @@ Internal Changes
 
 - Refactor getting replication updates from database. ([\#7636](https://github.com/matrix-org/synapse/issues/7636))
 - Clean-up the login fallback code. ([\#7657](https://github.com/matrix-org/synapse/issues/7657))
-- Increase the default SAML session expirary time to 15 minutes. ([\#7664](https://github.com/matrix-org/synapse/issues/7664))
+- Increase the default SAML session expiry time to 15 minutes. ([\#7664](https://github.com/matrix-org/synapse/issues/7664))
 - Convert the device message and pagination handlers to async/await. ([\#7678](https://github.com/matrix-org/synapse/issues/7678))
 - Convert typing handler to async/await. ([\#7679](https://github.com/matrix-org/synapse/issues/7679))
 - Require `parameterized` package version to be at least 0.7.0. ([\#7680](https://github.com/matrix-org/synapse/issues/7680))
@@ -68,26 +110,6 @@ Internal Changes
 - Add some metrics for inbound and outbound federation latencies: `synapse_federation_server_pdu_process_time` and `synapse_event_processing_lag_by_event`. ([\#7755](https://github.com/matrix-org/synapse/issues/7755))
 
 
-Synapse 1.15.2 (2020-07-02)
-===========================
-
-Due to the two security issues highlight below, server administrators are
-encouraged to update Synapse. We are not aware of these vulnerabilities being
-exploited in the wild.
-
-Security advisory
------------------
-
-* A malicious homeserver could force Synapse to reset the state in a room to a
-  small subset of the correct state. This affects all Synapse deployments which
-  federate with untrusted servers.
-* HTML pages served via Synapse were vulnerable to clickjacking attacks. This
-  predominantly affects homeservers with single-sign-on enabled, but all server
-  administrators are encouraged to upgrade.
-
-  This was reported by [Quentin Gliech](https://sandhose.fr/).
-
-
 Synapse 1.15.1 (2020-06-16)
 ===========================
 

changelog.d/7021.bugfix

@@ -0,0 +1 @@
+Fix inconsistent handling of upper and lower case in email addresses when used as identifiers for login, etc. Contributed by @dklimpel.

changelog.d/7732.bugfix

@@ -0,0 +1 @@
+Fix "Tried to close a non-active scope!" error messages when opentracing is enabled.

changelog.d/7760.bugfix

@@ -0,0 +1 @@
+Fix incorrect error message when database CTYPE was set incorrectly.

changelog.d/7765.misc

@@ -0,0 +1 @@
+Send push notifications with a high or low priority depending upon whether they may generate user-observable effects.

changelog.d/7768.misc

@@ -0,0 +1 @@
+Use symbolic names for replication stream names.

changelog.d/7769.misc

@@ -0,0 +1 @@
+Add early returns to `_check_for_soft_fail`.

changelog.d/7770.misc

@@ -0,0 +1 @@
+Fix up `synapse.handlers.federation` to pass mypy.

changelog.d/7779.bugfix

@@ -0,0 +1 @@
+Fix synctl to handle empty config files correctly. Contributed by @kotovalexarian.

synapse/__init__.py

@@ -36,7 +36,7 @@
 except ImportError:
     pass
 
-__version__ = "1.15.2"
+__version__ = "1.16.0rc2"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/federation/sender/__init__.py

@@ -209,7 +209,7 @@ async def handle_event(event: EventBase) -> None:
 
                         synapse.metrics.event_processing_lag_by_event.labels(
                             "federation_sender"
-                        ).observe(now - ts)
+                        ).observe((now - ts) / 1000)
 
                 async def handle_room_events(events: Iterable[EventBase]) -> None:
                     with Measure(self.clock, "handle_room_events"):

synapse/federation/transport/server.py

@@ -362,11 +362,7 @@ def register(self, server):
                 continue
 
             server.register_paths(
-                method,
-                (pattern,),
-                self._wrap(code),
-                self.__class__.__name__,
-                trace=False,
+                method, (pattern,), self._wrap(code), self.__class__.__name__,
             )
 
 

synapse/handlers/appservice.py

@@ -118,7 +118,7 @@ def start_scheduler():
                         ts = yield self.store.get_received_ts(event.event_id)
                         synapse.metrics.event_processing_lag_by_event.labels(
                             "appservice_sender"
-                        ).observe(now - ts)
+                        ).observe((now - ts) / 1000)
 
                     @defer.inlineCallbacks
                     def handle_room_events(events):

synapse/handlers/auth.py

@@ -45,6 +45,7 @@
 from synapse.module_api import ModuleApi
 from synapse.push.mailer import load_jinja2_templates
 from synapse.types import Requester, UserID
+from synapse.util.threepids import canonicalise_email
 
 from ._base import BaseHandler
 
@@ -928,7 +929,7 @@ async def add_threepid(
         # for the presence of an email address during password reset was
         # case sensitive).
         if medium == "email":
-            address = address.lower()
+            address = canonicalise_email(address)
 
         await self.store.user_add_threepid(
             user_id, medium, address, validated_at, self.hs.get_clock().time_msec()
@@ -956,7 +957,7 @@ async def delete_threepid(
 
         # 'Canonicalise' email addresses as per above
         if medium == "email":
-            address = address.lower()
+            address = canonicalise_email(address)
 
         identity_handler = self.hs.get_handlers().identity_handler
         result = await identity_handler.try_unbind_threepid(