Merge branch 'develop' into babolivier/mark_unread

CHANGES.md

@@ -1,3 +1,9 @@
+Synapse 1.15.0 (2020-06-11)
+===========================
+
+No significant changes.
+
+
 Synapse 1.15.0rc1 (2020-06-09)
 ==============================
 
@@ -8,14 +14,14 @@ Features
 - Add an option to disable autojoining rooms for guest accounts. ([\#6637](https://github.com/matrix-org/synapse/issues/6637))
 - For SAML authentication, add the ability to pass email addresses to be added to new users' accounts via SAML attributes. Contributed by Christopher Cooper. ([\#7385](https://github.com/matrix-org/synapse/issues/7385))
 - Add admin APIs to allow server admins to manage users' devices. Contributed by @dklimpel. ([\#7481](https://github.com/matrix-org/synapse/issues/7481))
-- Add support for generating thumbnails for WebP images. Previously, users would see an empty box instead of preview image. ([\#7586](https://github.com/matrix-org/synapse/issues/7586))
+- Add support for generating thumbnails for WebP images. Previously, users would see an empty box instead of preview image. Contributed by @WGH-. ([\#7586](https://github.com/matrix-org/synapse/issues/7586))
 - Support the standardized `m.login.sso` user-interactive authentication flow. ([\#7630](https://github.com/matrix-org/synapse/issues/7630))
 
 
 Bugfixes
 --------
 
-- Allow new users to be registered via the admin API even if the monthly active user limit has been reached. Contributed by @dkimpel. ([\#7263](https://github.com/matrix-org/synapse/issues/7263))
+- Allow new users to be registered via the admin API even if the monthly active user limit has been reached. Contributed by @dklimpel. ([\#7263](https://github.com/matrix-org/synapse/issues/7263))
 - Fix email notifications not being enabled for new users when created via the Admin API. ([\#7267](https://github.com/matrix-org/synapse/issues/7267))
 - Fix str placeholders in an instance of `PrepareDatabaseException`. Introduced in Synapse v1.8.0. ([\#7575](https://github.com/matrix-org/synapse/issues/7575))
 - Fix a bug in automatic user creation during first time login with `m.login.jwt`. Regression in v1.6.0. Contributed by @olof. ([\#7585](https://github.com/matrix-org/synapse/issues/7585))
@@ -44,7 +50,7 @@ Internal Changes
 ----------------
 
 - Convert the identity handler to async/await. ([\#7561](https://github.com/matrix-org/synapse/issues/7561))
-- Improve query performance for fetching state from a PostgreSQL database. ([\#7567](https://github.com/matrix-org/synapse/issues/7567))
+- Improve query performance for fetching state from a PostgreSQL database. Contributed by @ilmari. ([\#7567](https://github.com/matrix-org/synapse/issues/7567))
 - Speed up processing of federation stream RDATA rows. ([\#7584](https://github.com/matrix-org/synapse/issues/7584))
 - Add comment to systemd example to show postgresql dependency. ([\#7591](https://github.com/matrix-org/synapse/issues/7591))
 - Refactor `Ratelimiter` to limit the amount of expensive config value accesses. ([\#7595](https://github.com/matrix-org/synapse/issues/7595))

changelog.d/7606.bugfix

@@ -0,0 +1 @@
+Remove `user_id` from the response to `GET /_matrix/client/r0/presence/{userId}/status` to match the specification.

changelog.d/7639.feature

@@ -0,0 +1 @@
+Add an option to enable encryption by default for new rooms.

changelog.d/7652.doc

@@ -0,0 +1 @@
+Spelling correction in sample_config.yaml.

changelog.d/7657.misc

@@ -0,0 +1 @@
+Clean-up the login fallback code.

changelog.d/7663.bugfix

@@ -0,0 +1 @@
+Fix intermittent exception during startup, introduced in Synapse 1.14.0.

changelog.d/7664.misc

@@ -0,0 +1 @@
+Increase the default SAML session expirary time to 15 minutes.

debian/changelog

@@ -1,3 +1,9 @@
+matrix-synapse-py3 (1.15.0) stable; urgency=medium
+
+  * New synapse release 1.15.0.
+
+ -- Synapse Packaging team <packages@matrix.org>  Thu, 11 Jun 2020 13:27:06 +0100
+
 matrix-synapse-py3 (1.14.0) stable; urgency=medium
 
   * New synapse release 1.14.0.

docs/sample_config.yaml

@@ -283,7 +283,7 @@ listeners:
 # number of monthly active users.
 #
 # 'limit_usage_by_mau' disables/enables monthly active user blocking. When
-# anabled and a limit is reached the server returns a 'ResourceLimitError'
+# enabled and a limit is reached the server returns a 'ResourceLimitError'
 # with error type Codes.RESOURCE_LIMIT_EXCEEDED
 #
 # 'max_mau_value' is the hard limit of monthly active users above which
@@ -1454,7 +1454,7 @@ saml2_config:
 
   # The lifetime of a SAML session. This defines how long a user has to
   # complete the authentication process, if allow_unsolicited is unset.
-  # The default is 5 minutes.
+  # The default is 15 minutes.
   #
   #saml_session_lifetime: 5m
 
@@ -1973,6 +1973,26 @@ spam_checker:
    #    example_stop_events_from: ['@bad:example.com']
 
 
+## Rooms ##
+
+# Controls whether locally-created rooms should be end-to-end encrypted by
+# default.
+#
+# Possible options are "all", "invite", and "off". They are defined as:
+#
+# * "all": any locally-created room
+# * "invite": any room created with the "private_chat" or "trusted_private_chat"
+#             room creation presets
+# * "off": this option will take no effect
+#
+# The default value is "off".
+#
+# Note that this option will only affect rooms created after it is set. It
+# will also not affect rooms created by other servers.
+#
+#encryption_enabled_by_default_for_room_type: invite
+
+
 # Uncomment to allow non-server-admin users to create groups on this server
 #
 #enable_group_creation: true

synapse/__init__.py

@@ -36,7 +36,7 @@
 except ImportError:
     pass
 
-__version__ = "1.15.0rc1"
+__version__ = "1.15.0"
 
 if bool(os.environ.get("SYNAPSE_TEST_PATCH_LOG_CONTEXTS", False)):
     # We import here so that we don't have to install a bunch of deps when

synapse/api/constants.py

@@ -150,3 +150,8 @@ class EventContentFields(object):
     # Timestamp to delete the event after
     # cf https://github.com/matrix-org/matrix-doc/pull/2228
     SELF_DESTRUCT_AFTER = "org.matrix.self_destruct_after"
+
+
+class RoomEncryptionAlgorithms(object):
+    MEGOLM_V1_AES_SHA2 = "m.megolm.v1.aes-sha2"
+    DEFAULT = MEGOLM_V1_AES_SHA2

synapse/config/cache.py

@@ -15,6 +15,7 @@
 
 import os
 import re
+import threading
 from typing import Callable, Dict
 
 from ._base import Config, ConfigError
@@ -25,6 +26,9 @@
 # Map from canonicalised cache name to cache.
 _CACHES = {}
 
+# a lock on the contents of _CACHES
+_CACHES_LOCK = threading.Lock()
+
 _DEFAULT_FACTOR_SIZE = 0.5
 _DEFAULT_EVENT_CACHE_SIZE = "10K"
 
@@ -66,7 +70,10 @@ def add_resizable_cache(cache_name: str, cache_resize_callback: Callable):
     # Some caches have '*' in them which we strip out.
     cache_name = _canonicalise_cache_name(cache_name)
 
-    _CACHES[cache_name] = cache_resize_callback
+    # sometimes caches are initialised from background threads, so we need to make
+    # sure we don't conflict with another thread running a resize operation
+    with _CACHES_LOCK:
+        _CACHES[cache_name] = cache_resize_callback
 
     # Ensure all loaded caches are sized appropriately
     #
@@ -87,7 +94,8 @@ def reset():
             os.environ.get(_CACHE_PREFIX, _DEFAULT_FACTOR_SIZE)
         )
         properties.resize_all_caches_func = None
-        _CACHES.clear()
+        with _CACHES_LOCK:
+            _CACHES.clear()
 
     def generate_config_section(self, **kwargs):
         return """\
@@ -193,6 +201,8 @@ def resize_all_caches(self):
         For each cache, run the mapped callback function with either
         a specific cache factor or the default, global one.
         """
-        for cache_name, callback in _CACHES.items():
-            new_factor = self.cache_factors.get(cache_name, self.global_factor)
-            callback(new_factor)
+        # block other threads from modifying _CACHES while we iterate it.
+        with _CACHES_LOCK:
+            for cache_name, callback in _CACHES.items():
+                new_factor = self.cache_factors.get(cache_name, self.global_factor)
+                callback(new_factor)

synapse/config/homeserver.py

@@ -36,6 +36,7 @@
 from .redis import RedisConfig
 from .registration import RegistrationConfig
 from .repository import ContentRepositoryConfig
+from .room import RoomConfig
 from .room_directory import RoomDirectoryConfig
 from .saml2_config import SAML2Config
 from .server import ServerConfig
@@ -79,6 +80,7 @@ class HomeServerConfig(RootConfig):
         PasswordAuthProviderConfig,
         PushConfig,
         SpamCheckerConfig,
+        RoomConfig,
         GroupsConfig,
         UserDirectoryConfig,
         ConsentConfig,

synapse/config/room.py

@@ -0,0 +1,80 @@
+# -*- coding: utf-8 -*-
+# Copyright 2020 The Matrix.org Foundation C.I.C.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+import logging
+
+from synapse.api.constants import RoomCreationPreset
+
+from ._base import Config, ConfigError
+
+logger = logging.Logger(__name__)
+
+
+class RoomDefaultEncryptionTypes(object):
+    """Possible values for the encryption_enabled_by_default_for_room_type config option"""
+
+    ALL = "all"
+    INVITE = "invite"
+    OFF = "off"
+
+
+class RoomConfig(Config):
+    section = "room"
+
+    def read_config(self, config, **kwargs):
+        # Whether new, locally-created rooms should have encryption enabled
+        encryption_for_room_type = config.get(
+            "encryption_enabled_by_default_for_room_type",
+            RoomDefaultEncryptionTypes.OFF,
+        )
+        if encryption_for_room_type == RoomDefaultEncryptionTypes.ALL:
+            self.encryption_enabled_by_default_for_room_presets = [
+                RoomCreationPreset.PRIVATE_CHAT,
+                RoomCreationPreset.TRUSTED_PRIVATE_CHAT,
+                RoomCreationPreset.PUBLIC_CHAT,
+            ]
+        elif encryption_for_room_type == RoomDefaultEncryptionTypes.INVITE:
+            self.encryption_enabled_by_default_for_room_presets = [
+                RoomCreationPreset.PRIVATE_CHAT,
+                RoomCreationPreset.TRUSTED_PRIVATE_CHAT,
+            ]
+        elif encryption_for_room_type == RoomDefaultEncryptionTypes.OFF:
+            self.encryption_enabled_by_default_for_room_presets = []
+        else:
+            raise ConfigError(
+                "Invalid value for encryption_enabled_by_default_for_room_type"
+            )
+
+    def generate_config_section(self, **kwargs):
+        return """\
+        ## Rooms ##
+
+        # Controls whether locally-created rooms should be end-to-end encrypted by
+        # default.
+        #
+        # Possible options are "all", "invite", and "off". They are defined as:
+        #
+        # * "all": any locally-created room
+        # * "invite": any room created with the "private_chat" or "trusted_private_chat"
+        #             room creation presets
+        # * "off": this option will take no effect
+        #
+        # The default value is "off".
+        #
+        # Note that this option will only affect rooms created after it is set. It
+        # will also not affect rooms created by other servers.
+        #
+        #encryption_enabled_by_default_for_room_type: invite
+        """

synapse/config/saml2_config.py

@@ -160,7 +160,7 @@ def read_config(self, config, **kwargs):
 
         # session lifetime: in milliseconds
         self.saml2_session_lifetime = self.parse_duration(
-            saml2_config.get("saml_session_lifetime", "5m")
+            saml2_config.get("saml_session_lifetime", "15m")
         )
 
         template_dir = saml2_config.get("template_dir")
@@ -286,7 +286,7 @@ def generate_config_section(self, config_dir_path, server_name, **kwargs):
 
           # The lifetime of a SAML session. This defines how long a user has to
           # complete the authentication process, if allow_unsolicited is unset.
-          # The default is 5 minutes.
+          # The default is 15 minutes.
           #
           #saml_session_lifetime: 5m
 

synapse/config/server.py

@@ -856,7 +856,7 @@ def generate_config_section(
         # number of monthly active users.
         #
         # 'limit_usage_by_mau' disables/enables monthly active user blocking. When
-        # anabled and a limit is reached the server returns a 'ResourceLimitError'
+        # enabled and a limit is reached the server returns a 'ResourceLimitError'
         # with error type Codes.RESOURCE_LIMIT_EXCEEDED
         #
         # 'max_mau_value' is the hard limit of monthly active users above which

synapse/handlers/federation.py

@@ -33,7 +33,12 @@
 from twisted.internet import defer
 
 from synapse import event_auth
-from synapse.api.constants import EventTypes, Membership, RejectedReason
+from synapse.api.constants import (
+    EventTypes,
+    Membership,
+    RejectedReason,
+    RoomEncryptionAlgorithms,
+)
 from synapse.api.errors import (
     AuthError,
     CodeMessageException,
@@ -742,7 +747,10 @@ async def _process_received_pdu(
                 if device:
                     keys = device.get("keys", {}).get("keys", {})
 
-                    if event.content.get("algorithm") == "m.megolm.v1.aes-sha2":
+                    if (
+                        event.content.get("algorithm")
+                        == RoomEncryptionAlgorithms.MEGOLM_V1_AES_SHA2
+                    ):
                         # For this algorithm we expect a curve25519 key.
                         key_name = "curve25519:%s" % (device_id,)
                         current_keys = [keys.get(key_name)]