Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Spell fix Matrix Summer Special 2022 #1402

Merged
merged 4 commits into from Aug 15, 2022
Merged
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@ Over the years we’ve done a lot of optimisation work on the core team implemen

### Sliding Sync (aka Sync v3)

Historically, /sync always assumed that the client would typically want to know about all the conversations its user is in - much as an IRC client or XMPP client is aware of all your current conversations. This provided some nice properties - such as automatically enabling perfect offline support, simplifying client and server development, and making features like “jump to room” and “tab complete” work instantly given the data is all client-side. In the early days of Matrix, when nobody was yet a poweruser, this wasn’t really a problem - but as users join more conversations and join bigger rooms, it’s become one of Matrix’s biggest performance bottlenecks. In practice, logging into a large account (~4000 rooms) can take ~10 minutes and hundreds of megabytes of network traffic, which is clearly ridiculous. Worse: if you go offline for a day or so, the incremental sync to catch back up can take minutes to calculate (and can even end up being worse than an initial sync).
Historically, /sync always assumed that the client would typically want to know about all the conversations its user is in - much as an IRC client or XMPP client is aware of all your current conversations. This provided some nice properties - such as automatically enabling perfect offline support, simplifying client and server development, and making features like “jump to room” and “tab complete” work instantly given the data is all client-side. In the early days of Matrix, when nobody was yet a power user, this wasn’t really a problem - but as users join more conversations and join bigger rooms, it’s become one of Matrix’s biggest performance bottlenecks. In practice, logging into a large account (~4000 rooms) can take ~10 minutes and hundreds of megabytes of network traffic, which is clearly ridiculous. Worse: if you go offline for a day or so, the incremental sync to catch back up can take minutes to calculate (and can even end up being worse than an initial sync).

To fix this, we started work on Sliding Sync ([MSC3575](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md)) in 2021: a complete reimagining of the /sync API used by Matrix clients to receive data from their homeserver. In Sliding Sync, we only send the client the data it needs to render its UI. Most importantly, we only tell it about the subset of rooms which it is visible in the scroll window of its room list (or that it needs to display notifications about). As the user scrolls around the room list, they slide the window up and down - hence the name “sliding sync”. Sliding Sync was originally called Sync v3, given it’s our 3rd iteration of the sync API - it got renamed Sliding Sync given the current sync API confusingly ended up with a prefix of /v3.
To fix this, we started work on Sliding Sync ([MSC3575](https://github.com/matrix-org/matrix-spec-proposals/blob/kegan/sync-v3/proposals/3575-sync.md)) in 2021: a complete re-imagining of the /sync API used by Matrix clients to receive data from their homeserver. In Sliding Sync, we only send the client the data it needs to render its UI. Most importantly, we only tell it about the subset of rooms which it is visible in the scroll window of its room list (or that it needs to display notifications about). As the user scrolls around the room list, they slide the window up and down - hence the name “sliding sync”. Sliding Sync was originally called Sync v3, given it’s our 3rd iteration of the sync API - it got renamed Sliding Sync given the current sync API confusingly ended up with a prefix of /v3.
This conversation was marked as resolved.
Show resolved Hide resolved

[Back in December](https://matrix.org/blog/2021/12/22/the-mega-matrix-holiday-special-2021#sync-v3) our work on Sliding Sync was still pretty early: we had the initial MSC, an experimental proxy that converted the existing sync v2 API into Sliding Sync, and a simple proof-of-concept web client to exercise it. Since then, however, there has been spectacular progress:

Expand Down Expand Up @@ -62,7 +62,7 @@ As you can see, this is palpably coming together, but there’s still a bunch of
* Adding back in loading room members
* Apply quality-of-service to to-device messages so we prioritise ones relevant to the current sliding window
* Sync encrypted rooms in the background to search for notifications (and for indexing).
* More local caching to speed up operations which now require checking the server (e.g. ctrl/cmd-K room switching)
* More local caching to speed up operations which now require checking the server (e.g. Ctrl/Cmd-K room switching)

We also need to determine whether it’s viable to run the sliding-sync proxy against matrix.org for general production use, or whether we’ll need native support in Synapse before we can turn it on by default for everyone. But these are good problems to have!!

Expand All @@ -75,7 +75,7 @@ Meanwhile, over in the land of Rust, we’ve been making huge progress in maturi

matrix-rust-sdk itself is now getting a [steady stream of releases](https://github.com/matrix-org/matrix-rust-sdk/releases) - including long-awaited official node bindings, providing excellent and performant encryption support via the [newly audited](https://matrix.org/blog/2022/05/16/independent-public-audit-of-vodozemac-a-native-rust-reference-implementation-of-matrix-end-to-end-encryption/) vodozemac Rust implementation of Olm. It’s also great to see loads of major contributions to matrix-rust-sdk from across the wider Matrix community - particularly from Ruma, Fractal, Famedly and others - thank you!! As a result the SDK is shaping up to be much more healthy and heterogeneous than the original matrix-{js,ios,android}-sdk projects.

On Element X itself: matrix-rust-sdk is being used first on iOS in [Element X iOS](https://github.com/vector-im/element-x-ios) - aiming first for launching a stable “barbecue” featureset (i.e. personal messaging) asap, followed by adding on “banquet” features (i.e. team collaboration) such as spaces and threads afterwards. We’ve shamelessly misappropriated the barbecue & banquet terminology from Tobias Bernard’s excellent blog post “[Banquets and Barbecues](https://blogs.gnome.org/tbernard/2018/05/16/banquets-and-barbecues/)” - although, ironically, unlike the post, our plan is still to have a single app which incrementally discloses the banquet functionality as the user’s barbecue starts to sprawl. We’ve just published the brand new [development roadmap](https://github.com/matrix-org/matrix-rust-sdk/projects/1) for Element X from the rust-sdk perspective on GitHub. Above all else, the goal of Element X is to be the fastest mobile messenger out there in terms of launch and sync time, thanks to Sliding Sync. Not just for Matrix - but the fastest messenger, full stop :D Watch this space to see how we do!
On Element X itself: matrix-rust-sdk is being used first on iOS in [Element X iOS](https://github.com/vector-im/element-x-ios) - aiming first for launching a stable “barbecue” feature set (i.e. personal messaging) asap, followed by adding on “banquet” features (i.e. team collaboration) such as spaces and threads afterwards. We’ve shamelessly misappropriated the barbecue & banquet terminology from Tobias Bernard’s excellent blog post “[Banquets and Barbecues](https://blogs.gnome.org/tbernard/2018/05/16/banquets-and-barbecues/)” - although, ironically, unlike the post, our plan is still to have a single app which incrementally discloses the banquet functionality as the user’s barbecue starts to sprawl. We’ve just published the brand new [development roadmap](https://github.com/matrix-org/matrix-rust-sdk/projects/1) for Element X from the rust-sdk perspective on GitHub. Above all else, the goal of Element X is to be the fastest mobile messenger out there in terms of launch and sync time, thanks to Sliding Sync. Not just for Matrix - but the fastest messenger, full stop :D Watch this space to see how we do!

Finally: Element is getting a major redesign of the core UI on both iOS and Android - both for today’s Element and Element X. I’m not going to spoil the final result (which is looking *amazing*) given it’ll have a proper glossy launch in a few weeks, but you can get a rough idea based on the earlier design previewed by Amsha back in June:

Expand Down Expand Up @@ -109,7 +109,7 @@ Finally, alongside faster remote joins, we’re also working on faster local joi

All the work above describes some pretty bold changes to speed up Matrix and improve usability - but in order to land these changes with confidence, avoiding regressions both now and in future, we have really levelled up our testing this year.

Looking at matrix-react-sdk as used by Element Web/Desktop: all PRs made to matrix-js-sdk must now pass 80% unit test coverage for new code (measured using [Sonarqube](https://sonarcloud.io/project/overview?id=matrix-js-sdk), enforced as a Github PR check). All matrix-react-sdk PRs must be accompanied by a mix of unit tests, end-to-end tests (via [Cypress](https://www.cypress.io/)) and screenshot tests (via [percy.io](https://percy.io/)). All regressions (in both nightly and stable) are retro’d to ensure fixed things stay fixed (usually via writing new tests), and we have converted [fully to typescript](https://arewetsyet.bit.ovh/) for full type safety.
Looking at matrix-react-sdk as used by Element Web/Desktop: all PRs made to matrix-js-sdk must now pass 80% unit test coverage for new code (measured using [Sonarqube](https://sonarcloud.io/project/overview?id=matrix-js-sdk), enforced as a GitHub PR check). All matrix-react-sdk PRs must be accompanied by a mix of unit tests, end-to-end tests (via [Cypress](https://www.cypress.io/)) and screenshot tests (via [percy.io](https://percy.io/)). All regressions (in both nightly and stable) are retro’d to ensure fixed things stay fixed (usually via writing new tests), and we have converted [fully to typescript](https://arewetsyet.bit.ovh/) for full type safety.

Concretely, since May, we’ve increased js-sdk unit test coverage by ~10% globally, increased react-sdk coverage by ~17%, and added ever more Cypress integration tests to cover the broad strokes. Cypress now [completely replaces](https://github.com/matrix-org/matrix-react-sdk/pull/9104) our old Puppeteer-based end-to-end tests, and Sliding Sync work in matrix-react-sdk is being extensively tested by Cypress from the outset (the Sliding Sync PR literally comes with a Cypress test suite).

Expand Down Expand Up @@ -141,9 +141,9 @@ Most importantly, foci are decentralised, just like Matrix: there is no single c

Now, the VoIP team have been busy polishing Element Call (e.g. chasing down end-to-end encryption edge cases and reliability), and also figuring out how to embed it into Element and other Matrix clients as a quick way to get excellent group VoIP (more on that later). As a result, work on building out foci for scalable conferencing had to be pushed down the line.

But in the last few months this completely changed, thanks to an **amazing** open source contribution from Sean DuBois, project lead over at [Pion](https://pion.ly/) - the excellent golang WebRTC implementation. Inspired by our [initial talk](https://2021.commcon.xyz/talks/extending-matrix-s-e2ee-calls-to-multiparty) about MSC3401 at CommCon, Sean independently decided to see how hard it’d be to build a Selective Forwarding Unit (SFU) focus that implemented MSC3401 semantics using Pion - and published it at [https://github.com/sean-der/sfu-to-sfu](https://github.com/sean-der/sfu-to-sfu) (subsequently donated to github.com/matrix-org). In many ways this was a flag day for Matrix: it’s the first time that a core MSC from the core team has been first implemented from outside the core team (let alone outside the Matrix community!). It’s the VoIP equivalent of Synapse starting off life as a community contribution rather than being written by the core team.
But in the last few months this completely changed, thanks to an **amazing** open source contribution from Sean DuBois, project lead over at [Pion](https://pion.ly/) - the excellent Go WebRTC implementation. Inspired by our [initial talk](https://2021.commcon.xyz/talks/extending-matrix-s-e2ee-calls-to-multiparty) about MSC3401 at CommCon, Sean independently decided to see how hard it’d be to build a Selective Forwarding Unit (SFU) focus that implemented MSC3401 semantics using Pion - and published it at [https://github.com/sean-der/sfu-to-sfu](https://github.com/sean-der/sfu-to-sfu) (subsequently donated to github.com/matrix-org). In many ways this was a flag day for Matrix: it’s the first time that a core MSC from the core team has been first implemented from outside the core team (let alone outside the Matrix community!). It’s the VoIP equivalent of Synapse starting off life as a community contribution rather than being written by the core team.
This conversation was marked as resolved.
Show resolved Hide resolved

Eitherway: Sean’s SFU work has opened the floodgates to making native Matrix conferencing actually scale, with Šimon Brandner and I jumping in to [implement SFU support](https://github.com/matrix-org/matrix-js-sdk/pull/2423) in matrix-js-sdk… and as of a few weeks ago we did the first ever SFU-powered Matrix call - which worked impressively well for 12 participants!
Either way: Sean’s SFU work has opened the floodgates to making native Matrix conferencing actually scale, with Šimon Brandner and I jumping in to [implement SFU support](https://github.com/matrix-org/matrix-js-sdk/pull/2423) in matrix-js-sdk… and as of a few weeks ago we did the first ever SFU-powered Matrix call - which worked impressively well for 12 participants!

![12 person Element Call](/blog/img/2022-08-15-sfu.jpg)

Expand All @@ -165,7 +165,7 @@ Meanwhile it’s worth noting that Element Call is not the only MSC3401 implemen

Elsewhere on VoIP, we’ve also been hard at work figuring out how to embed Element Call into Matrix clients in general, starting with Element Web, iOS & Android. Given MSC3401 is effectively a superset of native 1:1 Matrix VoIP calling, we’d ideally like to replace the current 1:1-only VoIP implementation in Element with an embedded instance of Element Call (not least so we don’t have to maintain it in triplicate over Web/iOS/Android, and because WebRTC-in-a-webview really isn’t very different to native WebRTC). To do this efficiently however, the embedded Element Call needs to share the same underlying Matrix client as the parent Element client (otherwise you end up wasting resources and devices and E2EE overhead between the two). Effectively Element Call ends up needing to parasite off the parent’s client. We call this approach “matryoshka embedding”, given it resembles nested Russian dolls. 🪆

In practice, we do this by extending the Widget API to let Matrix clients within the widget share the parent’s Matrix client for operations such as sending and receiving to-device messages and accessing TURN servers (c.f. [MSC3819](https://github.com/matrix-org/matrix-spec-proposals/blob/travis/msc/widgets-send-recv-toDevice/proposals/3819-to-device-messages-for-widgets.md) and [MSC3846](https://github.com/robintown/matrix-doc/blob/widget-turn-servers/proposals/3846-widget-turn-servers.md)). This in turn has been [implemented](https://github.com/matrix-org/matrix-widget-api/pull/57) in the matrix-widget-api helper library for widget implementors - and then a few days ago Robin demonstrated the world’s first ever matryoshka embedded Element Call call, looking like this:
In practice, we do this by extending the Widget API to let Matrix clients within the widget share the parent’s Matrix client for operations such as sending and receiving to-device messages and accessing TURN servers (c.f. [MSC3819](https://github.com/matrix-org/matrix-spec-proposals/blob/travis/msc/widgets-send-recv-toDevice/proposals/3819-to-device-messages-for-widgets.md) and [MSC3846](https://github.com/robintown/matrix-doc/blob/widget-turn-servers/proposals/3846-widget-turn-servers.md)). This in turn has been [implemented](https://github.com/matrix-org/matrix-widget-api/pull/57) in the matrix-widget-api helper library for widget implementers - and then a few days ago Robin demonstrated the world’s first ever matryoshka embedded Element Call call, looking like this:

![Matryoshka embedded Element Call](/blog/img/2022-08-15-matryoshka.jpg)

Expand Down Expand Up @@ -200,7 +200,7 @@ Now, we’re hoping to give Third Room a proper launch in a few weeks, so I’m
* Building an asset pipeline from Unity and Blender through to the glTF assets which Third Room uses.
* Initial framework for an in-world direct-manipulation [editor](https://twitter.com/matrixdotorg/status/1550534909089189888)
* Lightmap support for beautiful high-performance static lighting and shadows
* Full post-processing pipeline (bloom, depth-of-field, antialiasing etc)
* Full post-processing pipeline (bloom, depth-of-field, anti-aliasing etc)
* Integrating with OIDC for login, registration, and account management (see OIDC below)

As a quick teaser - here’s an example of a [Unity asset](https://assetstore.unity.com/packages/3d/environments/sci-fi/3d-scifi-kit-vol-3-121447) exported into Third Room, showing off lightmaps (check out the light and shadows cast by the strip lighting inside, or the shadow on the ground outside). Ignore the blurry HDR environment map of Venice in the background, which is just there to give the metals something to reflect. Check out the stats on the right-hand side: on Robert’s M1 Macbook Pro we’re getting a solid 60fps at 2000x1244px, with 13.12ms of unused gametime available for every 16.67ms frame, despite already showing a relatively complicated asset!
Expand Down Expand Up @@ -258,7 +258,7 @@ Research on the [Pinecone](https://github.com/matrix-org/pinecone/) overlay netw

Comparing all of the above with the [predictions for 2022](https://matrix.org/blog/2021/12/22/the-mega-matrix-holiday-special-2021#2022) section of the end-of-year blog post, we’re making very strong progress in a tonne of areas - and the list above isn’t comprehensive. For instance, we haven’t called out all the work that the Trust & Safety team are doing to roll out advanced moderation features by default to all communities - or the work that Eric has been doing to close the remaining gap between Gitter and Matrix by creating new static archives throughout Matrix. Hydrogen has also been beavering away to provide a tiny but perfectly formed web client suitable for embedding, including the new embeddable Hydrogen SDK. We haven’t spoken about the work that the Cryptography team have been doing to adopt vodozemac and matrix-rust-sdk-crypto throughout matrix-{js,ios,android}-sdk, or improve encryption stability and security throughout. We’ve also not spoken about the new initiative to fix long-term chronic bugs (outside of the work above) in general - or all the work being done around [Digital Markets Act interoperability](https://matrix.org/blog/2022/03/30/technical-faq-on-the-digital-markets-act)…

Other things left on the menu for this year include getting Threads out of beta: we’ve had a bit of an adventure here figuring out how to get the right semantics for notification badges and unread state in rooms with threads (especially if you use a mix of clients which support and don’t spuport threads), and once that’s done we’ll be returning to Spaces (performance, group permissions etc).
Other things left on the menu for this year include getting Threads out of beta: we’ve had a bit of an adventure here figuring out how to get the right semantics for notification badges and unread state in rooms with threads (especially if you use a mix of clients which support and don’t support threads), and once that’s done we’ll be returning to Spaces (performance, group permissions etc).


## Matrix 2.0?
Expand Down