Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

MSC4033: Explicit ordering of events for receipts #4033

Open
wants to merge 51 commits into
base: main
Choose a base branch
from
Open
Changes from 1 commit
Commits
Show all changes
51 commits
Select commit Hold shift + click to select a range
3c50306
Initial draft
andybalaam Jul 4, 2023
e7983f2
Update number to MSC4033
andybalaam Jul 4, 2023
a75a8e5
Fix mis-spelling
andybalaam Jul 4, 2023
6841e75
Fix another mis-spelling
andybalaam Jul 4, 2023
2521212
Add some more alternatives
andybalaam Jul 4, 2023
fe83ad9
Add TODO list
andybalaam Jul 4, 2023
3ae4995
Add missing bracket
andybalaam Jul 4, 2023
8414c2e
Add an acknowledgements section
andybalaam Jul 4, 2023
7e26924
Make thread roots not in their thread
andybalaam Jul 5, 2023
ee19047
Add a changelog
andybalaam Jul 5, 2023
0e17420
Clarify stream order and handle redactions
andybalaam Jul 5, 2023
a32a3f9
Small clarifications
andybalaam Jul 5, 2023
58102cf
Include updated defintion of read/unread event
andybalaam Jul 5, 2023
8ecb0bb
Provide JSON examples
andybalaam Jul 5, 2023
c452342
Remove TODO list
andybalaam Jul 5, 2023
2d6a03e
Wrap code blocks in pre
andybalaam Jul 5, 2023
fba3bbe
Format code with pre only
andybalaam Jul 5, 2023
f4899a0
Fix misplaced space
andybalaam Jul 5, 2023
afa9629
Add missing "in"
andybalaam Jul 5, 2023
90e5798
Explain what the spec issue is about
andybalaam Jul 5, 2023
76217a0
Remove redundant part of main thread definition
andybalaam Jul 5, 2023
c9bdac8
Reflect the spec saying thread roots are not in main
andybalaam Jul 5, 2023
501bfac
Restrict to just redacted events being always read
andybalaam Jul 5, 2023
17bfe74
Add missing "a"
andybalaam Jul 5, 2023
c02285b
Servers probably also think thread roots are in main
andybalaam Jul 5, 2023
a48c1d0
Move thread_id into content thanks to conversation with Nico
andybalaam Jul 5, 2023
aeb0650
Mention 3051 as an alternative to thread_id
andybalaam Jul 5, 2023
d58758f
Move thread_id to cleartext of content
andybalaam Jul 6, 2023
bbf1c94
Cut down to just ordering
andybalaam Jul 6, 2023
36a28e5
Small brevity improvements
andybalaam Jul 7, 2023
be77a58
Note that order should be inserted by servers
andybalaam Jul 7, 2023
04c4606
Move order to be a sibling of ts in receipts
andybalaam Jul 7, 2023
6e027e2
Reword motivation paragraph
andybalaam Jul 13, 2023
5499b04
Formatting
andybalaam Jul 13, 2023
7493391
Mention the specific API used to get older messages
andybalaam Jul 13, 2023
7d3df44
Make clear the order only needs to increase relative to messages in t…
andybalaam Jul 13, 2023
c45a5e4
Formatting
andybalaam Jul 13, 2023
1b668ce
Clarify the consistency guarantees and meaning of order
andybalaam Jul 13, 2023
d2fe0f4
Include order in redacted events too
andybalaam Jul 13, 2023
694317e
Emphasise the room level when talking about server calculation
andybalaam Jul 13, 2023
be0f7ac
Add a note that this applies to other users' receipts too
andybalaam Jul 17, 2023
e650a11
Acknowledge that we actually are changing the definition of read/unread
andybalaam Jul 17, 2023
7d1728e
Fix typo
andybalaam Aug 3, 2023
12cde91
Fix missing e in mxid
andybalaam Aug 3, 2023
6950a59
Make order on a receipt mandatory
andybalaam Aug 4, 2023
2543428
Document the idea of just never sending receipts that don't have orde…
andybalaam Aug 4, 2023
605eadd
Add a note about fully-read markers being out of scope
andybalaam Nov 30, 2023
d749fb1
Add notes about negative event order
andybalaam Nov 30, 2023
6859b8d
Add a note about negative order meaning event is read
andybalaam Nov 30, 2023
d2cc49d
Add a note about order not being identical
andybalaam Nov 30, 2023
c7a8192
Fix spelling error
andybalaam Dec 1, 2023
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
Prev Previous commit
Next Next commit
Add a note about fully-read markers being out of scope
andybalaam committed Nov 30, 2023
commit 605eaddf12f8487d5d3c1e8bcad442816754ce8a
4 changes: 4 additions & 0 deletions proposals/4033-event-thread-and-order.md
Original file line number Diff line number Diff line change
@@ -7,6 +7,10 @@ so it is difficult to decide whether an event is before or after a receipt.
We propose adding an explicit order number to all events, so that it is clear
which events are read.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

The problem with numbers is that you can't fill gaps between them, so if you already have 15 and 16, there's no place inbetween them. Instead, you could treat the order as a string of digits instead with the property that
15 < 151 < 16

For easier understanding, you can compare this to decimal numbers: 0.15 < 0.151 < 0.16

Copy link

@programmerjake programmerjake Jan 29, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you need to be able to insert a new value between any existing pair, you can use Dyadic Rationals, basically a value of the form a * 2^b where a and b are integers.
e.g. if you have 23 and 24, you can use 47 * 2^-1 = 23.5

Copy link

@toger5 toger5 Feb 12, 2024

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Using an array as the index is also a concept that can be found in CRDT's.
[1], [1,1], [1,2], [1,2,1],[2]
(very similar to what timokoesters is proposing but strings bring more typing ambiguity/issues.)

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

In terms of storage/memory used per event (thus bandwidth usage), dyadic rationals sound optimal: sufficiently large space, only require two u32 or two u64. Strings or array representations would require bigger allocations. (Nobody really proposed it, but FP numbers would be a waste in terms of storage space b/o all the NaNs + IEEE754 is hard to get right.)

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

dyadic rationals only require one byte for the exponent, unless you want to support a value with >256 bits (imo excessive), so they only need 5 bytes (32-bit mantissa) or 9 bytes (64-bit mantissa), though you can definitely use more for alignment or convenience.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Note that we will need to be able to handle edge cases where we run out of exponents, e.g. you have an event A and you keep inserting events just after it (which may very well end up a common case). If you only have a one byte exponent then you quickly run out of room.

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

if you run out of exponent bits, that also means the mantissa has to be at least 256-bits wide (or something like that), so you will need big-integer arithmetic for that...i think that's likely excessive.

maybe a better idea is to maintain a binary tree with one node per message and have the server send the list of changed nodes (if the message's contents doesn't change, all that needs to be sent is the node ids of the tree's children, since that's all that changes during tree balancing), this allows tree balancing to avoid the tree getting too deep.

messages would then be ordered by their position in an in-order traversal of that tree.


This proposal covers receipts, and not fully-read markers. Fully-read markers
have the same issue in terms of ordering, and should probably be fixed in a
similar way, but they are not addressed here.

## Motivation

To decide whether a room is unread, a Matrix client must decide whether it