Failed to send_join via any server due to "Object of type Decimal is not JSON serializable"

[nightred]

Description

When trying to join the channel #nextcloud:matrix.org an error is shown that reads 'Failed to send_join via matrix.org'
The issue persists on every join attempt.

Steps to reproduce

Join the channel #nextcloud:matrix.org
Review Logs and get the following error.

2021-10-20 09:17:35,911 - synapse.federation.federation_client - 616 - WARNING - POST-61221 - Failed to send_join via matrix.org

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
result = current_context.run(gen.send, result)
StopIteration: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
result = current_context.run(gen.send, result)

ation: {'ed25519:auto': FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x151c2e943f70>, valid_until_ts=1576767829750), 'ed25519:a_RXGa': FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x151be8e8f790>, valid_until_ts=1634773315345)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 576, in _try_destination_list
return await callback(destination)
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 825, in send_request
await concurrently_execute(
File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
result = current_context.run(gen.send, result)
File "/usr/local/lib/python3.8/site-packages/synapse/util/async_helpers.py", line 192, in _concurrently_execute_inner
await maybe_awaitable(func(value))
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 815, in _execute
valid_pdu = await self._check_sigs_and_hash_and_fetch_one(
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 458, in _check_sigs_and_hash_and_fetch_one
res = await self._check_sigs_and_hash(room_version, pdu)
File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_base.py", line 65, in _check_sigs_and_hash
if not check_event_content_hash(pdu):
File "/usr/local/lib/python3.8/site-packages/synapse/crypto/event_signing.py", line 42, in check_event_content_hash
name, expected_hash = compute_content_hash(event.get_pdu_json(), hash_algorithm)
File "/usr/local/lib/python3.8/site-packages/synapse/crypto/event_signing.py", line 96, in compute_content_hash
event_json_bytes = encode_canonical_json(event_dict)
File "/usr/local/lib/python3.8/site-packages/canonicaljson.py", line 73, in encode_canonical_json
s = _canonical_encoder.encode(json_object)
File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
chunks = self.iterencode(o, _one_shot=True)
File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
return _iterencode(o, 0)
File "/usr/local/lib/python3.8/site-packages/canonicaljson.py", line 30, in _default
raise TypeError(
TypeError: Object of type Decimal is not JSON serializable
2021-10-20 09:17:35,911 - synapse.http.server - 88 - INFO - POST-61221 - <XForwardedForRequest at 0x151c15df5370 method='POST' uri='/_matrix/client/r0/join/!TEwfEWdDwdaFazXmwD%3Amatrix.org?server_name=matrix.org' clientproto='HTTP/1.1' site='8008'> SynapseError: 502 - Failed to send_join via any server

Version information

• Homeserver: matrix.jenovarain.com

• Version: {"server_version":"1.45.0","python_version":"3.8.12"}

• Install method: Docker

• Platform: Docker

[squahtx]

Duplicate of a comment in #10021: #10021 (comment) . #10021's become a dumping ground for various send_join failures with different causes, so let's keep this issue for this particular cause.

The remote homeserver is returning a /send_join response containing floats with decimal points, which makes ijson return a Decimal to Synapse.

@DMRobertson suggests using use_float=True, which would solve the immediate error. Whether the event hash matches afterwards is another matter.

Note that the problematic room in the post above is #NextCloud:matrix.org (!TEwfEWdDwdaFazXmwD:matrix.org), not #nextcloud:matrix.org (!BTFFlyEeuhcFipzRnZ:matrix.org). The problematic event is $15668961892578633CPOCY:matrix.org:

{"origin":"matrix.org","origin_server_ts":1634756526450,"pdus":[{"auth_events":[["$14889644723000232lIUEU:matrix.org",{"sha256":"ct0/Rr2g95h8cUZoPFXgSBXbyHOSYSwnyDHK+hwOfD4"}],["$156604474753394BKyyU:matrix.org",{"sha256":"iHuRPwfobe45nb1NDHZgpd3xUurLYDacTWzQELlxg+8"}],["$14889644723000230Leyrk:matrix.org",{"sha256":"uvY+tfZUSsN1r4r3pR30ywK/8DOvsrxRjGcmQPN95CA"}]],"content":{"ban":50,"events":{"im.vector.modular.widgets":50,"m.room.avatar":100,"m.room.canonical_alias":100,"m.room.history_visibility":100,"m.room.name":100,"m.room.power_levels":100,"m.room.topic":100},"events_default":0,"invite":0,"kick":50,"notifications":{"room":100.0},"redact":50,"state_default":100,"users":{"@Creak:matrix.org":50,"@brad:koehn.com":50,"@illukas:matrix.org":50,"@natrius:matrix.org":50,"@satriani:matrix.org":100},"users_default":0},"depth":2581,"event_id":"$15668961892578633CPOCY:matrix.org","hashes":{"sha256":"BI82yYRWZ6H0p146e+HICkPztlKfObAuBJW8xb54Yo8"},"origin":"matrix.org","origin_server_ts":1566896189852,"prev_events":[["$15668924972564026SXoZV:matrix.org",{"sha256":"H6YJjoffHc/Q85GdNzsIOJ7/Bm2LmAf5/p3R8mPUctA"}]],"prev_state":[],"room_id":"!TEwfEWdDwdaFazXmwD:matrix.org","sender":"@satriani:matrix.org","state_key":"","type":"m.room.power_levels","signatures":{"matrix.org":{"ed25519:auto":"viI1Pp4qxVs1S4jDCqutGZuWdd4UWUjzplWgH+zBYOJnNkrAdW6tzA8sDMqqByDc5QK1O0M0tCJYrcFmE/xjBg"}},"unsigned":{"replaces_state":"$156604474753394BKyyU:matrix.org","age":67860336598}}]}

content.notifications.room is 100.0 instead of 100

As noted in #10021, joining #ghidra:matrix.org / !RistbroappQPsmzqnr:matrix.org is broken, and it's for the same reason. Event $1574893667299573PRkGI:matrix.org:

{"origin":"matrix.org","origin_server_ts":1634755280753,"pdus":[{"auth_events":[["$1553000572438nTzgN:matrix.org",{"sha256":"NGWtwRETqCrBH1fMpyE+8wTaYE022TJ6kxCWGxzvxZY"}],["$1553617609357765HBZlL:matrix.org",{"sha256":"tsXB6/GkpfASvz+cqGJ2WUqh+IjFRg5Vd6ihMva5bS0"}],["$156771980867333fZwzd:matrix.org",{"sha256":"1R54GWyy24wtLdKgtZrCvQmdHreFZO8iKQ9hbBPU1oQ"}]],"content":{"ban":50,"events":{"im.vector.modular.widgets":50,"m.room.avatar":49,"m.room.canonical_alias":50,"m.room.history_visibility":100,"m.room.name":45,"m.room.power_levels":100,"m.room.topic":44},"events_default":0,"invite":0,"kick":47,"notifications":{"room":46.0},"redact":40,"state_default":49,"users":{"@_discord_bot:t2bot.io":40,"@kasifdekel:matrix.org":100,"@steps-:matrix.org":100,"@telegram:t2bot.io":40,"@telegram_240575894:t2bot.io":50,"@telegram_604709192:t2bot.io":50},"users_default":0},"depth":6771,"event_id":"$1574893667299573PRkGI:matrix.org","hashes":{"sha256":"k5n0pljeGmYFhgoKLf9CUxcHE9Ckl65qyA9gNGlLsvw"},"origin":"matrix.org","origin_server_ts":1574893667507,"prev_events":[["$157489351328421Jeqab:t2bot.io",{"sha256":"vbe12RzeyxM1n+p5o9Ay6f6zKpc086EUEpZBO4aDv8k"}]],"prev_state":[],"room_id":"!RistbroappQPsmzqnr:matrix.org","sender":"@steps-:matrix.org","state_key":"","type":"m.room.power_levels","signatures":{"matrix.org":{"ed25519:auto":"dvxSidd5vBdQSSA7YG+P8UmBuxteImK2rCeDFEJig7iJuG56ezyBFZRq1x7sKy4xk8N20hOWtBLO/vrOt+YYAw"}},"unsigned":{"replaces_state":"$156771980867333fZwzd:matrix.org","age":59861613246}}]}

content.notifications.room is 46.0 instead of 46

Both of these affected rooms are version 1

[squahtx]

The hashes for both the above events match, as long as the trailing .0s are included

[localjo]

Is there any workaround for this? The Nextcloud room is the main reason I use Matrix and I'm trying to switch to a Matrix account on a different server, but I'm blocked because that account can't join the room that is most important to me.

What will the fix for this issue involve? Is it on the Nextcloud room folks to fix it? Is there any solution I could work on?

[squahtx]

@localjo A fix for this particular failure is on the todo list for the Synapse team. It's likely to be fixed in 1.47.0rc1, in two weeks, though this is a very rough estimate and I can't make any promises with regards to timelines.

In the meantime, if you're handy with Python and are the admin of the homeserver, you could try modifying the code to pass use_float=True wherever Synapse uses the ijson package and reporting back on whether it works.

[BBaoVanC]

Weird, I don't have this issue on v1.45.1 in Docker, and my server hasn't joined the room before.

[squahtx]

Weird, I don't have this issue on v1.45.1 in Docker, and my server hasn't joined the room before.

Room aliases are case sensitive. It's #NextCloud:matrix.org that's broken, not #nextcloud:matrix.org.

[BBaoVanC]

Weird, I don't have this issue on v1.45.1 in Docker, and my server hasn't joined the room before.

Room aliases are case sensitive. It's #NextCloud:matrix.org that's broken, not #nextcloud:matrix.org.

Oh, I misread it then. I'll test it when I get home