Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Failed to send_join via any server for #emacs:matrix.org #10021

Closed
MTRNord opened this issue May 19, 2021 · 14 comments
Closed

Failed to send_join via any server for #emacs:matrix.org #10021

MTRNord opened this issue May 19, 2021 · 14 comments
Labels
A-Federated-Join joins over federation generally suck P3 (OBSOLETE: use S- labels.) Approved backlog: not yet scheduled, will accept patches S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Needs-Info This issue is blocked awaiting information from the reporter

Comments

@MTRNord
Copy link
Contributor

MTRNord commented May 19, 2021

Description

Joining #emacs:matrix.org fails with a Failed to send_join via any server. Join method was the room directory via element web.

Steps to reproduce

  • Search emacs in the matrix.org room dir via ele web
  • Click and join the room
2021-05-19 16:19:31,049 - synapse.federation.federation_client - 548 - WARNING - POST-1001498 - Failed to send_join via matrix.org
Traceback (most recent call last):
  File "/opt/synapse/synapse_git/synapse/federation/federation_client.py", line 517, in _try_destination_list
    return await callback(destination)
  File "/opt/synapse/synapse_git/synapse/federation/federation_client.py", line 711, in send_request
    room_version=room_version,
  File "/opt/synapse/synapse_git/synapse/federation/federation_client.py", line 427, in _check_sigs_and_hash_and_fetch
    ).addErrback(unwrapFirstError)
  File "/opt/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py", line 761, in send
    raise result.value
twisted.internet.defer.FirstError: FirstError[#0, [Failure instance: Traceback: <class 'synapse.api.errors.SynapseError'>: 403: event id $1449445060123025NBoor:matrix.org: unable to verify signature for sender ghostcorp.net: 400: Not signed by ghostcorp.net
/opt/synapse/synapse_git/synapse/federation/federation_base.py:76:_check_sigs_and_hashes
/opt/synapse/synapse_git/synapse/federation/federation_base.py:219:_check_sigs_on_pdus
/opt/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py:348:addErrback
/opt/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py:330:addCallbacks
--- <exception caught here> ---
/opt/synapse/env/lib/python3.7/site-packages/twisted/internet/defer.py:662:_runCallbacks
/opt/synapse/synapse_git/synapse/federation/federation_base.py:216:sender_err
]]

Which seems to suggest that room is broken from the perspective of my HS?

This has the side effect that my voyager bot tries to join the room over and over, causes a membership event but never actually joins from my synapse perspective.

Perspective from other servers is that synapse sends membership join events over and over again. On Element android this spams the timeline and as my synapse doesnt realise this I cannot really prevent this in my code I think unless I cache this data myself.

Version information

  • Homeserver: matrix.nordgedanken.dev

If not matrix.org:

  • Version: 1.34.0 (b=develop,9752849e2,dirty)

  • Install method: Git clone with pip

  • Platform: Debian 10
@MTRNord
Copy link
Contributor Author

MTRNord commented May 19, 2021

View from other Homeservers in the emacs room:

image

@erikjohnston
Copy link
Member

That stack trace is really odd, we should be catching that error correctly:

synapse/synapse/federation/federation_client.py

Lines 391 to 394 in 1350b05

try:
res = await make_deferred_yieldable(deferred)
except SynapseError:
res = None

@erikjohnston erikjohnston added S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. labels May 20, 2021
@MTRNord
Copy link
Contributor Author

MTRNord commented May 20, 2021 •
edited
Loading

@erikjohnston Not sure if this makes any difference but I have a workaround too applied that fixes #9956 for me. I do multiple by 10000 in this line:

synapse/synapse/http/site.py

Line 106 in 3ff2251

if self.content.tell() + len(data) > self._max_request_body_size:
but I dont think this makes a difference in this very issue. But I thought it is better to mention than to not mention it.

@Hodapp87
Copy link

Seeing the same thing on a fresh installation of 1.34.0.

@Necronian
Copy link

I've had the issue since early December.

@hiredman
Copy link

I see the same error when trying to join #fedora:matrix.org also referencing ghostcorp.net

@callahad callahad added the P3 (OBSOLETE: use S- labels.) Approved backlog: not yet scheduled, will accept patches label May 27, 2021
@richvdh richvdh added the A-Federated-Join joins over federation generally suck label May 27, 2021
@mkroman
Copy link

mkroman commented Jul 15, 2021

I'm having a similar issue when trying to join #ghidra:matrix.org on synapse v1.38.0.

2021-07-15 06:04:35,228 - synapse.access.http.8008 - 389 - INFO - GET-67- 10.42.1.122 - 8008 - {@mk:rwx.im} Processed request: 30.138sec/-0.000sec (0.001sec, 0.000sec) (0.001sec/0.130sec/1) 473B 200 "GET /_matrix/client/r0/sync?filter=0&timeout=30000&since=s19082_198914_0_3360_451_5_82_8486_1 HTTP/1.1" "Mozilla/5.0 (X11; Linux x86_64) AppleWebKit/537.36 (KHTML, like Gecko) Element/1.7.29 Chrome/89.0.4389.128 Electron/12.0.14 Safari/537.36" [0 dbevts]
2021-07-15 06:04:36,755 - synapse.http.matrixfederationclient - 278 - INFO - POST-71- {GET-O-18} [matrix.org] Completed request: 200 OK in 0.14 secs, got 69 bytes - GET matrix://matrix.org/_matrix/federation/v1/query/directory?room_alias=%23ghidra%3Amatrix.org
2021-07-15 06:04:37,283 - synapse.http.matrixfederationclient - 278 - INFO - POST-71- {GET-O-19} [matrix.org] Completed request: 200 OK in 0.25 secs, got 1056 bytes - GET matrix://matrix.org/_matrix/federation/v1/make_join/%21RistbroappQPsmzqnr%3Amatrix.org/%40mk%3Arwx.im?ver=1&ver=2&ver=3&ver=4&ver=5&ver=6&ver=org.matrix.msc2176&ver=org.matrix.msc3083&ver=7
2021-07-15 06:04:38,520 - synapse.metrics - 598 - INFO - sentinel- Collecting gc 1
2021-07-15 06:04:43,870 - synapse.http.matrixfederationclient - 278 - INFO - POST-71- {PUT-O-20} [matrix.org] Completed request: 200 OK in 6.55 secs, got 11110268 bytes - PUT matrix://matrix.org/_matrix/federation/v2/send_join/%21RistbroappQPsmzqnr%3Amatrix.org/%2416263290773VTuNy%3Arwx.im
2021-07-15 06:04:43,870 - synapse.federation.federation_client - 732 - INFO - POST-71- Processing from send_join 9706 events
2021-07-15 06:04:44,502 - synapse.federation.federation_client - 575 - WARNING - POST-71- Failed to send_join via matrix.org
Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
StopIteration: 0

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
StopIteration: {'ed25519:auto': FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x7ff6039cf340>, valid_until_ts=1576767829750), 'ed25519:a_RXGa': FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x7ff6039da1f0>, valid_until_ts=1626404282697)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 545, in _try_destination_list
    return await callback(destination)
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 752, in send_request
    await concurrently_execute(
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1445, in _inlineCallbacks
    result = current_context.run(g.send, result)
  File "/usr/local/lib/python3.8/site-packages/synapse/util/async_helpers.py", line 187, in _concurrently_execute_inner
    await maybe_awaitable(func(value))
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 742, in _execute
    valid_pdu = await self._check_sigs_and_hash_and_fetch_one(
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 432, in _check_sigs_and_hash_and_fetch_one
    res = await self._check_sigs_and_hash(room_version, pdu)
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_base.py", line 65, in _check_sigs_and_hash
    if not check_event_content_hash(pdu):
  File "/usr/local/lib/python3.8/site-packages/synapse/crypto/event_signing.py", line 42, in check_event_content_hash
    name, expected_hash = compute_content_hash(event.get_pdu_json(), hash_algorithm)
  File "/usr/local/lib/python3.8/site-packages/synapse/crypto/event_signing.py", line 96, in compute_content_hash
    event_json_bytes = encode_canonical_json(event_dict)
  File "/usr/local/lib/python3.8/site-packages/canonicaljson.py", line 73, in encode_canonical_json
    s = _canonical_encoder.encode(json_object)
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/canonicaljson.py", line 30, in _default
    raise TypeError(
TypeError: Object of type Decimal is not JSON serializable
2021-07-15 06:04:44,503 - synapse.http.server - 88 - INFO - POST-71- <XForwardedForRequest at 0x7ff60ad854c0 method='POST' uri='/_matrix/client/r0/join/%23ghidra%3Amatrix.org' clientproto='HTTP/1.1' site='8008'> SynapseError: 502 - Failed to send_join via any server```

@d-513
Copy link

d-513 commented Aug 7, 2021

same issue #linux:libera.chat

@benparsons
Copy link
Member

I'm have the same issue:

Aug 09 13:39:51 benp-synapse matrix-synapse[2552]: 2021-08-09 13:39:51,844 - synapse.federation.federation_client - 575 - WARNING - POST-1250672 - Failed to make_join via vector.modular.im
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]: Traceback (most recent call last):
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     result = current_context.run(gen.send, result)
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]: StopIteration: DestinationRetryTimings(failure_ts=1628516389382, retry_last_ts=1628516389382, retry_interval=600000)
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]: During handling of the above exception, another exception occurred:
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]: Traceback (most recent call last):
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 545, in _try_destination_list
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     return await callback(destination)
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 632, in send_request
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     ret = await self.transport_layer.make_membership_event(
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/synapse/federation/transport/client.py", line 243, in make_membership_event
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     content = await self.client.get_json(
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 1007, in get_json
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     response = await self._send_request_with_optional_trailing_slash(
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 391, in _send_request_with_optional_trailing_slash
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     response = await self._send_request(request, **send_request_args)
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/synapse/http/matrixfederationclient.py", line 479, in _send_request
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     limiter = await synapse.util.retryutils.get_retry_limiter(
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:   File "/usr/local/lib/python3.8/site-packages/synapse/util/retryutils.py", line 92, in get_retry_limiter
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]:     raise NotRetryingDestination(
Aug 09 13:39:51 benp-synapse matrix-synapse[2552]: synapse.util.retryutils.NotRetryingDestination: Not retrying server vector.modular.im.

@benparsons benparsons mentioned this issue Aug 9, 2021
Open
@MTRNord
Copy link
Contributor Author

MTRNord commented Aug 9, 2021

@benparsons looking at your trace its a different issue causing the same error? As mine is a missing a signature while yours is not retrying the server? 🤔

@anoadragon453
Copy link
Member

@benparsons hmm, that traceback shows Synapse raising a NotRetryingDestination, which is just what is raised when Synapse refuses to contact a server that it has failed to contact recently. The traceback doesn't contain the actually reason as to why it failed before. Do you perhaps have older logs of the initial failure?

@squahtx squahtx mentioned this issue Oct 20, 2021
Closed
@squahtx squahtx added the X-Needs-Info This issue is blocked awaiting information from the reporter label Oct 21, 2021
@TheGreatestJannet
Copy link

TheGreatestJannet commented Nov 6, 2021 •
edited
Loading

Is there a fix for this? I can't join #nextcloud:matrix.org at all.
I also get the following error:

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
StopIteration: {'ed25519:auto': FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x7f1dc06d7d90>, valid_until_ts=1576767829750), 'ed25519:a_RXGa': FetchKeyResult(verify_key=<nacl.signing.VerifyKey object at 0x7f1dc45c2100>, valid_until_ts=1636323490080)}

During handling of the above exception, another exception occurred:

Traceback (most recent call last):
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 576, in _try_destination_list
    return await callback(destination)
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 825, in send_request
    await concurrently_execute(
  File "/usr/local/lib/python3.8/site-packages/twisted/internet/defer.py", line 1661, in _inlineCallbacks
    result = current_context.run(gen.send, result)
  File "/usr/local/lib/python3.8/site-packages/synapse/util/async_helpers.py", line 192, in _concurrently_execute_inner
    await maybe_awaitable(func(value))
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 815, in _execute
    valid_pdu = await self._check_sigs_and_hash_and_fetch_one(
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_client.py", line 458, in _check_sigs_and_hash_and_fetch_one
    res = await self._check_sigs_and_hash(room_version, pdu)
  File "/usr/local/lib/python3.8/site-packages/synapse/federation/federation_base.py", line 70, in _check_sigs_and_hash
    if not check_event_content_hash(pdu):
  File "/usr/local/lib/python3.8/site-packages/synapse/crypto/event_signing.py", line 42, in check_event_content_hash
    name, expected_hash = compute_content_hash(event.get_pdu_json(), hash_algorithm)
  File "/usr/local/lib/python3.8/site-packages/synapse/crypto/event_signing.py", line 96, in compute_content_hash
    event_json_bytes = encode_canonical_json(event_dict)
  File "/usr/local/lib/python3.8/site-packages/canonicaljson.py", line 83, in encode_canonical_json
    s = _canonical_encoder.encode(json_object)
  File "/usr/local/lib/python3.8/json/encoder.py", line 199, in encode
    chunks = self.iterencode(o, _one_shot=True)
  File "/usr/local/lib/python3.8/json/encoder.py", line 257, in iterencode
    return _iterencode(o, 0)
  File "/usr/local/lib/python3.8/site-packages/canonicaljson.py", line 36, in _default
    raise TypeError(
TypeError: Object of type Decimal is not JSON serializable

Edit: Looks like it will be fixed in the next release.

@anoadragon453
Copy link
Member 

TypeError: Object of type Decimal is not JSON serializable

errors should be fixed in the next release of Synapse, v1.47.0. That, however, is a different error from what was originally posted, which was the result of event signature verification failure.

@richvdh
Copy link
Member

richvdh commented Jan 4, 2022

This issue seems to have become a bit of a pile-on for everyone who ever saw any sort of error joining any sort of room. I was able to join #emacs:matrix.org with no problem, so I think the original problem has probably been solved, and I'm going to close this.

If people have problems joining rooms, please create a new issue rather than piling onto one that sounds vaguely related - there are a lot of reasons a join can fail and it's much easier for us to close duplicate issues than it is to split out unrelated problems.

@richvdh richvdh closed this as completed Jan 4, 2022
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Assignees
No one assigned
Labels
A-Federated-Join joins over federation generally suck P3 (OBSOLETE: use S- labels.) Approved backlog: not yet scheduled, will accept patches S-Major Major functionality / product severely impaired, no satisfactory workaround. T-Defect Bugs, crashes, hangs, security vulnerabilities, or other reported issues. X-Needs-Info This issue is blocked awaiting information from the reporter
Projects
None yet
Milestone
No milestone
Development

No branches or pull requests