/sync randomly tightloops
#15824
Comments
|
the logs at the beginning of the loop don't reveal much: |
|
I think that's related to #15082. |
MadLittleMods
added
A-Sync
|
Also seeing this in a local development environment with Synapse 1.85.2, doesn't appear to be related to #15082 because payloads are not empty. Element web's calls to 3 examples, notice they are all exactly the same: http://localhost:8008/_matrix/client/r0/sync?filter=0&timeout=30000&since=s1_447_0_1_1_1_1_36_0_1 http://localhost:8008/_matrix/client/r0/sync?filter=0&timeout=30000&since=s1_447_0_1_1_1_1_36_0_1 http://localhost:8008/_matrix/client/r0/sync?filter=0&timeout=30000&since=s1_447_0_1_1_1_1_36_0_1 |
The request URLs you have given us are also exactly the same. Synapse caches The since token in your examples is (I note that the logs in the description and in #15824 (comment) also repeat a request with the same since token.) I would guess that either
|
|
I have a similar issue on Synapse 1.88.0rc1 running locally on docker where sync (via Hydrogen) is tight looped:
|
|
I noticed this yesterday and today. Turning off presence seems to stop the continuous sync. |
|
Same problem on |
|
Repro steps involve using docker/sqlite synapse with MAS enabled and hooked up correctly. For some reason, sending messages into a room can fail: When this happens, the next /sync request will always tightloop. The problem is this line is always being hit https://github.com/matrix-org/synapse/blob/v1.92.1/synapse/handlers/sync.py#L1993 ... because this is always being hit: # _cache is not valid at or before the earliest known stream position, so
# return that the entity has changed.
if stream_pos <= self._earliest_known_stream_pos:
self.metrics.inc_misses()
return True.. because |
|
fwiw, I see this without sending messages. MAS is not configured. sqlite is used, though. It will be sitting completely idle for several minutes and then suddenly tightloop. No user interaction required. |
|
@DMRobertson the sync token is not advanced by Synapse, there's a mistake in my example above. The query string returned by synapse is provided by element but Synapse returns the same since token that was passed in the query string. @kegsay we tried disabling presence and that seems to resolve the issue for a moment but then it returned |
I had the same experience yesterday but I can't reproduce anymore. It suddenly started to loop extremely fast, even if I restarted the containers or refreshed the browser page, until I deactivated presence. |
|
Hopefully #16540 fixes this |
|
Sounds like it hasn't #16540 (comment) |
|
@erikjohnston should #16540 be reverted? |
No, that fix is still correct, but not enough to fix this it seems. |
|
Anything we can do to help move this along? It's quite problematic for us. |
|
@thebalaa if you have an example request URL and response body of a |
|
Here's two separate client responses from sync: From Element: Request URL Response {
"next_batch": "s385_307_0_11_70_1_1_19_0_1",
"device_lists": {
"changed": [
"@admin:localhost"
]
},
"device_one_time_keys_count": {
"signed_curve25519": 50
},
"org.matrix.msc2732.device_unused_fallback_key_types": [
"signed_curve25519"
],
"device_unused_fallback_key_types": [
"signed_curve25519"
]
}From a Python client (matrix-nio): Response {
"next_batch": "s385_312_0_11_70_1_1_19_0_1",
"device_lists": {
"changed": [
"@admin:localhost"
]
},
"device_one_time_keys_count": {
"signed_curve25519": 50
},
"org.matrix.msc2732.device_unused_fallback_key_types": [
"signed_curve25519"
],
"device_unused_fallback_key_types": [
"signed_curve25519"
]
}I'm running Synapse version Here's my # Configuration file for Synapse.
#
# This is a YAML file: see [1] for a quick introduction. Note in particular
# that *indentation is important*: all the elements of a list or dictionary
# should have the same indentation.
#
# [1] https://docs.ansible.com/ansible/latest/reference_appendices/YAMLSyntax.html
#
# For more information on how to configure Synapse, including a complete accounting of
# each option, go to docs/usage/configuration/config_documentation.md or
# https://matrix-org.github.io/synapse/latest/usage/configuration/config_documentation.html
server_name: "localhost"
pid_file: /data/homeserver.pid
listeners:
- port: 8008
tls: false
type: http
x_forwarded: true
resources:
- names: [client, federation]
compress: false
database:
name: sqlite3
args:
database: /data/homeserver.db
log_config: "/data/localhost.log.config"
media_store_path: /data/media_store
registration_shared_secret: "NKeqBJ3@=aEpq@m6&tyg,7Mp.PHsS=SKH#0gJWVH*Nhig0H9nt"
report_stats: false
macaroon_secret_key: "mRL:x_0p*VuKb2ZAio,^h&YCY77ISYs.OKr*+PBM:E0XJz:ZzV"
form_secret: "gr1E3TptS,_v-UfE6vN+l*dgc6:eSk;V3S6ZYm6B:D30O.St-h"
signing_key_path: "/data/localhost.signing.key"
trusted_key_servers:
- server_name: "matrix.org"
# vim:ft=yamlI'm able to reproduce this issue by doing the following:
I was messing around a little more and found that when my client sync is doing the tight loop, if I exec into the synapse container and create another user with |
|
This is particularly frustrating in test environments where it seems easily reproducible due to it being a low traffic environment. I'm guessing actual deployments see this less because new events are somehow able to unwedge things. In my scenario, the error on foreign key constraint will reliably reproduce this bug, but it's a convoluted setup as it involves MAS. I'll see if I can dig into this more this week. |
|
I used the account and password in the Mjolnir configuration file to log in to Element-web, which caused a large number of never-ending _matrix/client/v3/sync requests in the console. 😂 |
Description
In my test environment for matrix-media-repo I've got a localhost homeserver running on sqlite in docker which suddenly returns immediately to
/syncrequests. This causes Element to dutifully send the next request, to which Synapse responds immediately, and repeat.Steps to reproduce
Unclear.
Homeserver
localhost
Synapse Version
v1.85.2
Installation Method
Docker (matrixdotorg/synapse)
Database
sqlite3
Workers
Single process
Platform
Running in Docker.
Configuration
Relevant log output
Anything else that would be useful to know?
Restarting the Docker Compose stack can sometimes help, but it eventually tightloops again.
The text was updated successfully, but these errors were encountered: