Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Provide list of all Synapse caches with default limits numbers and current sizes in bytes #8811

Open
MurzNN opened this issue Nov 25, 2020 · 4 comments
Labels
A-Docs things relating to the documentation T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. Z-Help-Wanted We know exactly how to fix this issue, and would be grateful for any contribution z-p2 (Deprecated Label)

Comments

@MurzNN
Copy link

MurzNN commented Nov 25, 2020

Description:
In homeserver.yaml we have caches section, where can fill cache factors - global, or per_cache.
But I can't find the way to lookup list of all cache names (best with short description) that use Synapse, with default values in numbers, and exact name for config file parameter.

For example, what is default number for get_users_who_share_room_with_user - 100, 1000, or 10000? If we will have the default numbers, we will can calculate result using simple math.

Other problem is cache name, for example, the *getEvent* cache must be named in yaml file as *getEvent*, getEvent, get_event, _get_event_ (with replacing * to _), or maybe event_cache_size - what is right version?

Also we need the way to lookup current cache size in bytes, not only in objects count, to understand which cache eats most of RAM, when we try to solve "Synapse eats all my RAM" problem.

P. S. And separate question about root parameter event_cache_size - is this limit for getEvent cache, and caches.global_factor value applies to it now? In yaml comment there are "Not affected by caches.global_factor." but I set it to 500K, and see in Prometheus that *getEvent* grows to 1600K with global_factor = 4.0!

Also in current homeserver.yaml template parameter event_cache_size located in root, not inside cache category, - is this right?

Regarding this change https://github.com/matrix-org/synapse/pull/6391/files/2619891343cc55b307ae36e6dda6247b2d6a5f73#diff-bfb2b1823a1f152109d0007dc9c8c6e83a99ea81ffa6edc08ecca5fc8b3475daR75 - event_cache_size variable in must be moved from the root to caches subcategory, right?

@MurzNN MurzNN changed the title Provide list of all caches with default limits numbers Provide list of all Synapse caches with default limits numbers Nov 25, 2020
@clokep clokep added A-Docs things relating to the documentation z-p2 (Deprecated Label) labels Nov 25, 2020
@callahad
Copy link
Contributor

It could be nice to implement this as a CLI command that prints the current defaults for everything, just to ensure it's always up-to-date.

We'd gladly review PRs that do this, but we're not sure we'll be able to implement that on this side in the near future.

@clokep clokep added the Z-Help-Wanted We know exactly how to fix this issue, and would be grateful for any contribution label Nov 30, 2020
@MurzNN
Copy link
Author

MurzNN commented Jan 3, 2021

I have found the list of cache names and per-cache max limits in Synapse prometheus metrics, here is cli command to lookup:

curl -s http://localhost:9102/_synapse/metrics | grep synapse_util_caches_cache_max_size | sed 's/[^"]*"\([^"]*\)"\}\(.*\)/\1\2/'

where http://localhost:9102/ is configured address for metrics in Synapse YAML:

  - type: metrics
    port: 9102
    bind_addresses:
      - '127.0.0.1'

And here is filter by current number of values per each cache:

curl -s http://localhost:9102/_synapse/metrics | grep synapse_util_caches_cache_size | sed 's/[^"]*"\([^"]*\)"\}\(.*\)/\1\2/'

Here is list of all default values (with caches.global_factor: 1.0):

regex_push_cache 50000.0
client_ip_last_seen 50000.0
*getEvent* 10240.0
device_id_exists 10000.0
_event_auth_cache 500000.0
*stateGroupCache* 50000.0
*stateGroupMembersCache* 500000.0
token_cache 10000.0
get_monthly_active_count 1000.0
user_last_seen_monthly_active 1000.0
get_monthly_active_count_by_service 1000.0
_get_server_verify_key 1000.0
_get_bare_e2e_cross_signing_keys 1000.0
get_user_by_access_token 1000.0
is_guest 1000.0
get_user_by_id 1000.0
get_user_filter 1000.0
is_support_user 1000.0
get_rooms_for_user_with_stream_ordering 500000.0
_get_presence_for_user 1000.0
get_room_version_id 10000.0
_get_state_group_for_event 50000.0
get_state_group_delta 10000.0
_get_rules_for_room 1000.0
get_push_rules_for_user 5000.0
get_push_rules_enabled_for_user 5000.0
_get_joined_users_from_context 100000.0
_get_joined_profile_from_event_id 10000.0
is_ignored_by 5000.0
get_global_account_data_by_type_for_user 5000.0
get_latest_event_ids_in_room 5000.0
matches_user_in_member_list 1000.0
get_users_in_room 100000.0
get_aliases_for_room 5000.0
_get_joined_hosts 10000.0
_get_joined_hosts_cache 10000.0
_get_linearized_receipts_for_room 1000.0
get_users_who_share_room_with_user 500000.0
count_e2e_one_time_keys 10000.0
get_e2e_unused_fallback_key_types 10000.0
is_room_blocked 10000.0
get_event_reference_hash 1000.0
is_user_erased 1000.0
get_retention_policy_for_room 1000.0
get_last_receipt_event_id_for_user 1000.0
get_unread_event_push_actions_by_room_for_user 5000.0
get_current_state_ids 100000.0
is_host_joined 10000.0
get_receipts_for_room 1000.0
get_receipts_for_user 1000.0
get_users_with_read_receipts_in_room 1000.0
get_device_list_last_stream_id_for_remote 10000.0
_get_cached_user_device 1000.0
get_cached_devices_for_user 1000.0
get_relations_for_event 1000.0
get_aggregation_groups_for_event 1000.0
get_invited_rooms_for_local_user 1000.0
get_room_summary 100000.0
get_earliest_token_for_stats 1000.0
get_user_in_directory 1000.0
get_forgotten_rooms_for_user 1000.0
get_applicable_edit 1000.0
get_account_data_for_room_and_type 5000.0
get_event_ordering 5000.0
get_account_data_for_user 1000.0
get_account_data_for_room 1000.0
generate_room_entry 1000.0
get_ratelimit_for_user 10000.0

So maybe we can simply reuse same function, that outputs this statistics for prometheus, in some command-line argument?

@MurzNN
Copy link
Author

MurzNN commented Jan 3, 2021

Also will be good to lookup memory usage size for each cache type, but this info seems missed in Prometheus metrics.

@MurzNN MurzNN changed the title Provide list of all Synapse caches with default limits numbers Provide list of all Synapse caches with default limits numbers and current sizes in bytes Jan 6, 2021
@MurzNN
Copy link
Author

MurzNN commented May 12, 2021

Seems here #9881 is implemented memory measurement for caches.

Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
A-Docs things relating to the documentation T-Task Refactoring, removal, replacement, enabling or disabling functionality, other engineering tasks. Z-Help-Wanted We know exactly how to fix this issue, and would be grateful for any contribution z-p2 (Deprecated Label)
Projects
None yet
Development

No branches or pull requests

4 participants