Skip to content
This repository has been archived by the owner on Apr 26, 2024. It is now read-only.

Add an admin API for users' media statistics #8700

Merged
merged 7 commits into from
Nov 5, 2020

Conversation

dklimpel
Copy link
Contributor

@dklimpel dklimpel commented Nov 1, 2020

Add GET /_synapse/admin/v1/statistics/users/media to get statisics about local media usage by users.
Related to #6094
It is the first API for statistics.
Goal is to avoid/reduce usage of sql queries like Wiki analyzing Synapse

Sorry for the long PR.

Pull Request Checklist

  • Pull request is based on the develop branch
  • Pull request includes a changelog file. The entry should:
    • Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
    • Use markdown where necessary, mostly for code blocks.
    • End with either a period (.) or an exclamation mark (!).
    • Start with a capital letter.
  • Pull request includes a sign off
  • Code style is correct (run the linters)

Signed-off-by: Dirk Klimpel dirk@klimpel.org

@dklimpel dklimpel marked this pull request as draft November 1, 2020 09:03
@dklimpel dklimpel force-pushed the admin_user_media_stats branch from d4ea72f to b83bede Compare November 1, 2020 11:37
@dklimpel dklimpel marked this pull request as ready for review November 1, 2020 11:57
@erikjohnston erikjohnston requested a review from a team November 2, 2020 16:21
Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Fantastic start, thanks for doing this! I have a few bits of feedback, but fundamentally it looks good.

It would be nice to be able to use this with external media repos as well, such as matrix-media-repo, but that will have to wait for a more robust communication channel between external media repos and the homeserver.

docs/admin_api/statistics.rst Outdated Show resolved Hide resolved
docs/admin_api/statistics.rst Outdated Show resolved Hide resolved
docs/admin_api/statistics.rst Outdated Show resolved Hide resolved
docs/admin_api/statistics.rst Outdated Show resolved Hide resolved
synapse/storage/databases/main/stats.py Outdated Show resolved Hide resolved
tests/rest/admin/test_statistics.py Outdated Show resolved Hide resolved
Comment on lines 449 to 463
def _set_displayname(self, user_id: str, accesss_token: str, displayname: str):
"""Set a displayname of a specific user
Args:
user_id: user ID of the user whose displayname should be set
access_token: Access token of the user whose displayname should be set
displayname: new displayname to be set
"""
request, channel = self.make_request(
"PUT",
"/profile/%s/displayname" % (user_id,),
content=json.dumps({"displayname": displayname}),
access_token=accesss_token,
)
self.render(request)
self.assertEqual(200, channel.code, msg=channel.json_body)
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

An alternative possibility is to add displayname as an argument to self.register_user, which should be relatively straightforward and would benefit other test cases.

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Which self.register_user do you mean? Does it call

synapse/tests/unittest.py

Lines 549 to 557 in 4fda58d

def register_user(self, username, password, admin=False):
"""
Register a user. Requires the Admin API be registered.
Args:
username (bytes/unicode): The user part of the new user.
password (bytes/unicode): The password of the new user.
admin (bool): Whether the user should be created as an admin
or not.
?
There is no chance to define a displayname.

Or do you mean I should implement this parameter at this point?

Copy link
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It's done.

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Apologies, I misread what this function did and assumed it called https://github.com/matrix-org/synapse/blob/develop/docs/admin_api/user_admin_api.rst#create-or-modify-account, which did already have a displayname parameter. I probably wouldn't have suggested it otherwise as it was considerably more work for you.

However, thank you for adding it to the shared secret registration API as well - that will benefit others even more!

docs/admin_api/statistics.rst Outdated Show resolved Hide resolved
docs/admin_api/statistics.rst Outdated Show resolved Hide resolved
docs/admin_api/statistics.rst Outdated Show resolved Hide resolved
Copy link
Member

@anoadragon453 anoadragon453 left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

lgtm, thank you!

tests/rest/admin/test_statistics.py Outdated Show resolved Hide resolved
docs/admin_api/statistics.md Outdated Show resolved Hide resolved
docs/admin_api/statistics.md Outdated Show resolved Hide resolved
docs/admin_api/statistics.md Outdated Show resolved Hide resolved
docs/admin_api/statistics.md Outdated Show resolved Hide resolved
docs/admin_api/statistics.md Outdated Show resolved Hide resolved
synapse/storage/databases/main/stats.py Outdated Show resolved Hide resolved
changelog.d/8700.feature Outdated Show resolved Hide resolved
@anoadragon453 anoadragon453 merged commit c3119d1 into matrix-org:develop Nov 5, 2020
@dklimpel dklimpel deleted the admin_user_media_stats branch November 5, 2020 19:05
netbsd-srcmastr pushed a commit to NetBSD/pkgsrc that referenced this pull request Nov 18, 2020
Synapse 1.23.0 (2020-11-18)
===========================

This release changes the way structured logging is configured. See the [upgrade notes](UPGRADE.rst#upgrading-to-v1230) for details.

**Note**: We are aware of a trivially exploitable denial of service vulnerability in versions of Synapse prior to 1.20.0. Complete details will be disclosed on Monday, November 23rd. If you have not upgraded recently, please do so.

Bugfixes
--------

- Fix a dependency versioning bug in the Dockerfile that prevented Synapse from starting. ([\#8767](matrix-org/synapse#8767))


Synapse 1.23.0rc1 (2020-11-13)
==============================

Features
--------

- Add a push rule that highlights when a jitsi conference is created in a room. ([\#8286](matrix-org/synapse#8286))
- Add an admin api to delete a single file or files that were not used for a defined time from server. Contributed by @dklimpel. ([\#8519](matrix-org/synapse#8519))
- Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. ([\#8539](matrix-org/synapse#8539))
- Support generating structured logs via the standard logging configuration. ([\#8607](matrix-org/synapse#8607), [\#8685](matrix-org/synapse#8685))
- Add an admin API to allow server admins to list users' pushers. Contributed by @dklimpel. ([\#8610](matrix-org/synapse#8610), [\#8689](matrix-org/synapse#8689))
- Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel. ([\#8647](matrix-org/synapse#8647))
- Add an admin API for local user media statistics. Contributed by @dklimpel. ([\#8700](matrix-org/synapse#8700))
- Add `displayname` to Shared-Secret Registration for admins. ([\#8722](matrix-org/synapse#8722))


Bugfixes
--------

- Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already. ([\#8455](matrix-org/synapse#8455))
- Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information. ([\#8580](matrix-org/synapse#8580))
- Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error. ([\#8620](matrix-org/synapse#8620))
- Fix email notifications for invites without local state. ([\#8627](matrix-org/synapse#8627))
- Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500. ([\#8628](matrix-org/synapse#8628))
- Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded. ([\#8632](matrix-org/synapse#8632))
- Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0. ([\#8643](matrix-org/synapse#8643))
- Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories. ([\#8682](matrix-org/synapse#8682))
- Fix bug that prevented Synapse from recovering after losing connection to the database. ([\#8726](matrix-org/synapse#8726))
- Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms. ([\#8728](matrix-org/synapse#8728))
- Fix PostgreSQL port script fails when DB has no backfilled events. Broke in v1.21.0. ([\#8729](matrix-org/synapse#8729))
- Fix PostgreSQL port script to correctly handle foreign key constraints. Broke in v1.21.0. ([\#8730](matrix-org/synapse#8730))
- Fix PostgreSQL port script so that it can be run again after a failure. Broke in v1.21.0. ([\#8755](matrix-org/synapse#8755))


Improved Documentation
----------------------

- Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk. ([\#8582](matrix-org/synapse#8582))
- Improve the sample configuration for single sign-on providers. ([\#8635](matrix-org/synapse#8635))
- Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs. ([\#8657](matrix-org/synapse#8657))
- Note support for Python 3.9. ([\#8665](matrix-org/synapse#8665))
- Minor updates to docs on running tests. ([\#8666](matrix-org/synapse#8666))
- Interlink prometheus/grafana documentation. ([\#8667](matrix-org/synapse#8667))
- Notes on SSO logins and media_repository worker. ([\#8701](matrix-org/synapse#8701))
- Document experimental support for running multiple event persisters. ([\#8706](matrix-org/synapse#8706))
- Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`. ([\#8714](matrix-org/synapse#8714))
- Migrate documentation `docs/admin_api/event_reports` to markdown. ([\#8742](matrix-org/synapse#8742))
- Add some helpful hints to the README for new Synapse developers. Contributed by @chagai95. ([\#8746](matrix-org/synapse#8746))


Internal Changes
----------------

- Optimise `/createRoom` with multiple invited users. ([\#8559](matrix-org/synapse#8559))
- Implement and use an `@lru_cache` decorator. ([\#8595](matrix-org/synapse#8595))
- Don't instansiate Requester directly. ([\#8614](matrix-org/synapse#8614))
- Type hints for `RegistrationStore`. ([\#8615](matrix-org/synapse#8615))
- Change schema to support access tokens belonging to one user but granting access to another. ([\#8616](matrix-org/synapse#8616))
- Remove unused OPTIONS handlers. ([\#8621](matrix-org/synapse#8621))
- Run `mypy` as part of the lint.sh script. ([\#8633](matrix-org/synapse#8633))
- Correct Synapse's PyPI package name in the OpenID Connect installation instructions. ([\#8634](matrix-org/synapse#8634))
- Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg. ([\#8636](matrix-org/synapse#8636))
- Fix typos and spelling errors in the code. ([\#8639](matrix-org/synapse#8639))
- Reduce number of OpenTracing spans started. ([\#8640](matrix-org/synapse#8640), [\#8668](matrix-org/synapse#8668), [\#8670](matrix-org/synapse#8670))
- Add field `total` to device list in admin API. ([\#8644](matrix-org/synapse#8644))
- Add more type hints to the application services code. ([\#8655](matrix-org/synapse#8655), [\#8693](matrix-org/synapse#8693))
- Tell Black to format code for Python 3.5. ([\#8664](matrix-org/synapse#8664))
- Don't pull event from DB when handling replication traffic. ([\#8669](matrix-org/synapse#8669))
- Abstract some invite-related code in preparation for landing knocking. ([\#8671](matrix-org/synapse#8671), [\#8688](matrix-org/synapse#8688))
- Clarify representation of events in logfiles. ([\#8679](matrix-org/synapse#8679))
- Don't require `hiredis` package to be installed to run unit tests. ([\#8680](matrix-org/synapse#8680))
- Fix typing info on cache call signature to accept `on_invalidate`. ([\#8684](matrix-org/synapse#8684))
- Fail tests if they do not await coroutines. ([\#8690](matrix-org/synapse#8690))
- Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`. ([\#8694](matrix-org/synapse#8694))
- Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8697](matrix-org/synapse#8697))
- Use Python 3.8 in Docker images by default. ([\#8698](matrix-org/synapse#8698))
- Remove the "draft" status of the Room Details Admin API. ([\#8702](matrix-org/synapse#8702))
- Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile. ([\#8705](matrix-org/synapse#8705))
- Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room. ([\#8708](matrix-org/synapse#8708))
- Add metrics the allow the local sysadmin to track 3PID `/requestToken` requests. ([\#8712](matrix-org/synapse#8712))
- Consolidate duplicated lists of purged tables that are checked in tests. ([\#8713](matrix-org/synapse#8713))
- Add some `mdui:UIInfo` element examples for `saml2_config` in the homeserver config. ([\#8718](matrix-org/synapse#8718))
- Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request. ([\#8719](matrix-org/synapse#8719))
- Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups. ([\#8752](matrix-org/synapse#8752))


Synapse 1.22.1 (2020-10-30)
===========================

Bugfixes
--------

- Fix a bug where an appservice may not be forwarded events for a room it was recently invited to. Broke in v1.22.0. ([\#8676](matrix-org/synapse#8676))
- Fix `Object of type frozendict is not JSON serializable` exceptions when using third-party event rules. Broke in v1.22.0. ([\#8678](matrix-org/synapse#8678))
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants