Add an admin api to delete local media. #8519

dklimpel · 2020-10-11T12:39:20Z

Add DELETE /_synapse/admin/v1/media/<server_name>/<media_id> to delete
a single file from server.

It is the first step. An api with timestamp (older than) is possible later. I will do small stepswith small PR.

Pull Request Checklist

Pull request is based on the develop branch
Pull request includes a changelog file. The entry should:
- Be a short description of your change which makes sense to users. "Fixed a bug that prevented receiving messages from other servers." instead of "Moved X method from EventStore to EventWorkerStore.".
- Use markdown where necessary, mostly for code blocks.
- End with either a period (.) or an exclamation mark (!).
- Start with a capital letter.
Pull request includes a sign off
Code style is correct (run the linters)

Signed-off-by: Dirk Klimpel dirk@klimpel.org

related to: matrix-org#6459 Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server

anoadragon453

Thanks for this! I'm afraid I'm a bit hung up on a response to #8519 (comment) before I go any further though.

docs/admin_api/media_admin_api.md

synapse/rest/admin/media.py

docs/admin_api/media_admin_api.md

anoadragon453

My idea was to create a small PR. I can add more functions in this PR.

Sorry, I was a bit unclear with my original comment. I was originally thinking of just having an API that would accept multiple media IDs instead of the single media ID API that was originally proposed.

However, after a bit of discussion in #synapse-dev I think what you've done is a better solution. Between the time/size-based endpoint and deleting an entire room, I think most of the usecases for batch-deleting media are taken care of. Thus this coupled with a single media ID delete endpoint makes sense to me 👍

Lots of wording suggestions below, plus a question. Thanks!

anoadragon453 · 2020-10-19T13:14:54Z

docs/admin_api/media_admin_api.md

-* `server_name` - The name of your local server (e.g `matrix.org`)
-* `media_id` - The ID of the media (e.g `abcdefghijklmnopqrstuvwx`)
+* ``server_name``: string - The name of your local server (e.g ``matrix.org``)
+* ``media_id``: string - The ID of the media (e.g ``abcdefghijklmnopqrstuvwx``)


Although it renders fine, we typically use single backticks in markdown files.

I have done an update.

docs/admin_api/media_admin_api.md

anoadragon453 · 2020-10-19T14:19:45Z

synapse/storage/databases/main/media_repository.py

+        sql = """
+            SELECT media_id
+            FROM local_media_repository AS lmr
+            WHERE last_access_ts < ? and media_length > ?


What about media that is never accessed? As far as I can tell this will have a NULL value in the DB. Perhaps in this case we can use created_ts instead?

anoadragon453 · 2020-10-19T14:31:35Z

changelog.d/8519.feature

@@ -0,0 +1 @@
+Add an admin api `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server. Contributed by @dklimpel.


This needs a small update.

synapse/rest/media/v1/media_repository.py

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

ufgkirk · 2020-10-23T09:48:07Z

Thanks for working on this, I think this is an important feature. Say for example someone uploaded a copyrighted file and a DMCA was issued for a specific file link on the media server, the ability to remove that file from the server is important. :)

anoadragon453

Apologies for taking a while to get back to this one. This lgtm, thanks!

synapse/rest/media/v1/media_repository.py

tests/rest/admin/test_media.py

synapse/rest/media/v1/media_repository.py

@dklimpel

Synapse 1.23.0 (2020-11-18) =========================== This release changes the way structured logging is configured. See the [upgrade notes](UPGRADE.rst#upgrading-to-v1230) for details. **Note**: We are aware of a trivially exploitable denial of service vulnerability in versions of Synapse prior to 1.20.0. Complete details will be disclosed on Monday, November 23rd. If you have not upgraded recently, please do so. Bugfixes -------- - Fix a dependency versioning bug in the Dockerfile that prevented Synapse from starting. ([\#8767](matrix-org/synapse#8767)) Synapse 1.23.0rc1 (2020-11-13) ============================== Features -------- - Add a push rule that highlights when a jitsi conference is created in a room. ([\#8286](matrix-org/synapse#8286)) - Add an admin api to delete a single file or files that were not used for a defined time from server. Contributed by @dklimpel. ([\#8519](matrix-org/synapse#8519)) - Split admin API for reported events (`GET /_synapse/admin/v1/event_reports`) into detail and list endpoints. This is a breaking change to #8217 which was introduced in Synapse v1.21.0. Those who already use this API should check their scripts. Contributed by @dklimpel. ([\#8539](matrix-org/synapse#8539)) - Support generating structured logs via the standard logging configuration. ([\#8607](matrix-org/synapse#8607), [\#8685](matrix-org/synapse#8685)) - Add an admin API to allow server admins to list users' pushers. Contributed by @dklimpel. ([\#8610](matrix-org/synapse#8610), [\#8689](matrix-org/synapse#8689)) - Add an admin API `GET /_synapse/admin/v1/users/<user_id>/media` to get information about uploaded media. Contributed by @dklimpel. ([\#8647](matrix-org/synapse#8647)) - Add an admin API for local user media statistics. Contributed by @dklimpel. ([\#8700](matrix-org/synapse#8700)) - Add `displayname` to Shared-Secret Registration for admins. ([\#8722](matrix-org/synapse#8722)) Bugfixes -------- - Fix fetching of E2E cross signing keys over federation when only one of the master key and device signing key is cached already. ([\#8455](matrix-org/synapse#8455)) - Fix a bug where Synapse would blindly forward bad responses from federation to clients when retrieving profile information. ([\#8580](matrix-org/synapse#8580)) - Fix a bug where the account validity endpoint would silently fail if the user ID did not have an expiration time. It now returns a 400 error. ([\#8620](matrix-org/synapse#8620)) - Fix email notifications for invites without local state. ([\#8627](matrix-org/synapse#8627)) - Fix handling of invalid group IDs to return a 400 rather than log an exception and return a 500. ([\#8628](matrix-org/synapse#8628)) - Fix handling of User-Agent headers that are invalid UTF-8, which caused user agents of users to not get correctly recorded. ([\#8632](matrix-org/synapse#8632)) - Fix a bug in the `joined_rooms` admin API if the user has never joined any rooms. The bug was introduced, along with the API, in v1.21.0. ([\#8643](matrix-org/synapse#8643)) - Fix exception during handling multiple concurrent requests for remote media when using multiple media repositories. ([\#8682](matrix-org/synapse#8682)) - Fix bug that prevented Synapse from recovering after losing connection to the database. ([\#8726](matrix-org/synapse#8726)) - Fix bug where the `/_synapse/admin/v1/send_server_notice` API could send notices to non-notice rooms. ([\#8728](matrix-org/synapse#8728)) - Fix PostgreSQL port script fails when DB has no backfilled events. Broke in v1.21.0. ([\#8729](matrix-org/synapse#8729)) - Fix PostgreSQL port script to correctly handle foreign key constraints. Broke in v1.21.0. ([\#8730](matrix-org/synapse#8730)) - Fix PostgreSQL port script so that it can be run again after a failure. Broke in v1.21.0. ([\#8755](matrix-org/synapse#8755)) Improved Documentation ---------------------- - Instructions for Azure AD in the OpenID Connect documentation. Contributed by peterk. ([\#8582](matrix-org/synapse#8582)) - Improve the sample configuration for single sign-on providers. ([\#8635](matrix-org/synapse#8635)) - Fix the filepath of Dex's example config and the link to Dex's Getting Started guide in the OpenID Connect docs. ([\#8657](matrix-org/synapse#8657)) - Note support for Python 3.9. ([\#8665](matrix-org/synapse#8665)) - Minor updates to docs on running tests. ([\#8666](matrix-org/synapse#8666)) - Interlink prometheus/grafana documentation. ([\#8667](matrix-org/synapse#8667)) - Notes on SSO logins and media_repository worker. ([\#8701](matrix-org/synapse#8701)) - Document experimental support for running multiple event persisters. ([\#8706](matrix-org/synapse#8706)) - Add information regarding the various sources of, and expected contributions to, Synapse's documentation to `CONTRIBUTING.md`. ([\#8714](matrix-org/synapse#8714)) - Migrate documentation `docs/admin_api/event_reports` to markdown. ([\#8742](matrix-org/synapse#8742)) - Add some helpful hints to the README for new Synapse developers. Contributed by @chagai95. ([\#8746](matrix-org/synapse#8746)) Internal Changes ---------------- - Optimise `/createRoom` with multiple invited users. ([\#8559](matrix-org/synapse#8559)) - Implement and use an `@lru_cache` decorator. ([\#8595](matrix-org/synapse#8595)) - Don't instansiate Requester directly. ([\#8614](matrix-org/synapse#8614)) - Type hints for `RegistrationStore`. ([\#8615](matrix-org/synapse#8615)) - Change schema to support access tokens belonging to one user but granting access to another. ([\#8616](matrix-org/synapse#8616)) - Remove unused OPTIONS handlers. ([\#8621](matrix-org/synapse#8621)) - Run `mypy` as part of the lint.sh script. ([\#8633](matrix-org/synapse#8633)) - Correct Synapse's PyPI package name in the OpenID Connect installation instructions. ([\#8634](matrix-org/synapse#8634)) - Catch exceptions during initialization of `password_providers`. Contributed by Nicolai Søborg. ([\#8636](matrix-org/synapse#8636)) - Fix typos and spelling errors in the code. ([\#8639](matrix-org/synapse#8639)) - Reduce number of OpenTracing spans started. ([\#8640](matrix-org/synapse#8640), [\#8668](matrix-org/synapse#8668), [\#8670](matrix-org/synapse#8670)) - Add field `total` to device list in admin API. ([\#8644](matrix-org/synapse#8644)) - Add more type hints to the application services code. ([\#8655](matrix-org/synapse#8655), [\#8693](matrix-org/synapse#8693)) - Tell Black to format code for Python 3.5. ([\#8664](matrix-org/synapse#8664)) - Don't pull event from DB when handling replication traffic. ([\#8669](matrix-org/synapse#8669)) - Abstract some invite-related code in preparation for landing knocking. ([\#8671](matrix-org/synapse#8671), [\#8688](matrix-org/synapse#8688)) - Clarify representation of events in logfiles. ([\#8679](matrix-org/synapse#8679)) - Don't require `hiredis` package to be installed to run unit tests. ([\#8680](matrix-org/synapse#8680)) - Fix typing info on cache call signature to accept `on_invalidate`. ([\#8684](matrix-org/synapse#8684)) - Fail tests if they do not await coroutines. ([\#8690](matrix-org/synapse#8690)) - Improve start time by adding an index to `e2e_cross_signing_keys.stream_id`. ([\#8694](matrix-org/synapse#8694)) - Re-organize the structured logging code to separate the TCP transport handling from the JSON formatting. ([\#8697](matrix-org/synapse#8697)) - Use Python 3.8 in Docker images by default. ([\#8698](matrix-org/synapse#8698)) - Remove the "draft" status of the Room Details Admin API. ([\#8702](matrix-org/synapse#8702)) - Improve the error returned when a non-string displayname or avatar_url is used when updating a user's profile. ([\#8705](matrix-org/synapse#8705)) - Block attempts by clients to send server ACLs, or redactions of server ACLs, that would result in the local server being blocked from the room. ([\#8708](matrix-org/synapse#8708)) - Add metrics the allow the local sysadmin to track 3PID `/requestToken` requests. ([\#8712](matrix-org/synapse#8712)) - Consolidate duplicated lists of purged tables that are checked in tests. ([\#8713](matrix-org/synapse#8713)) - Add some `mdui:UIInfo` element examples for `saml2_config` in the homeserver config. ([\#8718](matrix-org/synapse#8718)) - Improve the error message returned when a remote server incorrectly sets the `Content-Type` header in response to a JSON request. ([\#8719](matrix-org/synapse#8719)) - Speed up repeated state resolutions on the same room by caching event ID to auth event ID lookups. ([\#8752](matrix-org/synapse#8752)) Synapse 1.22.1 (2020-10-30) =========================== Bugfixes -------- - Fix a bug where an appservice may not be forwarded events for a room it was recently invited to. Broke in v1.22.0. ([\#8676](matrix-org/synapse#8676)) - Fix `Object of type frozendict is not JSON serializable` exceptions when using third-party event rules. Broke in v1.22.0. ([\#8678](matrix-org/synapse#8678))

dklimpel · 2021-01-18T07:00:23Z

Can be a basis for #6832

dklimpel added 2 commits October 11, 2020 14:33

Add an admin api to delete local media.

8e63746

related to: matrix-org#6459 Add `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server

add newsfile

1dc57c7

dklimpel marked this pull request as ready for review October 11, 2020 12:49

clokep requested a review from a team October 13, 2020 11:45

anoadragon453 suggested changes Oct 14, 2020

View reviewed changes

dklimpel marked this pull request as draft October 14, 2020 13:23

clokep added the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Oct 15, 2020

dklimpel added 2 commits October 16, 2020 20:35

Add additional api for request by date or size

d8167de

tox

e853bc5

dklimpel marked this pull request as ready for review October 16, 2020 19:13

dklimpel requested a review from anoadragon453 October 16, 2020 19:13

anoadragon453 suggested changes Oct 19, 2020

View reviewed changes

dklimpel and others added 3 commits October 19, 2020 20:15

Apply suggestions from code review - Part 1

be9ee45

Co-authored-by: Andrew Morgan <1342360+anoadragon453@users.noreply.github.com>

Apply suggestions from code review - Part 2

542a11e

Add query and test for media that is never accessed

1cb5c44

dklimpel requested a review from anoadragon453 October 19, 2020 20:45

ufgkirk mentioned this pull request Oct 23, 2020

Delete file attachments if message is removed. #8638

Closed

anoadragon453 approved these changes Oct 26, 2020

View reviewed changes

synapse/rest/media/v1/media_repository.py Outdated Show resolved Hide resolved

tests/rest/admin/test_media.py Outdated Show resolved Hide resolved

synapse/rest/media/v1/media_repository.py Outdated Show resolved Hide resolved

Apply suggestions from code review

3b9a691

anoadragon453 removed the X-Awaiting-Changes A contributed PR which needs changes and re-review before it can be merged label Oct 26, 2020

anoadragon453 merged commit 49d72de into matrix-org:develop Oct 26, 2020

anoadragon453 mentioned this pull request Oct 26, 2020

We should have an admin api to delete local media #6459

Closed

dklimpel deleted the admin_delete_media branch October 26, 2020 17:03

anoadragon453 mentioned this pull request Oct 26, 2020

Need to be able to safely cleanup local content #3479

Closed

xster mentioned this pull request Jan 30, 2021

Limit size of room storage (files and history) in Matrix server #2315

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Add an admin api to delete local media. #8519

Add an admin api to delete local media. #8519

dklimpel commented Oct 11, 2020 •

edited

Loading

anoadragon453 left a comment •

edited

Loading

anoadragon453 left a comment

anoadragon453 Oct 19, 2020

dklimpel Oct 19, 2020

anoadragon453 Oct 19, 2020

anoadragon453 Oct 19, 2020

ufgkirk commented Oct 23, 2020

anoadragon453 left a comment

dklimpel commented Jan 18, 2021

		@@ -0,0 +1 @@
		Add an admin api `DELETE /_synapse/admin/v1/media/<server_name>/<media_id>` to delete a single file from server. Contributed by @dklimpel.

Add an admin api to delete local media. #8519

Add an admin api to delete local media. #8519

Conversation

dklimpel commented Oct 11, 2020 • edited Loading

Pull Request Checklist

anoadragon453 left a comment • edited Loading

Choose a reason for hiding this comment

anoadragon453 left a comment

Choose a reason for hiding this comment

anoadragon453 Oct 19, 2020

Choose a reason for hiding this comment

dklimpel Oct 19, 2020

Choose a reason for hiding this comment

anoadragon453 Oct 19, 2020

Choose a reason for hiding this comment

anoadragon453 Oct 19, 2020

Choose a reason for hiding this comment

ufgkirk commented Oct 23, 2020

anoadragon453 left a comment

Choose a reason for hiding this comment

dklimpel commented Jan 18, 2021

dklimpel commented Oct 11, 2020 •

edited

Loading

anoadragon453 left a comment •

edited

Loading