[NEW] Message retention policy and pruning

[vynmera]

Closes #6749
Closes #8321
Closes #9374
Closes #2700
Closes #2639
Closes #2355
Closes #1861
Closes #8757
Closes #7228
Closes #10870
Closes #6193
Closes #11299
Closes #11468
Closes #9317
Closes #11300 (will incorporate a fix to this PR's issue)
Closes #11046 (will incorporate a fix to this PR's issue)
Contributes to RocketChat/feature-requests#761
Contributes to #11475
...and possibly more!

This PR makes deleting messages (automatically and manually) a lot easier on Rocket.Chat.

• Implement a bulk message deletion notification, to quickly push large message deletions to users without reload
  • Use it in rooms.cleanHistory
  • Use it in user deletions
• Completely remove cleanChannelHistory as required by v0.67
  • Remove server method cleanChannelHistory
  • Remove REST API channels.cleanHistory
• Implement a sidebar option to clean history
  • Basic implementation
  • Allow excluding pinned messages
  • Allow attachment-only mode
  • Allow specifying user(s) to narrow down to
    • Also update REST API
    • Also update docs
  • Break the deletion into multiple different requests, so the client can keep track of progress
  • Clear animation / progress bar for deleting
• Retention policy
  • Global, set by admin
    • Global timer that runs every second and deletes messages over the set limit
      • Can change its timer's resolution to prevent insane CPU overhead
    • Admin can decide what room types to target (channels, groups and/or DMs)
    • Allow excluding pinned messages
    • Allow attachment-only mode
  • Per-channel, set by those with a new permission
    • Disabled when master switch off
    • Set in channel info
    • Can override global policy with a switch that requires edit-privileged-setting
    • Allow excluding pinned messages
    • Allow attachment-only mode
    • Uses same global timer for cleanup
  • Message at start of channel history / in channel info if there is a retention policy set
  • Message in channel info if there is a retention policy set on that channel specifically
• Make cleaning history also delete files (completely!)
  • Manual purging
  • Automatic purging
• Make other deletions also delete files
  • User deletion
    • Own messages
    • DMs with them's partner messages
  • Room deletion
• Cleanup
• Finish related docs
• Link to the docs in the settings

Please suggest any cool changes/additions! Any support is greatly appreciated.

Breaking change: This PR removes REST API endpoint channels.cleanHistory and Meteor callable cleanChannelHistory as per the protocol specified for them.

[vynmera]

A simple implementation for the flexbar is here! 🎉

[vynmera]

The flexbar now has a progress indicator, and sends the request in small pieces (it uses a new limit parameter), to keep track! 🎉

[vynmera]

A simple implementation of a global retention policy now exists! 🎉 It's configurable in Administration.

The example above looks like this in chat:

(Of course, most sane users would set it to something like 30 or 7 days, but short history like this is possible too and makes it easier to show how it works in a GIF)

[xmovu]

This is awesome, thank you so much!!!

[rodrigok]

Wow, awesome work @vynmera
Can't wait to review this PR 😄

[TehPeGaSuS]

This is top top top!
Great work @vynmera
🎉

[hExPY]

Aaawesome!!!!

[mddvul22]

Is it possible to have different maximum message age values for direct messages, private groups, and public channels? It would be nice to have different values for those three things.

[marzie82]

Great feature- many thanks !!!
Just one question- how do I get it? My Snap was updated two days ago- now I am on Version 0.66.3?
Any idea what's wrong?

[himpierre]

@marzie82 You need rocketchat 0.68.0.

[himpierre]

Ability to delete channels / dm automatically after x time with no message would be really nice.

[marzie82]

@himpierre Thanks ;)
can you please let me know when SNAP 0.68.0 will be released?

[chatnl]

As far as I can see, SNAP 0.68.0 is not available yet. Not in the stable update channel and also not in the candidate channel. The latest version is still 0.66.3. I would also advice not to use updates from the candidate channel in a production environment. Better wait until there is a stable version. Maybe development can let us know when they expect it to be published as a stable version?

[svennd]

Amazing feature, the only thing I would love to see is the ability to set rules per channel. Some channels don't require long-term storage, while others are nice to keep. Thanks for the great work !

[vynmera]

@svennd This is already possible with the per-channel rules: for every channel, if you have the edit-room-retention-policy permission, you can change the retention policy on a channel-to-channel basis (no policy, different policy, etc..) :)

[svennd]

I apologize, I clearly missed where to find this feature, super ! Perhaps other people not finding it :

[claures]

Hello,
Great feature does this also delete the file attachments?
-BR

[vynmera]

@claures Yes, it does! You can also make it only delete attachments.

[CH1PSET1]

Hello. Tell me, I have a version 0.68.3 server. Why does my automatic cleaning function not work for private messages? With what it can be connected? Global settings enable cleaning for channels, rooms and private messages. But it cleans only in channels and groups. In personal messages from the top it is written, Messages older than 14 days are automatically cropped here, but messages remain for an older period. In manual, if clean, it cleans.

And the second question. In the channel, in groups and private messages, if I choose to clear all messages, they are cleared in the web and desktop versions of the application, and in the mobile application the story remains, with what it too can be connected? And if I delete messages individually, they are deleted everywhere. Thank you.

[vynmera]

@CH1PSET1 As for the messages not being cleared automatically, there may be not enough activity in the channel: #11725 (comment) discusses this further. (I've started some tests in the background and will return with the results once I've tested some more, but this might be the cause of the issue)

As for prunes not showing up in the mobile apps, this PR implements a new type of Notification. So the mobile apps will have to be updated to reflect this change. The messages are gone, but the phone app doesn't know this because it not yet handles these new notifications. Once the mobile app is updated, this will work.

[maxdwit]

@vynmera thanks for bringing this into rocket.chat.

[denogio]

Will this be implemented in livechat also?

[GoetheG]

Hi guys. I wish I had the chance to decide for every channe whether a user can see historic messages. That's important for clubs for example. A chat can get messy and sometimes to informal and new guys may have a different wording. It would come in handy if I could set specific settings for a user. For example if I want to exclude a new added user in a channel to see previous chats.

[GoetheG]

This is what I am referring to:
RocketChat/feature-requests#640

[semartsal]

I apologize, I clearly missed where to find this feature, super ! Perhaps other people not finding it :

hi, anybody find this features ? I need to auto clear channel message on last 3-5 days
can't find this in 2.1.0 release (

[reetp]

hi, anybody find this features ? I need to auto clear channel message on last 3-5 days
can't find this in 2.1.0 release (

I don't believe you can do that. You can only clear messages 'older than'

You can open a NFR if you want:
https://github.com/RocketChat/feature-requests

[semartsal]

hi, anybody find this features ? I need to auto clear channel message on last 3-5 days
can't find this in 2.1.0 release (

I don't believe you can do that. You can only clear messages 'older than'

You can open a NFR if you want:
https://github.com/RocketChat/feature-requests

I need AUTO delete message older then 5 day for expamle, for some channel, not for all.
and on the screen exactly what I need, but a can't find this in channel prune properties
only delete "newer" or "older".
2.1.0 release include this prune property ?
thnx for any answer

[reetp]

Please read this.
https://rocket.chat/docs/administrator-guides/retention-policies/

If you can't find what you need then open a feature request. Just add a reference to this PR.

Obviously a PR with your code to add new functions is always welcome.

[semartsal]

I apologize, I clearly missed where to find this feature, super ! Perhaps other people not finding it :

i found this.
enable RP first, then go in bottom of Room Info. profit
tnx for this features!

[Gummikavalier]

A question regarding completeness of pruning functionality:

Does this also prune user data files created by Export Messages feature currently available for admins in RC 3.7.1?

The export feature offers the messages and attachments as a zip file by a direct link to the server.
For example: https://example.com/ufs/GridFS:UserDataFiles/cge2fwH2UFHJv12D33/2020-10-21-username-data.zip