Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

New: Use database change streams when available #18892

Merged
merged 26 commits into from
Sep 22, 2020
Merged

Conversation

rodrigok
Copy link
Member

@rodrigok rodrigok commented Sep 14, 2020

Proposed changes

The intent of this PR is to implement our own solution of oplog tailing in replacement of the meteor's one to prevent the internal processing made by meteor. With this change we implemented the possibility to use change streams which is the most efficient way to listen for data changes in database but it's only available from mongodb version 3.6 and when using wiredtiguer storage type.

To make this change we had to:

  1. Change all the places where we were still using observers to use our event change from the collections that we've implemented some time ago when we started this movement to move away from the meteor's reactivity solutions.
  2. Disable the default presence's and instance's monitor to make the reactivity using the collection's change event mentioned before.
  3. Override the meteor's internal mongodb observer to prevent database polling for the internal observers it has.
  4. Reimplement the reactivity for the internal meteor users' observer to close connections when the user's token related to that connection was removed.
  5. Implement a class to replace the oplog handler that we were listening to fire the change events, now implementing the option to open our own oplog tailing or use change stream when available.

It's possible to configure the behavior with 2 environment variables:

  • IGNORE_CHANGE_STREAM=true: which will force the usage of our oplog tailing implementation in place of the change stream.
  • USE_NATIVE_OPLOG=true: which will restore the usage of the meteor's internal oplog tailing.

Issue(s)

How to test or reproduce

Screenshots

Types of changes

  • Bugfix (non-breaking change which fixes an issue)
  • Improvement (non-breaking change which improves a current function)
  • New feature (non-breaking change which adds functionality)
  • Breaking change (fix or feature that would cause existing functionality to not work as expected)
  • Hotfix (a major bugfix that has to be merged asap)
  • Documentation Update (if none of the other choices apply)

Checklist

  • I have read the CONTRIBUTING doc
  • I have signed the CLA
  • Lint and unit tests pass locally with my changes
  • I have added tests that prove my fix is effective or that my feature works (if applicable)
  • I have added necessary documentation (if applicable)

Changelog

Further comments

@rodrigok rodrigok marked this pull request as ready for review September 18, 2020 00:15
@rodrigok rodrigok added this to the 3.7.0 milestone Sep 18, 2020
# Conflicts:
#	server/publications/settings/index.js
app/settings/server/observer.js Outdated Show resolved Hide resolved
app/settings/server/raw.js Show resolved Hide resolved
app/search/server/events/events.js Outdated Show resolved Hide resolved
Copy link
Member

@sampaiodiego sampaiodiego left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  • when running a mongodb version not compatible with change streams, if I don't set MONGO_OPLOG_URL I get the following error:
(node:29097) UnhandledPromiseRejectionWarning: Error: URL malformed, cannot be parsed
    at module.exports (/Users/diegosampaio/dev/Rocket.Chat/node_modules/mongodb/lib/url_parser.js:18:21)
    at internal/util.js:297:30
    at new Promise (<anonymous>)
    at internal/util.js:296:12
    at app/models/server/models/_oplogHandle.ts:35:27
    at /Users/diegosampaio/.meteor/packages/promise/.0.11.2.1xavwef.9sbp++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/meteor-promise/fiber_pool.js:43:40

@sampaiodiego sampaiodiego merged commit 707aa1f into develop Sep 22, 2020
@sampaiodiego sampaiodiego deleted the change-streams branch September 22, 2020 03:03
@ankar84
Copy link

ankar84 commented Sep 22, 2020

Looks like breaking changes improve performance a lot.
Good job!

@sampaiodiego sampaiodiego mentioned this pull request Sep 29, 2020
ear-dev pushed a commit to WideChat/Rocket.Chat that referenced this pull request Oct 5, 2020
* Bump version to 3.7.0-develop

* Do not use deprecated express method (RocketChat#18686)

* Replace assets copy on postinstall with symlinks (RocketChat#18707)

* Update Meteor to 1.11 (RocketChat#18754)

* Update Meteor to 1.11

* Update Node version

* Fix livechat view creation

* LingoHub Update 🚀 (RocketChat#18761)

Manual push by LingoHub User: Diego Sampaio.
Project: Rocket.Chat

Made with ❤️ by https://lingohub.com

Co-authored-by: Diego Sampaio <chinello@gmail.com>

* Fix french translations (RocketChat#18746)

Replace [Nom du site] by [Site_Name]...

* Refactor: Omnichannel Realtime Monitoring (RocketChat#18666)

Co-authored-by: Guilherme Gazzo <guilherme@gazzo.xyz>

* [FIX] Create Custom OAuth services from environment variables (RocketChat#17377)

* Fix saveRoomSettings method complexity (RocketChat#18840)

* [FIX] e.sendToBottomIfNecessaryDebounced is not a function (RocketChat#18834)

* Log WebDav upload errors (RocketChat#18849)

* [FIX] File upload (Avatars, Emoji, Sounds) (RocketChat#18841)

* [FIX] IE11 support livechat widget  (RocketChat#18850)

* [FIX] Can't change password (RocketChat#18836)

* [FIX] Admin user blank page (RocketChat#18851)

* [FIX][ENTERPRISE] Omnichannel service status switching to unavailable (RocketChat#18835)

* [FIX] User can't invite or join other Omnichannel rooms (RocketChat#18852)

* [FIX] Omnichannel Current Chats open status filter not working (RocketChat#18795)

* [FIX] Jitsi call start updating subscriptions (RocketChat#18837)

* [FIX] Showing alerts during setup wizard (RocketChat#18862)

* Remember users' 2FA right after registration

* Change register server to false by default

* [FIX] Omnichannel Current Chats open status filter not working (RocketChat#18795)

* [FIX][ENTERPRISE] Omnichannel service status switching to unavailable (RocketChat#18835)

* [FIX] File upload (Avatars, Emoji, Sounds) (RocketChat#18841)

* [FIX] IE11 support livechat widget  (RocketChat#18850)

* [FIX] Admin user blank page (RocketChat#18851)

* [FIX] User can't invite or join other Omnichannel rooms (RocketChat#18852)

* [FIX] Showing alerts during setup wizard (RocketChat#18862)

* Remember users' 2FA right after registration

* Change register server to false by default

* Bump version to 3.6.1

* LingoHub based on develop (RocketChat#18828)

* LingoHub Update 🚀

Manual push by LingoHub User: Diego Sampaio.
Project: Rocket.Chat

Made with ❤️ by https://lingohub.com

* LingoHub Update 🚀

Manual push by LingoHub User: Diego Sampaio.
Project: Rocket.Chat

Made with ❤️ by https://lingohub.com

Co-authored-by: Diego Sampaio <chinello@gmail.com>

* [FIX] invite-all-from and invite-all-to commands don't work with multibyte room names (RocketChat#18919)

* [FIX] If there is `ufs` somewhere in url the request to api always returns 404 (RocketChat#18874)

* chore(packages): add fixed version of ufs

* test(e2e): test usernames with 'ufs'

* test(ufs-router): remove expect for header

* test: turn test callback synchronous

* test: fix async to use done

* chore(packages/meteor-jalik-ufs): add ufs package

* Revert "test: fix async to use done"

This reverts commit 6276e0b.

* Revert "test: turn test callback synchronous"

This reverts commit 2af11bb.

* Revert "test(ufs-router): remove expect for header"

This reverts commit 2c4eeb0.

* [FIX] "Save to WebDav" not working (RocketChat#18883)

* [FIX] Read receipts showing blank names and not marking messages as read (RocketChat#18918)

* [FIX] Non-upload requests being passed to UFS proxy middleware (RocketChat#18931)

* fix(ufs-proxy): correct routing pattern

* chore(ufs-proxy): add deprecation console warning

* [FIX] Version update check cron job (RocketChat#18916)

* [FIX] Ignore User action from user card (RocketChat#18866)

* [FIX] Custom fields required if minLength set and no text typed (RocketChat#18838)

* [FIX] Dutch: add translations for missing variables (RocketChat#18814)

* [FIX] French: Add missing __online__ var (RocketChat#18813)

* [FIX] Deactivate users that are the last owner of a room using REST API (RocketChat#18864)

* test: add e2e tests for REST API user deactivation

* fix(app): read confirmRelinquish from HTTP request

* chore(app): remove unnecessary console.log

* [FIX] Show custom fields of invalid type (RocketChat#18794)

* [FIX] Create Custom OAuth services from environment variables (RocketChat#17377)

* [FIX] Read receipts showing blank names and not marking messages as read (RocketChat#18918)

* [FIX] Version update check cron job (RocketChat#18916)

* [FIX] Ignore User action from user card (RocketChat#18866)

* [FIX] Deactivate users that are the last owner of a room using REST API (RocketChat#18864)

* test: add e2e tests for REST API user deactivation

* fix(app): read confirmRelinquish from HTTP request

* chore(app): remove unnecessary console.log

* [FIX] Show custom fields of invalid type (RocketChat#18794)

* Bump version to 3.6.2

* [FIX] Stop adding push messages to queue if push is disabled (RocketChat#18830)

* [NEW] "Room avatar changed" system messages (RocketChat#18839)

* [FIX] PDF not rendering (RocketChat#18956)

* [FIX] Spurious expert role in startup data (RocketChat#18667)

* [FIX] Open room after guest registration (RocketChat#18755)

* [FIX] Block user action (RocketChat#18950)

* Refactor: Omnichannel Analytics (RocketChat#18766)

* [FIX] Deactivated users show as offline (RocketChat#18767)

* [FIX] Reaction buttons not behaving properly (RocketChat#18832)

* Refactor: Message Audit page & Audit logs (RocketChat#18808)

* [FIX] "Download my data" popup showing HTML code. (RocketChat#18947)

* [IMPROVE] Move jump to message outside menu (RocketChat#18928)

* [FIX] Users not being able to activate/deactivate E2E in DMs (RocketChat#18943)

* Refactor: Admin permissions page (RocketChat#18932)

Co-authored-by: Guilherme Gazzo <guilhermegazzo@gmail.com>

* [IMPROVEMENT] Add "Allow_Save_Media_to_Gallery" setting to mobile (RocketChat#18875)

* Bump lodash.merge from 4.6.1 to 4.6.2 (RocketChat#18800)

* [FIX] Errors in LDAP avatar sync preventing login (RocketChat#18948)

* [NEW][Apps] Add support for new livechat guest's and room's events (RocketChat#18946)

* Add support for new livechat's guest and room events

* Update trigger calls

* Update Apps-Engine version

Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com>

* [NEW][Apps] Add a Livechat API - setCustomFields (RocketChat#18912)

* Map livechatData to customFields

* Add livechatData field to apps converter

Co-authored-by: Thassio Victor <tvmcarvalho@gmail.com>
Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com>

* [NEW] UploadFS respects $TMPDIR environment variable (RocketChat#17012)

* [IMPROVE] Stop re-sending push notifications rejected by the gateway (RocketChat#18608)

Co-authored-by: Diego Sampaio <chinello@gmail.com>

* [FIX] User Info: Email and name/username display (RocketChat#18976)

* Update Meteor to 1.11.1 (RocketChat#18959)

* [FIX] API call users.setStatus does not trigger status update of clients (RocketChat#18961)

* LingoHub Update 🚀 (RocketChat#18973)

Manual push by LingoHub User: Gabriel Engel.
Project: Rocket.Chat

Made with ❤️ by https://lingohub.com

Co-authored-by: Gabriel Engel <gabriel.engel@gmail.com>

* Bump marked from 0.6.3 to 0.7.0 (RocketChat#18801)

Bumps [marked](https://github.com/markedjs/marked) from 0.6.3 to 0.7.0.
- [Release notes](https://github.com/markedjs/marked/releases)
- [Commits](markedjs/marked@v0.6.3...v0.7.0)

Signed-off-by: dependabot[bot] <support@github.com>

Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>

* Bump Livechat widget (RocketChat#18977)

* [NEW] Option to require settings on wizard UI via ENV variables (RocketChat#18974)

Co-authored-by: Martin <martin.schoeler@rocket.chat>

* [NEW] Retention policy precision defined by a cron job expression (RocketChat#18975)

Co-authored-by: Diego Sampaio <chinello@gmail.com>

* Set some queries to prefer the secondary database (RocketChat#18887)

* Check i18n file for missing variables (RocketChat#18762)

* [FIX] Scrollbar mention ticks always rendering as white (RocketChat#18979)

* [FIX] Purged threads still show as unread (RocketChat#18944)

Co-authored-by: Diego Sampaio <chinello@gmail.com>
Co-authored-by: Rodrigo Nascimento <rodrigoknascimento@gmail.com>

* [NEW][Apps] Add a new upload API (RocketChat#18955)

Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com>

* [NEW] Send E2E encrypted messages’ content on push notifications (RocketChat#18882)

* New: Use database change streams when available (RocketChat#18892)

Co-authored-by: Diego Sampaio <chinello@gmail.com>

* Bump version to 3.7.0-rc.0

* Regression: Handle MongoDB authentication issues (RocketChat#18993)

* [FIX] LDAP avatar upload (RocketChat#18994)

* Bump version to 3.7.0-rc.1

* [FIX] Federation issues (RocketChat#18978)

* fixed message sending, you should not filter the domains, specially filter by only the local origin/source

* Fixing callback registration

* Increased the rate limiting

* Obbey to settings properties (RocketChat#19020)

* [FIX] PDF not rendering (RocketChat#18956)

* [FIX] Errors in LDAP avatar sync preventing login (RocketChat#18948)

* [FIX] LDAP avatar upload (RocketChat#18994)

* [FIX] Federation issues (RocketChat#18978)

* fixed message sending, you should not filter the domains, specially filter by only the local origin/source

* Fixing callback registration

* Increased the rate limiting

* Obbey to settings properties (RocketChat#19020)

* Bump version to 3.7.0-rc.2

* Bump version to 3.6.3

* Create VIP Sponsors.md

* Regression: Fix login screen reactivity of external login providers (RocketChat#19033)

* [NEW][Apps] Add support to the "encoding" option in http requests from Apps (RocketChat#19002)

* Allow specify the encoding of the response data

* Update Apps-Engine version

Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com>
Co-authored-by: Douglas Gubert <d-gubert@users.noreply.github.com>

* Regression: File upload via apps not working in some scenarios (RocketChat#18995)

* [FIX] [Apps] Fix app user doesn't has permission to upload files

* Fix errorClass [Error]: Forbidden [forbidden]

In `app/file-upload/lib/FileUploadBase.js`, we configured `UploadFS.config.defaultStorePermissions`, which validates `insert(userId, doc)` etc. But the parameter userId cann't be always obtained by this validation method correctly (sometimes it's undefined). Meteor use the Meteor.userId() as its fallback option. So we can wrap the original call with `Meteor.runAsUser` to solve issue.

* Add a new validator into canAccessRoom

canAccessRoom (`app/authorization/server/functions/canAccessRoom.js`) is an essential validator for Rocket.Chat to check whether some user has permissions to access some room. In this PR, we added a new validator that allows app users to access any room on a Rocket.Chat server even if it is not a member of the room.

*  An attempt to fix Meteor code must always run within a Fiber Error

Original Error: "Error: Meteor code must always run within a Fiber. Try wrapping callbacks that you pass to non-Meteor libraries with Meteor.bindEnvironment."

* Add support for uploading files by a livecaht visitor

* Support upload files with livechat visitors

* Reduce an unnecessary DB query - Users.findOneById

* Move the "bypass" out of canAccessRoom

Adding a bypass inside canAccessRoom can potentially allow apps to do stuff we're not prepared (yet)

* Update Apps-Engine version

* Some refactoring

* Fix a rateada

Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com>

* [NEW] Apps-Engine v1.18.0 (RocketChat#19047)

* Regression: Elements select & multiSelect not rendered correctly in the App Settings (RocketChat#19005)

Co-authored-by: Douglas Gubert <d-gubert@users.noreply.github.com>

* Bump version to 3.7.0-rc.3

* Bump version to 3.7.0-rc.4

* Bump version to 3.7.0

* Update package-lock.json

* Fix import in Omnichannel Filter

* Remove duplicate function

Co-authored-by: Diego Sampaio <chinello@gmail.com>
Co-authored-by: Tasso Evangelista <tasso.evangelista@rocket.chat>
Co-authored-by: Gabriel Engel <gabriel.engel@gmail.com>
Co-authored-by: lsignac <lsignac@users.noreply.github.com>
Co-authored-by: gabriellsh <40830821+gabriellsh@users.noreply.github.com>
Co-authored-by: Guilherme Gazzo <guilherme@gazzo.xyz>
Co-authored-by: Maarten <me@maartendewaard.nl>
Co-authored-by: pierre-lehnen-rc <55164754+pierre-lehnen-rc@users.noreply.github.com>
Co-authored-by: Guilherme Gazzo <guilhermegazzo@gmail.com>
Co-authored-by: Renato Becker <renato.augusto.becker@gmail.com>
Co-authored-by: Felipe Parreira <felipeparreirafb@gmail.com>
Co-authored-by: William Reiske <wreiske@mieweb.com>
Co-authored-by: Karting06 <Karting06@users.noreply.github.com>
Co-authored-by: Gilles Meyer <gilles@gmind.net>
Co-authored-by: Martin Schoeler <martin.schoeler@rocket.chat>
Co-authored-by: Diego Mello <diegolmello@gmail.com>
Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com>
Co-authored-by: Thassio Victor <tvmcarvalho@gmail.com>
Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com>
Co-authored-by: Shiqi Mei <shiqi.mei@rocket.chat>
Co-authored-by: Dustin Skoracki <d-sko@users.noreply.github.com>
Co-authored-by: lingohub[bot] <69908207+lingohub[bot]@users.noreply.github.com>
Co-authored-by: Rodrigo Nascimento <rodrigoknascimento@gmail.com>
Co-authored-by: Alan Sikora <alansikora@gmail.com>
Co-authored-by: Marcelo Schmidt <marcelo.schmidt@rocket.chat>
Co-authored-by: Douglas Gubert <d-gubert@users.noreply.github.com>
Co-authored-by: Shailesh Baldaniya <shaileshbaldaniya351@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

3 participants