-
Notifications
You must be signed in to change notification settings - Fork 10.9k
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
New: Use database change streams when available #18892
Merged
Merged
Conversation
This file contains bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
rodrigok
force-pushed
the
change-streams
branch
from
September 14, 2020 11:14
4234470
to
335f45e
Compare
rodrigok
force-pushed
the
change-streams
branch
from
September 14, 2020 11:28
335f45e
to
553a9d7
Compare
# Conflicts: # server/publications/settings/index.js
sampaiodiego
requested changes
Sep 21, 2020
# Conflicts: # app/retention-policy/server/cronPruneMessages.js
sampaiodiego
requested changes
Sep 22, 2020
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
- when running a mongodb version not compatible with change streams, if I don't set
MONGO_OPLOG_URL
I get the following error:
(node:29097) UnhandledPromiseRejectionWarning: Error: URL malformed, cannot be parsed
at module.exports (/Users/diegosampaio/dev/Rocket.Chat/node_modules/mongodb/lib/url_parser.js:18:21)
at internal/util.js:297:30
at new Promise (<anonymous>)
at internal/util.js:296:12
at app/models/server/models/_oplogHandle.ts:35:27
at /Users/diegosampaio/.meteor/packages/promise/.0.11.2.1xavwef.9sbp++os+web.browser+web.browser.legacy+web.cordova/npm/node_modules/meteor-promise/fiber_pool.js:43:40
sampaiodiego
approved these changes
Sep 22, 2020
Looks like breaking changes improve performance a lot. |
10 tasks
Merged
This was referenced Oct 1, 2020
ear-dev
pushed a commit
to WideChat/Rocket.Chat
that referenced
this pull request
Oct 5, 2020
* Bump version to 3.7.0-develop * Do not use deprecated express method (RocketChat#18686) * Replace assets copy on postinstall with symlinks (RocketChat#18707) * Update Meteor to 1.11 (RocketChat#18754) * Update Meteor to 1.11 * Update Node version * Fix livechat view creation * LingoHub Update 🚀 (RocketChat#18761) Manual push by LingoHub User: Diego Sampaio. Project: Rocket.Chat Made with ❤️ by https://lingohub.com Co-authored-by: Diego Sampaio <chinello@gmail.com> * Fix french translations (RocketChat#18746) Replace [Nom du site] by [Site_Name]... * Refactor: Omnichannel Realtime Monitoring (RocketChat#18666) Co-authored-by: Guilherme Gazzo <guilherme@gazzo.xyz> * [FIX] Create Custom OAuth services from environment variables (RocketChat#17377) * Fix saveRoomSettings method complexity (RocketChat#18840) * [FIX] e.sendToBottomIfNecessaryDebounced is not a function (RocketChat#18834) * Log WebDav upload errors (RocketChat#18849) * [FIX] File upload (Avatars, Emoji, Sounds) (RocketChat#18841) * [FIX] IE11 support livechat widget (RocketChat#18850) * [FIX] Can't change password (RocketChat#18836) * [FIX] Admin user blank page (RocketChat#18851) * [FIX][ENTERPRISE] Omnichannel service status switching to unavailable (RocketChat#18835) * [FIX] User can't invite or join other Omnichannel rooms (RocketChat#18852) * [FIX] Omnichannel Current Chats open status filter not working (RocketChat#18795) * [FIX] Jitsi call start updating subscriptions (RocketChat#18837) * [FIX] Showing alerts during setup wizard (RocketChat#18862) * Remember users' 2FA right after registration * Change register server to false by default * [FIX] Omnichannel Current Chats open status filter not working (RocketChat#18795) * [FIX][ENTERPRISE] Omnichannel service status switching to unavailable (RocketChat#18835) * [FIX] File upload (Avatars, Emoji, Sounds) (RocketChat#18841) * [FIX] IE11 support livechat widget (RocketChat#18850) * [FIX] Admin user blank page (RocketChat#18851) * [FIX] User can't invite or join other Omnichannel rooms (RocketChat#18852) * [FIX] Showing alerts during setup wizard (RocketChat#18862) * Remember users' 2FA right after registration * Change register server to false by default * Bump version to 3.6.1 * LingoHub based on develop (RocketChat#18828) * LingoHub Update 🚀 Manual push by LingoHub User: Diego Sampaio. Project: Rocket.Chat Made with ❤️ by https://lingohub.com * LingoHub Update 🚀 Manual push by LingoHub User: Diego Sampaio. Project: Rocket.Chat Made with ❤️ by https://lingohub.com Co-authored-by: Diego Sampaio <chinello@gmail.com> * [FIX] invite-all-from and invite-all-to commands don't work with multibyte room names (RocketChat#18919) * [FIX] If there is `ufs` somewhere in url the request to api always returns 404 (RocketChat#18874) * chore(packages): add fixed version of ufs * test(e2e): test usernames with 'ufs' * test(ufs-router): remove expect for header * test: turn test callback synchronous * test: fix async to use done * chore(packages/meteor-jalik-ufs): add ufs package * Revert "test: fix async to use done" This reverts commit 6276e0b. * Revert "test: turn test callback synchronous" This reverts commit 2af11bb. * Revert "test(ufs-router): remove expect for header" This reverts commit 2c4eeb0. * [FIX] "Save to WebDav" not working (RocketChat#18883) * [FIX] Read receipts showing blank names and not marking messages as read (RocketChat#18918) * [FIX] Non-upload requests being passed to UFS proxy middleware (RocketChat#18931) * fix(ufs-proxy): correct routing pattern * chore(ufs-proxy): add deprecation console warning * [FIX] Version update check cron job (RocketChat#18916) * [FIX] Ignore User action from user card (RocketChat#18866) * [FIX] Custom fields required if minLength set and no text typed (RocketChat#18838) * [FIX] Dutch: add translations for missing variables (RocketChat#18814) * [FIX] French: Add missing __online__ var (RocketChat#18813) * [FIX] Deactivate users that are the last owner of a room using REST API (RocketChat#18864) * test: add e2e tests for REST API user deactivation * fix(app): read confirmRelinquish from HTTP request * chore(app): remove unnecessary console.log * [FIX] Show custom fields of invalid type (RocketChat#18794) * [FIX] Create Custom OAuth services from environment variables (RocketChat#17377) * [FIX] Read receipts showing blank names and not marking messages as read (RocketChat#18918) * [FIX] Version update check cron job (RocketChat#18916) * [FIX] Ignore User action from user card (RocketChat#18866) * [FIX] Deactivate users that are the last owner of a room using REST API (RocketChat#18864) * test: add e2e tests for REST API user deactivation * fix(app): read confirmRelinquish from HTTP request * chore(app): remove unnecessary console.log * [FIX] Show custom fields of invalid type (RocketChat#18794) * Bump version to 3.6.2 * [FIX] Stop adding push messages to queue if push is disabled (RocketChat#18830) * [NEW] "Room avatar changed" system messages (RocketChat#18839) * [FIX] PDF not rendering (RocketChat#18956) * [FIX] Spurious expert role in startup data (RocketChat#18667) * [FIX] Open room after guest registration (RocketChat#18755) * [FIX] Block user action (RocketChat#18950) * Refactor: Omnichannel Analytics (RocketChat#18766) * [FIX] Deactivated users show as offline (RocketChat#18767) * [FIX] Reaction buttons not behaving properly (RocketChat#18832) * Refactor: Message Audit page & Audit logs (RocketChat#18808) * [FIX] "Download my data" popup showing HTML code. (RocketChat#18947) * [IMPROVE] Move jump to message outside menu (RocketChat#18928) * [FIX] Users not being able to activate/deactivate E2E in DMs (RocketChat#18943) * Refactor: Admin permissions page (RocketChat#18932) Co-authored-by: Guilherme Gazzo <guilhermegazzo@gmail.com> * [IMPROVEMENT] Add "Allow_Save_Media_to_Gallery" setting to mobile (RocketChat#18875) * Bump lodash.merge from 4.6.1 to 4.6.2 (RocketChat#18800) * [FIX] Errors in LDAP avatar sync preventing login (RocketChat#18948) * [NEW][Apps] Add support for new livechat guest's and room's events (RocketChat#18946) * Add support for new livechat's guest and room events * Update trigger calls * Update Apps-Engine version Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com> * [NEW][Apps] Add a Livechat API - setCustomFields (RocketChat#18912) * Map livechatData to customFields * Add livechatData field to apps converter Co-authored-by: Thassio Victor <tvmcarvalho@gmail.com> Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com> * [NEW] UploadFS respects $TMPDIR environment variable (RocketChat#17012) * [IMPROVE] Stop re-sending push notifications rejected by the gateway (RocketChat#18608) Co-authored-by: Diego Sampaio <chinello@gmail.com> * [FIX] User Info: Email and name/username display (RocketChat#18976) * Update Meteor to 1.11.1 (RocketChat#18959) * [FIX] API call users.setStatus does not trigger status update of clients (RocketChat#18961) * LingoHub Update 🚀 (RocketChat#18973) Manual push by LingoHub User: Gabriel Engel. Project: Rocket.Chat Made with ❤️ by https://lingohub.com Co-authored-by: Gabriel Engel <gabriel.engel@gmail.com> * Bump marked from 0.6.3 to 0.7.0 (RocketChat#18801) Bumps [marked](https://github.com/markedjs/marked) from 0.6.3 to 0.7.0. - [Release notes](https://github.com/markedjs/marked/releases) - [Commits](markedjs/marked@v0.6.3...v0.7.0) Signed-off-by: dependabot[bot] <support@github.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> * Bump Livechat widget (RocketChat#18977) * [NEW] Option to require settings on wizard UI via ENV variables (RocketChat#18974) Co-authored-by: Martin <martin.schoeler@rocket.chat> * [NEW] Retention policy precision defined by a cron job expression (RocketChat#18975) Co-authored-by: Diego Sampaio <chinello@gmail.com> * Set some queries to prefer the secondary database (RocketChat#18887) * Check i18n file for missing variables (RocketChat#18762) * [FIX] Scrollbar mention ticks always rendering as white (RocketChat#18979) * [FIX] Purged threads still show as unread (RocketChat#18944) Co-authored-by: Diego Sampaio <chinello@gmail.com> Co-authored-by: Rodrigo Nascimento <rodrigoknascimento@gmail.com> * [NEW][Apps] Add a new upload API (RocketChat#18955) Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com> * [NEW] Send E2E encrypted messages’ content on push notifications (RocketChat#18882) * New: Use database change streams when available (RocketChat#18892) Co-authored-by: Diego Sampaio <chinello@gmail.com> * Bump version to 3.7.0-rc.0 * Regression: Handle MongoDB authentication issues (RocketChat#18993) * [FIX] LDAP avatar upload (RocketChat#18994) * Bump version to 3.7.0-rc.1 * [FIX] Federation issues (RocketChat#18978) * fixed message sending, you should not filter the domains, specially filter by only the local origin/source * Fixing callback registration * Increased the rate limiting * Obbey to settings properties (RocketChat#19020) * [FIX] PDF not rendering (RocketChat#18956) * [FIX] Errors in LDAP avatar sync preventing login (RocketChat#18948) * [FIX] LDAP avatar upload (RocketChat#18994) * [FIX] Federation issues (RocketChat#18978) * fixed message sending, you should not filter the domains, specially filter by only the local origin/source * Fixing callback registration * Increased the rate limiting * Obbey to settings properties (RocketChat#19020) * Bump version to 3.7.0-rc.2 * Bump version to 3.6.3 * Create VIP Sponsors.md * Regression: Fix login screen reactivity of external login providers (RocketChat#19033) * [NEW][Apps] Add support to the "encoding" option in http requests from Apps (RocketChat#19002) * Allow specify the encoding of the response data * Update Apps-Engine version Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com> Co-authored-by: Douglas Gubert <d-gubert@users.noreply.github.com> * Regression: File upload via apps not working in some scenarios (RocketChat#18995) * [FIX] [Apps] Fix app user doesn't has permission to upload files * Fix errorClass [Error]: Forbidden [forbidden] In `app/file-upload/lib/FileUploadBase.js`, we configured `UploadFS.config.defaultStorePermissions`, which validates `insert(userId, doc)` etc. But the parameter userId cann't be always obtained by this validation method correctly (sometimes it's undefined). Meteor use the Meteor.userId() as its fallback option. So we can wrap the original call with `Meteor.runAsUser` to solve issue. * Add a new validator into canAccessRoom canAccessRoom (`app/authorization/server/functions/canAccessRoom.js`) is an essential validator for Rocket.Chat to check whether some user has permissions to access some room. In this PR, we added a new validator that allows app users to access any room on a Rocket.Chat server even if it is not a member of the room. * An attempt to fix Meteor code must always run within a Fiber Error Original Error: "Error: Meteor code must always run within a Fiber. Try wrapping callbacks that you pass to non-Meteor libraries with Meteor.bindEnvironment." * Add support for uploading files by a livecaht visitor * Support upload files with livechat visitors * Reduce an unnecessary DB query - Users.findOneById * Move the "bypass" out of canAccessRoom Adding a bypass inside canAccessRoom can potentially allow apps to do stuff we're not prepared (yet) * Update Apps-Engine version * Some refactoring * Fix a rateada Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com> * [NEW] Apps-Engine v1.18.0 (RocketChat#19047) * Regression: Elements select & multiSelect not rendered correctly in the App Settings (RocketChat#19005) Co-authored-by: Douglas Gubert <d-gubert@users.noreply.github.com> * Bump version to 3.7.0-rc.3 * Bump version to 3.7.0-rc.4 * Bump version to 3.7.0 * Update package-lock.json * Fix import in Omnichannel Filter * Remove duplicate function Co-authored-by: Diego Sampaio <chinello@gmail.com> Co-authored-by: Tasso Evangelista <tasso.evangelista@rocket.chat> Co-authored-by: Gabriel Engel <gabriel.engel@gmail.com> Co-authored-by: lsignac <lsignac@users.noreply.github.com> Co-authored-by: gabriellsh <40830821+gabriellsh@users.noreply.github.com> Co-authored-by: Guilherme Gazzo <guilherme@gazzo.xyz> Co-authored-by: Maarten <me@maartendewaard.nl> Co-authored-by: pierre-lehnen-rc <55164754+pierre-lehnen-rc@users.noreply.github.com> Co-authored-by: Guilherme Gazzo <guilhermegazzo@gmail.com> Co-authored-by: Renato Becker <renato.augusto.becker@gmail.com> Co-authored-by: Felipe Parreira <felipeparreirafb@gmail.com> Co-authored-by: William Reiske <wreiske@mieweb.com> Co-authored-by: Karting06 <Karting06@users.noreply.github.com> Co-authored-by: Gilles Meyer <gilles@gmind.net> Co-authored-by: Martin Schoeler <martin.schoeler@rocket.chat> Co-authored-by: Diego Mello <diegolmello@gmail.com> Co-authored-by: dependabot[bot] <49699333+dependabot[bot]@users.noreply.github.com> Co-authored-by: Thassio Victor <tvmcarvalho@gmail.com> Co-authored-by: Douglas Gubert <douglas.gubert@gmail.com> Co-authored-by: Shiqi Mei <shiqi.mei@rocket.chat> Co-authored-by: Dustin Skoracki <d-sko@users.noreply.github.com> Co-authored-by: lingohub[bot] <69908207+lingohub[bot]@users.noreply.github.com> Co-authored-by: Rodrigo Nascimento <rodrigoknascimento@gmail.com> Co-authored-by: Alan Sikora <alansikora@gmail.com> Co-authored-by: Marcelo Schmidt <marcelo.schmidt@rocket.chat> Co-authored-by: Douglas Gubert <d-gubert@users.noreply.github.com> Co-authored-by: Shailesh Baldaniya <shaileshbaldaniya351@gmail.com>
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Proposed changes
The intent of this PR is to implement our own solution of oplog tailing in replacement of the meteor's one to prevent the internal processing made by meteor. With this change we implemented the possibility to use change streams which is the most efficient way to listen for data changes in database but it's only available from mongodb version 3.6 and when using wiredtiguer storage type.
To make this change we had to:
change
from the collections that we've implemented some time ago when we started this movement to move away from the meteor's reactivity solutions.change
event mentioned before.change
events, now implementing the option to open our own oplog tailing or use change stream when available.It's possible to configure the behavior with 2 environment variables:
IGNORE_CHANGE_STREAM=true
: which will force the usage of our oplog tailing implementation in place of the change stream.USE_NATIVE_OPLOG=true
: which will restore the usage of the meteor's internal oplog tailing.Issue(s)
How to test or reproduce
Screenshots
Types of changes
Checklist
Changelog
Further comments