Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

[IMPROVE] Major overhaul on data importers #16279

Merged
merged 58 commits into from
Feb 4, 2020
Merged

[IMPROVE] Major overhaul on data importers #16279

merged 58 commits into from
Feb 4, 2020

Conversation

pierre-lehnen-rc
Copy link
Contributor

@pierre-lehnen-rc pierre-lehnen-rc commented Jan 18, 2020 •
edited
Loading

Changes

  • Reduces amount of RAM used by all importers (It can still be improved individually)
  • Decoupled UI from the Import Process. Now there's no risk of accidentally canceling an operation by refreshing or navigating.
  • You can now send a file to the server at one point and perform the import at another, even after a server restart.
  • Huge performance improvements on the Slack Importer
  • Performance Improvements on the CSV Importer
  • Modified Slack Importer to include support for DMs, Groups and Multi Party IMs
  • Download attachments from slack server and generate image previews.

Tests

  • Slack Importer
  • Slack Users (CSV) Importer
  • CSV Importer
  • HipChat Importer
  • HipChat Enterprise Importer

Screenshots

image

image

Closes #13905
Closes #13942
Closes #15659
Closes #13628
Closes #13351
Closes #12075
Closes #9781
Closes #8435
Closes #8426
Closes #8191
Closes #7047
Closes #6778

@lgtm-com

This comment has been minimized.

Sign in to view
@lgtm-com

This comment has been minimized.

Sign in to view
@lgtm-com

This comment has been minimized.

Sign in to view
@lgtm-com

This comment has been minimized.

Sign in to view
@lgtm-com

This comment has been minimized.

Sign in to view
@lgtm-com

This comment has been minimized.

Sign in to view
@geekgonecrazy
Copy link
Contributor

image

for some reason it seems it won't take anything but http? kind of strange especially since would never want to have something like this available via http://

@geekgonecrazy geekgonecrazy mentioned this pull request Jan 29, 2020
Closed
2 tasks
rodrigok
rodrigok requested changes Feb 2, 2020
View reviewed changes
app/lib/server/functions/insertMessage.js Outdated Show resolved Hide resolved
app/models/server/models/Messages.js Outdated

findAllSlackImportedMessagesWithFilesToDownload() {
const query = {
'slackFile.url_private_download': {
Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Would be possible to be generic here rather than tied to Slack? Like _importFile with a property source: "slack"?

Copy link
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Is it possible to keep the camelcase here?

package-lock.json Outdated Show resolved Hide resolved
@sampaiodiego sampaiodiego changed the title [NEW] Importer [IMPROVE] Major overhaul on data importers Feb 4, 2020
sampaiodiego
sampaiodiego previously approved these changes Feb 4, 2020
View reviewed changes
@sampaiodiego sampaiodiego merged commit f0f06e8 into develop Feb 4, 2020
@sampaiodiego sampaiodiego deleted the new-importer branch February 4, 2020 20:59
gabriellsh added a commit that referenced this pull request Feb 5, 2020
@gabriellsh
Merge branch 'develop' of github.com:RocketChat/Rocket.Chat into new/…
5623364 
…default-favorite-channels

* 'develop' of github.com:RocketChat/Rocket.Chat: (27 commits)
  [FIX] Missing edited icon in newly created messages (#16484)
  Regression: allow private channels to hide system messages  (#16483)
  Regression: Fix uikit modal closing on click (#16475)
  Regression: Fix undefined presence after reconnect (#16477)
  [FIX] Read Message after receive a message and the room is opened (#16473)
  [FIX] Send message with pending messages (#16474)
  Fix tests (#16469)
  Bump version to 3.0.0-develop
  [NEW] Button to download admin server info (#16059)
  Revert importer streamed uploads (#16465)
  [NEW] UiKit - Interactive UI elements for Rocket.Chat Apps (#16048)
  [FIX] Result of get avatar from url can be null (#16123)
  [IMPROVE] Request user presence on demand (#16348)
  [IMPROVE] Major overhaul on data importers (#16279)
  [BREAK] Filter System messages per room (#16369)
  Regression: Fix app user status change for non-existing user (#16458)
  [FIX] `stdout` streamer infinite loop (#16452)
  Regression: Fix sending a message not scrolling to bottom (#16451)
  LingoHub based on develop (#16450)
  [IMPROVE] Changes App user's status when the app was enabled/disabled (#16392)
  ...
@sampaiodiego sampaiodiego mentioned this pull request Feb 15, 2020
Merged
This was referenced Apr 7, 2020
Closed
Closed
Open
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@rodrigok rodrigok rodrigok requested changes

@sampaiodiego sampaiodiego sampaiodiego approved these changes

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
No milestone
4 participants
@pierre-lehnen-rc @geekgonecrazy @rodrigok @sampaiodiego