Slack Import: corporate-export data (direct messages, private channels, etc.) is not imported

[niklasdahlheimer]

Description:

Right now Rocket.Chat´s Slack Import only imports public channels, users and messages.
The coporate-export of Slack gives the opportunity to also export:

• private channels ("groups")(groups.json)
• direct messages (dms.json)
• multi-user-conversations (mpims.json)

The importer.js is processing all the message-subfolders but only takes care about the public channels, described in channels.json.

(To me) it looks like the routines are already there to process the other json-files mentioned above, especially because the importer already handles all the message-subfolders. So it would be great if this could be implemented.

Steps to reproduce:

1. Go to AdminPanel->Import->Slack
2. Select Sack-Coporate-Export-Zip-file
3. Wait for Rocket-Chat Importer to process the file
4. See that all users, all messages but only public channels are importable

Expected behavior:

Also private channels, direct messages and multi-user-conversations should be importable

Server Setup Information:

• Version of Rocket.Chat Server: latest (0.74.3)
• Operating System: Synology DSM
• Deployment Method: Docker Container
• Number of Running Instances: 1
• NodeJS Version: 8.11.4
• MongoDB Version: 4.0.5

I would be happy to provide more information or a testfile if nessecary.

[geekgonecrazy]

Does the export its self actually include these? If I remember right if the person who exported is not part of a channel slack doesn't even export them unless you submit for some special data retention thing

[niklasdahlheimer]

Does the export its self actually include these? If I remember right if the person who exported is not part of a channel slack doesn't even export them unless you submit for some special data retention thing

Thats true in case you use Slacks standard export. In the plus-plan you can use the coporate-export, which includes all information I mentioned above. You can read about the difference export-types here:
Sack : Export your workspace data

The structure of the coporate export zip file:

├── channels.json
├── subfolder_channel_1 (i.e. "General", "public Information")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)
├── groups.json
├── subfolder_group_1 (i.e. "secret Group", "Management")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)
├── mpims.json
├── subfolder_multi_person_chat_1 (i.e. "mpdm-joe-lisa-anna-john", "mpdm-lisa-john-peter")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)
├── dms.json
├── subfolder_direct_messages_1 (i.e. "D38RUJBCP", "D39AJJG7L")
│   ├── 2019-12-20.json            
│   ├── 2019-12-21.json
│   └── ... 
├── ... (more subfolders)

groups.json and mpims.json looks just like the channels.json.

The structure of the dms.json:

[
    {
        "id": "D38RUJBCP",
        "created": 1529308826,
        "members": [
            "U63MAB7U5",
            "U1D4CHJLV"
        ]
    },
    {
        "id": "D39AJJG7L",
        "created": 1527578772,
        "members": [
            "U63MAB7U5",
            "U1GKUUV7H"
        ]
    },...
]

I made an attempt to add the processing of groups, dms and mpims to the importer.js:
groups are easy to handle by taking care of them at the folder search:

...
//parse group data (private Channels)
if (entry.entryName === 'groups.json') {
   super.updateProgress(ProgressStep.PREPARING_GROUPS);
   tempGroups = JSON.parse(entry.getData().toString()).filter((channel) => channel.creator != null);
   return;
}
...

adding them to the selectionChannel:

...
if(tempGroups){
  selectionChannels = selectionChannels.concat(tempGroups.map((group) => new SelectionChannel(group.id, group.name, group.is_archived, true, true)));
}
...
//selectionChannels must not be a const then!

Im struggeling with adding the dms. Should the importing user choose them? So there need to be a lot of changes to the data model (i.e adding a selectionDirectChat-Class and so on).

If the direct chats would be imported depending on which users are imported there need to be a bit of code checking if a direct chat (type "d") document for the certain direct chat already exists in the the rocketchat_room-collection and exchange the id of the direct chat then.

Any thoughts about that? Should a branch be opened for that? Should I push my code somewhere?

I guess adding this feature would be a great step forward for small companies to change over to RocketChat without losing any of there chat-data.

[geekgonecrazy]

Nice! Yes if you open a PR we can likely collaborate and get it merged.

Cc: @Hudell probably would be involved here