Get entire message history from channel

[AbdulahadGhuman]

I want to get the entire message history from a channel but a lot of methods don't give me more than 100 messages. There's one method that gets the entire history but when it's too large, it sends an error, here's the code.

if (event.getMessage().getContentRaw().equals("Save Conversation")) {
            try {
                FileWriter newWriter = new FileWriter("C:\\Users\\abdul\\IdeaProjects\\DiscordBot\\" + event.getChannel().getName() + ".txt");
                List<Message> messageList = new ArrayList<>();
                try {
                    messageList = event.getChannel().getIterableHistory().takeAsync(100000)//
                            .thenApply(list -> list.stream()
                                    .collect(Collectors.toList()))
                            .get();
                } catch (InterruptedException e) {
                    e.printStackTrace();
                } catch (ExecutionException e) {
                    e.printStackTrace();
                }
                Collections.reverse(messageList);
                for (Message message : messageList) {
                    newWriter.write("Author: " + message.getAuthor().getName() + "\n" + "Content: " + message.getContentRaw() + "\n\n\n");
                }
                newWriter.close();
            } catch (IOException e) {
                throw new RuntimeException(e);
            }
        }

The error is
[JDA Gateway-Worker 1] WARN WebSocketClient - Missed 2 heartbeats! Trying to reconnect... for the .get() function

[DV8FromTheWorld]

If you're using the default event manager and this code is running in an event (which it is as I see you referencing event) then when you call .get() you're blocking the thread which the WebSocket uses to send and receive data from discord. This is why the WebSocket ends up complaining about missing heartbeats. You've prevented the WebSocket from processing anything by blocking!

The .get() method is a blocking method that means "Block the current thread until the asynchronous process is complete. You are asking for 10000 messages which means you're making 100 requests to discord (10000/100 = 100). This is far above the ratelimit, so JDA is waiting a certain amount of time before sending additional requests.

This might look like:

• send N requests
• wait X seconds (whatever discord tells us to wait to respect the rate limit)
• send N more requests
• wait X seconds again
• send N more requests
• .... [repeat]

Where N requests is the total amount of requests we can make before hitting the ratelimit and X is the amount of seconds to wait until the ratelimit is refreshed.

As you can tell, this is a process which can take a while, especially if you're making 100 requests! Instead of blocking, you should build an async pipeline to process the data.

RestAction<?> latestUpdate = null;

@Override
public void onMessageReceived(MessageReceivedEvent event) {
    String content = event.getMessage().getContentRaw();
    if (content.equals("Save Conversation")) {
        handleSaveConversation(event);
    }
}

private void handleSaveConversation(MessageReceivedEvent event) {
    String fileName = event.getChannel().getName() + ".txt";
    String filePath = "C:\\Users\\abdul\\IdeaProjects\\DiscordBot\\" + fileName;

    final FileWriter fileWriter;
    try {
        fileWriter = new FileWriter(filePath);
    } catch (IOException e) {
        e.printStackTrace();
        event.getChannel().sendMessage("Failed to save message due to error getting file writer: " + e.getMessage()).queue();
        return;
    }

    Message progressMessage;
    try {
        // Using .complete for simplicity, but could also be moved to queue and use its callback.
        progressMessage = event.getChannel().sendMessage(getProgressInfo(filePath, 0)).complete();
    }
    catch (Exception e) {
        e.printStackTrace();
        silentClose(fileWriter);
        return;
    }

    int MAX_DESIRED = 10000;
    List<Message> messages = new ArrayList<Message>();
    event.getChannel().getIterableHistory()
            .forEachAsync(message -> {
                messages.add(message);
                handleProgressUpdate(progressMessage, filePath, messages.size());
                return messages.size() < MAX_DESIRED;
            })
            .thenAccept(_ignored -> {
                Collections.reverse(messages);
                for (Message message : messages) {
                    // Don't save the progress message as it isn't part of chat.
                    if (message.getIdLong() == progressMessage.getIdLong()) {
                        continue;
                    }

                    try {
                        fileWriter.write("Author: " + message.getAuthor().getName() + "\n" + "Content: " + message.getContentRaw() + "\n\n\n");
                    }
                    catch (IOException e) {
                        // Rethrow as a RuntimeException so that the .exceptioally(...) will tell user what happened
                        throw new RuntimeException(e);
                    }
                }
                event.getChannel().sendMessage("Conversation was successfully saved").queue();
            })
            .exceptionally(error -> {
                error.printStackTrace();
                event.getChannel().sendMessage("Conversation saving failed due to error: " + error.getMessage()).queue();
                return null;
            })
            .whenComplete((_ignored, _ignored2) -> {
                // Set latestUpdate to null to try and prevent any updates being sent now that we're done
                // as we're about to delete the progressMessage we've been updating.
                latestUpdate = null;
                progressMessage.delete().queue();
                silentClose(fileWriter);
            });
}

private void handleProgressUpdate(Message progressMessage, String filePath, int totalMessages) {
    RestAction<?> action = progressMessage.editMessage(getProgressInfo(filePath, totalMessages));
    latestUpdate = action;

    action.setCheck(() -> {
        // Only send out the latest progress update.
        return action == latestUpdate;
    });
    //Submit is similar to .queue(...), but gives easier access to a "finally" like method.
    action.submit().whenComplete((_ignored, _ignored2) -> {
        // Clear the latest update if we just sent it.
        if (latestUpdate == action) {
            latestUpdate = null;
        }
    });
}

public static void silentClose(AutoCloseable closeable)
{
    try
    {
        closeable.close();
    }
    catch (Exception ignored) {}
}

private String getProgressInfo(String filePath, int totalMessages) {
    String message = "";
    message += "Processing conversation to file:\n";
    message += "`" + filePath + "`\n\n";
    message += "Total messages retrieved thus far: " + totalMessages;

    return message;
}