Out of memory errors after migrating to Dokka v2.0.0-Beta

[lauzadis]

Describe the bug
We are running into java.lang.OutOfMemoryError: Java heap space after migrating to Dokka 2.0.0-Beta, but only when building docs for our AWS SDK service clients. We have no trouble generating docs for our runtime projects.

Expected behaviour
Dokka should not run out of memory.

Screenshots
N/A

To Reproduce

# Clone both of our repositories into the same parent directory
git clone git@github.com:smithy-lang/smithy-kotlin.git
git clone git@github.com:awslabs/aws-sdk-kotlin.git

# Checkout the dev branches
cd smithy-kotlin
git checkout try-dokka-2
cd ../aws-sdk-kotlin
git checkout try-dokka-2

# Build docs, see they generate fine.
./gradlew :dokkaGenerate

# Code-generate an AWS service client
./gradlew :codegen:sdk:bootstrap -Paws.services=s3 

# Build docs again, see an out of memory error
./gradlew :dokkaGenerate

Dokka configuration
We use a fairly simple configuration, but it's spread across many projects. Check the root project's build.gradle.kts, the services project's build.gradle.kts, and the newly-created Dokka conventions plugin for the relevant config.

I've also tried the two tips provided in the troubleshooting section of the migration guide, which didn't work.

Installation

• Operating system: macOS
• Build tool: Gradle v8.5
• Dokka version: 2.0.0-Beta

Additional context
Add any other context about the problem here

[atyrin]

Hello! We are aware of high memory consumption in the Dokka engine, but so far the only way to address it is increasing heap size for Gradle workers.

This patch applied to the dokka-convention plugin helped me to build documentation for aws-sdk-kotlin (for all bootstrapped services as well)

Index: buildSrc/src/main/kotlin/dokka-convention.gradle.kts
IDEA additional info:
Subsystem: com.intellij.openapi.diff.impl.patch.CharsetEP
<+>UTF-8
===================================================================
diff --git a/buildSrc/src/main/kotlin/dokka-convention.gradle.kts b/buildSrc/src/main/kotlin/dokka-convention.gradle.kts
--- a/buildSrc/src/main/kotlin/dokka-convention.gradle.kts	(revision 3d2b3ea35bc2b6247974812bd7e77bbde282a4c5)
+++ b/buildSrc/src/main/kotlin/dokka-convention.gradle.kts	(date 1734427013842)
@@ -11,6 +11,11 @@
     val sdkVersion: String by project
     moduleVersion.set(sdkVersion)
 
+    dokkaGeneratorIsolation = ProcessIsolation {
+        // Configures heap size
+        maxHeapSize = "4g"
+    }
+
     pluginsConfiguration.html {
         customStyleSheets.from(
             rootProject.file("docs/dokka-presets/css/logo-styles.css"),

[adam-enko]

Hi, I've tried to reproduce the problem but I get an error about a missing plugin Plugin [id: 'aws.sdk.kotlin.service-module'] was not found

➜  aws-sdk-kotlin git:(try-dokka-2) ✗ ./gradlew :dokkaGenerate
Including build '/Users/dev/projects/external/amazon/smithy-kotlin'

> Configure project :build-support
WARNING: Unsupported Kotlin plugin version.
The `embedded-kotlin` and `kotlin-dsl` plugins rely on features of Kotlin `1.9.20` that might work differently than in the requested version `2.0.21`.

> Configure project :smithy-kotlin
    
    Thank you for enabling Dokka Gradle plugin V2!
    To learn about migrating read the migration guide https://kotl.in/dokka-gradle-migration
    
    We would appreciate your feedback!
     - Please report any feedback or problems https://kotl.in/dokka-issues
     - Chat with the community visit #dokka in https://kotlinlang.slack.com/ (To sign up visit https://kotl.in/slack)
    
    You can suppress this message by adding
        org.jetbrains.dokka.experimental.gradle.pluginMode.noWarn=true
    to your project's `gradle.properties` file.
    

> Configure project :
warning: Dokka Gradle plugin V2 migration helpers are enabled
    
    Thank you for migrating to Dokka Gradle plugin V2!
    Migration is in progress, and helpers have been enabled.
    To learn about migrating read the migration guide https://kotl.in/dokka-gradle-migration
    
    Once you have finished migrating disable the migration helpers by adding
        org.jetbrains.dokka.experimental.gradle.pluginMode=V2Enabled
    to your project's `gradle.properties` file.
    
    We would appreciate your feedback!
     - Please report any feedback or problems https://kotl.in/dokka-issues
     - Chat with the community visit #dokka in https://kotlinlang.slack.com/ (To sign up visit https://kotl.in/slack)
    

FAILURE: Build failed with an exception.

* Where:
Build file '/Users/dev/projects/external/amazon/aws-sdk-kotlin/services/accessanalyzer/build.gradle.kts' line: 1

* What went wrong:
Plugin [id: 'aws.sdk.kotlin.service-module'] was not found in any of the following sources:

- Gradle Core Plugins (plugin is not in 'org.gradle' namespace)
- Included Builds (None of the included builds contain this plugin)
- Plugin Repositories (plugin dependency must include a version number for this source)

* Try:
> Run with --stacktrace option to get the stack trace.
> Run with --info or --debug option to get more log output.
> Run with --scan to get full insights.
> Get more help at https://help.gradle.org.

BUILD FAILED in 5s
24 actionable tasks: 2 executed, 22 up-to-date

[lauzadis]

@atyrin Thanks for the suggestion. I saw this in the migration guide and it didn't work for me at one point. Trying it again, and on the latest release Dokka 2.0.0, it's working great. Thanks for your help!

@adam-enko I haven't seen this issue before... that's not a plugin we publish or depend on. 🤔 If you're still curious to reproduce the issue, I recommend bootstrapping a single service to start (pick your favorite one) and making sure it can build. If it still fails, let me know. There shouldn't be any plugins declared in that location: aws-sdk-kotlin/services/accessanalyzer/build.gradle.kts' line: 1