feat: add Schema generator Gradle plugin #1385

lauzadis · 2024-08-21T17:32:03Z

Issue #

Description of changes

By submitting this pull request, I confirm that my contribution is made under the terms of the Apache 2.0 license.

…rer, add metadata to annotation processor module

…`requireNotNull`)

…-anno

…nto feat-gp-anno

github-actions · 2024-08-21T22:12:26Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

lauzadis · 2024-08-21T23:18:13Z

...or-plugin/src/main/kotlin/aws/sdk/kotlin/hll/dynamodbmapper/plugins/SchemaGeneratorPlugin.kt

+        // FIXME Remove hardcoded versioning
+        dependencies.add("ksp", "aws.sdk.kotlin:dynamodb-mapper-codegen:1.3.2-SNAPSHOT")


Looking for ideas / suggestions on this. I can't seem to find a way to grab the sdkVersion from build.gradle.kts and use it here.

In our low-level codegen modules, we persist at build time a resource containing a version number and then later read it at runtime when we need the version. (Only works for JVM, obvs.) Would something similar work here?

This is something I was considering and forgot we used this solution already. I think it would work here, thanks!

It did end up working, thanks for the suggestion

ianbotsf · 2024-08-22T14:34:54Z

hll/dynamodb-mapper/dynamodb-mapper-schema-generator-plugin/build.gradle.kts

+gradlePlugin {
+    plugins {
+        create("dynamodb-mapper-schema-generator") {
+            id = "aws.sdk.kotlin.hll.dynamodbmapper.schema.generator"
+            implementationClass = "aws.sdk.kotlin.hll.dynamodbmapper.plugins.SchemaGeneratorPlugin"
+            description = "Plugin used to generate DynamoDbMapper schemas from user classes"
+        }
+    }
+}


Question: The Gradle documentation shows several additional metadata we could set about the plugin such as website URL, repository URL, display name, etc. Do we know how those are used? Is it worth setting them on this plugin?

I think it's displayed on Maven Central ("External Resources") in the sidebar. I can add them!

KSP plugin, for example: https://central.sonatype.com/artifact/com.google.devtools.ksp/com.google.devtools.ksp.gradle.plugin

ianbotsf · 2024-08-22T14:42:42Z

...or-plugin/src/main/kotlin/aws/sdk/kotlin/hll/dynamodbmapper/plugins/SchemaGeneratorPlugin.kt

+        // FIXME Remove hardcoded versioning
+        dependencies.add("ksp", "aws.sdk.kotlin:dynamodb-mapper-codegen:1.3.2-SNAPSHOT")


In our low-level codegen modules, we persist at build time a resource containing a version number and then later read it at runtime when we need the version. (Only works for JVM, obvs.) Would something similar work here?

ianbotsf · 2024-08-22T14:47:31Z

...lugin/src/test/kotlin/aws/sdk/kotlin/hll/dynamodbmapper/plugins/SchemaGeneratorPluginTest.kt

+    @Test
+    fun `applies the plugin`() {
+        val buildFileContent = """
+         plugins {
+            id("org.jetbrains.kotlin.jvm") version "2.0.0"
+            id("aws.sdk.kotlin.hll.dynamodbmapper.schema.generator")
+         }
+         configure<aws.sdk.kotlin.hll.dynamodbmapper.plugins.SchemaGeneratorPluginExtension>{ }
+        """.trimIndent()
+
+        buildFile.writeText(buildFileContent)
+
+        val result = GradleRunner.create()
+            .withProjectDir(testProjectDir)
+            .withArguments("--info", "build")
+            .withPluginClasspath()
+            .withGradleVersion("8.5")
+            .forwardOutput()
+            .build()
+
+        assertContains(setOf(TaskOutcome.SUCCESS, TaskOutcome.UP_TO_DATE), result.task(":build")?.outcome)
+    }


Opinion: This test/module may benefit from more dynamic versioning retrieved at runtime to avoid hardcoding things like the Gradle runner version, Kotlin version, etc.

Ideally we would have a parameterized test with multiple versions of Kotlin and Gradle. I think it should be added in follow-on PRs, I will add a TODO here.

ianbotsf · 2024-08-22T14:52:58Z

...lugin/src/test/kotlin/aws/sdk/kotlin/hll/dynamodbmapper/plugins/SchemaGeneratorPluginTest.kt

+import java.io.File
+import kotlin.test.assertContains
+
+class SchemaGeneratorPluginTest {


Question: Oh interesting—I was expecting your plugin tests to be another module sort of like dynamodb-mapper-annotation-processor-test. This looks like it works but I wonder if we miss anything by not having actual test modules as part of the overall Gradle project. Do you know if this is the recommended style of testing Gradle plugins and, if so, what tradeoffs it might have?

This is using the Gradle TestKit which seemed recommended for testing plugins. I think having another test module would be another good test to have, I'll look at adding it.

I added another test package which consumes the plugin. It's basically a copy of dynamodb-mapper-annotation-processor-test without the manual KSP configuration. We can expand this as more features are added like codegen configuration.

ianbotsf · 2024-08-22T14:56:06Z

hll/dynamodb-mapper/dynamodb-mapper-codegen/build.gradle.kts

+publishing {
+    publications {
+        create<MavenPublication>("dynamodb-mapper-codegen") {
+            from(components["java"])
+        }
+    }
+}


Question: Elsewhere where we manually create Maven publications we also add the sources JAR as an artifact. Should we be doing that here?

Yes I think we should. It doesn't prevent using this package as a dependency but it might help with autocompletion/search in IDEs.

This worked for all of the new publications except the Gradle plugin, which seems to already publish the JAR.

Execution failed for task ':hll:dynamodb-mapper:dynamodb-mapper-schema-generator-plugin:publishDynamodb-mapper-schema-generator-pluginPublicationToMavenLocal'. > Failed to publish publication 'dynamodb-mapper-schema-generator-plugin' to repository 'mavenLocal' > Invalid publication 'dynamodb-mapper-schema-generator-plugin': multiple artifacts with the identical extension and classifier ('jar', 'sources').

0marperez · 2024-08-22T15:15:20Z

hll/dynamodb-mapper/dynamodb-mapper/build.gradle.kts

@@ -80,7 +80,7 @@ if (project.NATIVE_ENABLED) {
    // Start by invoking the JVM-only KSP configuration
    dependencies.kspJvm(project(":hll:dynamodb-mapper:dynamodb-mapper-codegen"))

-    // Then we need to move the generated source from jvm to common. Gradle lacks a move task so we roll our own!


Question: I see we have a solution already but would this be useful here?

aws-sdk-kotlin/codegen/sdk/build.gradle.kts

Lines 175 to 178 in d13b314

copy {

from("$projectionOutputDir/src")

into("${it.destinationDir}/generated-src")

}

Good find, that's a copy but we want to move the files in this case

sonarqubecloud · 2024-08-22T19:00:33Z

Quality Gate failed

Failed conditions
6.2% Duplication on New Code (required ≤ 3%)

See analysis details on SonarCloud

github-actions · 2024-08-22T19:11:33Z

A new generated diff is ready to view.

No codegen difference in the AWS SDK

…lin (#1451) * initial poc commit of DynamoDB Mapper (#1232) * add support for Mapper initialization (#1237) * implement mapper pipeline (#1266) * initial implementation of codegen for low-level operations/types (#1357) * initial implementation of secondary index support (#1375) * Create new codegen module and refactor annotation processor to use it (#1382) * feat: add Schema generator Gradle plugin (#1385) * Fix plugin test package * add attribute converters for "standard" values (#1381) * fix: schema generator plugin test module (#1394) * feat: annotation processor codegen configuration (#1392) * feat: add `@DynamoDbIgnore` annotation (#1402) * DDB Mapper filter expressions (runtime components) (#1401) * feat: basic annotation processing (#1399) * add DSL overloads, paginators, and better builder integration for DDB Mapper ops codegen (#1409) * chore: split dynamodb-mapper-codegen into two modules (#1414) * emit DDB_MAPPER business metric (#1426) * feat: setup DynamoDbMapper publication (#1419) * DDB Mapper filter expressions (codegen components) (#1424) * correct docs * mark every HLL/DDBM API experimental (#1428) * fix accidental inclusion of expression attribute members in high-level DynamoDB Mapper requests (#1432) * Upgrade to latest build plugin version * fix: various issues found during testing (#1450) * chore: update Athena changelog notes for 1.3.57 (2024-10-18) release (#1449) * feat: update AWS API models * feat: update AWS service endpoints metadata * chore: release 1.3.60 * chore: bump snapshot version to 1.3.61-SNAPSHOT * feat: initial release of Developer Preview of DynamoDB Mapper for Kotlin * Fix Kotlin gradle-plugin version * fix: ddb mapper tests (#1453) * Bump build plugin version --------- Co-authored-by: Matas <lauzmata@amazon.com> Co-authored-by: aws-sdk-kotlin-ci <aws-kotlin-sdk-automation@amazon.com>

lauzadis added 30 commits August 9, 2024 09:47

Create codegen module

a51de4b

Create codegen module

0a779a1

Add tests and .api

011f367

Bootstrap

75e208a

tests

d8c65c9

Fix package name in StringsTest

5edef54

Latest

df80d66

Rendering correctly now

3d5818e

Small style changes

73514a5

ktlint

ba04d3e

ktlint

dbd737e

ktlint

1da4704

Remove unused types

48455a9

StringMap -> stringMap

77f0792

Remove old-form codegen

725a391

cleanup

77c3264

KDocs

a0134e0

apiDump

165e59b

bootstrap _something_

6120b6b

rename module: annotation-processor -> annotations-processor

abae3b0

More skeleton

924a04e

.api

d864987

Add extra metadata

3513113

Remove unused function

e995954

Pass around a codeGeneratorName

a1cfc8e

annotations -> annotatedClasses

f5cdd9a

rename: AnnotationRenderer -> SchemaRenderer, add common BuilderRende…

54ec0b3

…rer, add metadata to annotation processor module

Promote internal objects back to public

c8148fc

Check property nullability when rendering build method (specifically …

1176517

…`requireNotNull`)

Add TODOs

aac0fda

lauzadis added 9 commits August 21, 2024 13:30

Clean up

7c9272f

Remove unused module declaration

5e03d53

Remove extra newline

21dc9e8

Clean up

ca3ec01

Add a FIXME

99a0465

Reverting removing kotest

72bb40f

Add TODO

90c4fd8

ktlint

17a5c5b

Make class open

6343977

lauzadis changed the title ~~Add "Schema generator" Gradle plugin~~ feat: add Schema generator Gradle plugin Aug 21, 2024

lauzadis force-pushed the feat-gp-anno branch from 751be6b to 6343977 Compare August 21, 2024 20:10

lauzadis added 2 commits August 21, 2024 17:56

Merge branch 'main' of github.com:awslabs/aws-sdk-kotlin into feat-gp…

a0c4d59

…-anno

Merge branch 'feat-ddb-mapper' of github.com:awslabs/aws-sdk-kotlin i…

c5b863b

…nto feat-gp-anno

lauzadis marked this pull request as ready for review August 21, 2024 23:17

lauzadis requested a review from a team as a code owner August 21, 2024 23:17

lauzadis commented Aug 21, 2024

View reviewed changes

ianbotsf approved these changes Aug 22, 2024

View reviewed changes

0marperez approved these changes Aug 22, 2024

View reviewed changes

lauzadis added 6 commits August 22, 2024 13:34

Publish sources JAR

8bfcf13

Add extra configuration to Gradle plugin publication

4106bed

Use sdk-version.txt to pass around the sdkVersion

48968c9

Add a test package which consumes the plugin

9642be3

Add a TODO to parameterize the test

09f2f9c

ktlint

2e29bf0

lauzadis merged commit 32108ae into feat-ddb-mapper Aug 22, 2024
10 of 11 checks passed

lauzadis deleted the feat-gp-anno branch August 22, 2024 21:01

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

feat: add Schema generator Gradle plugin #1385

feat: add Schema generator Gradle plugin #1385

lauzadis commented Aug 21, 2024

github-actions bot commented Aug 21, 2024

lauzadis Aug 21, 2024

ianbotsf Aug 22, 2024

lauzadis Aug 22, 2024

lauzadis Aug 22, 2024

ianbotsf Aug 22, 2024

lauzadis Aug 22, 2024

ianbotsf Aug 22, 2024

ianbotsf Aug 22, 2024

lauzadis Aug 22, 2024

ianbotsf Aug 22, 2024

lauzadis Aug 22, 2024

lauzadis Aug 22, 2024

ianbotsf Aug 22, 2024

lauzadis Aug 22, 2024

lauzadis Aug 22, 2024

0marperez Aug 22, 2024

lauzadis Aug 22, 2024

sonarqubecloud bot commented Aug 22, 2024

github-actions bot commented Aug 22, 2024

		// FIXME Remove hardcoded versioning
		dependencies.add("ksp", "aws.sdk.kotlin:dynamodb-mapper-codegen:1.3.2-SNAPSHOT")

	copy {
	from("$projectionOutputDir/src")
	into("${it.destinationDir}/generated-src")
	}

feat: add Schema generator Gradle plugin #1385

feat: add Schema generator Gradle plugin #1385

Conversation

lauzadis commented Aug 21, 2024

Issue #

Description of changes

github-actions bot commented Aug 21, 2024

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

sonarqubecloud bot commented Aug 22, 2024

Quality Gate failed

github-actions bot commented Aug 22, 2024