Make Gradle tasks cacheable #295

marcospereira · 2023-11-03T03:37:54Z

What?

Gradle has a build cache that can be used to speed up the build process. From Gradle docs:

The Gradle build cache is a cache mechanism that aims to save time by reusing outputs produced by other builds. The build cache works by storing (locally or remotely) build outputs and allowing builds to fetch these outputs from the cache when it is determined that inputs have not changed, avoiding the expensive work of regenerating them.

I noticed that the build cache for the compileKotlin task was not working when using jte's generate or precompile, because the inputs generated by those tasks were not cached. Therefore, the generated code would be considered new input for compileKotlin, meaning the task cannot use the build cache. I suspect the same applies to compileJava, but I didn't check.

Local validation

Comparison when running the compileKotlin task with and without -Dorg.gradle.caching=true.

Before

Running with --build-cache multiple times has no effect since the cache is always stale:

time -h -p gradle --warning-mode=none --build-cache clean compileKotlin
> Task :cleanGenerateJte
> Task :cleanPrecompileJte UP-TO-DATE
> Task :clean
> Task :checkKotlinGradlePluginConfigurationErrors

> Task :generateJte
Extension gg.jte.models.generator.ModelExtension generated 3 files.

> Task :kspKotlin
> Task :compileKotlin

BUILD SUCCESSFUL in 6s
7 actionable tasks: 6 executed, 1 up-to-date
real 6.96
user 1.10
sys 0.16

After

With no cache (--warning-mode=none for brevity):

time -h -p gradle --warning-mode=none clean compileKotlin
> Task :cleanGenerateJte
> Task :cleanPrecompileJte UP-TO-DATE
> Task :clean
> Task :checkKotlinGradlePluginConfigurationErrors

> Task :generateJte
Extension gg.jte.models.generator.ModelExtension generated 3 files.

> Task :kspKotlin
> Task :compileKotlin

BUILD SUCCESSFUL in 8s
7 actionable tasks: 6 executed, 1 up-to-date
real 9.30
user 1.17
sys 0.15

With a populated build-cache (notice the FROM-CACHE):

time -h -p gradle --warning-mode=none --build-cache clean compileKotlin
> Task :cleanGenerateJte
> Task :cleanPrecompileJte UP-TO-DATE
> Task :clean
> Task :checkKotlinGradlePluginConfigurationErrors
> Task :generateJte FROM-CACHE
> Task :kspKotlin FROM-CACHE
> Task :compileKotlin FROM-CACHE

BUILD SUCCESSFUL in 1s
7 actionable tasks: 3 executed, 3 from cache, 1 up-to-date
real 1.49
user 1.13
sys 0.16

Let's see the effect on testClasses after running the command above:

time -h -p gradle --warning-mode=none --build-cache clean testClasses
> Task :cleanGenerateJte
> Task :cleanPrecompileJte UP-TO-DATE
> Task :clean
> Task :checkKotlinGradlePluginConfigurationErrors
> Task :generateJte FROM-CACHE
> Task :kspKotlin FROM-CACHE
> Task :compileKotlin FROM-CACHE
> Task :compileJava NO-SOURCE
> Task :processResources
> Task :classes
> Task :kspTestKotlin
> Task :processTestResources
> Task :compileTestKotlin
> Task :compileTestJava NO-SOURCE
> Task :testClasses

BUILD SUCCESSFUL in 8s
11 actionable tasks: 7 executed, 3 from cache, 1 up-to-date
real 8.94
user 1.21
sys 0.18

And then, since the build cache was populated:

time -h -p gradle --warning-mode=none --build-cache clean testClasses
> Task :cleanGenerateJte
> Task :cleanPrecompileJte UP-TO-DATE
> Task :clean
> Task :checkKotlinGradlePluginConfigurationErrors
> Task :generateJte FROM-CACHE
> Task :kspKotlin FROM-CACHE
> Task :compileKotlin FROM-CACHE
> Task :compileJava NO-SOURCE
> Task :processResources
> Task :classes
> Task :kspTestKotlin FROM-CACHE
> Task :compileTestKotlin FROM-CACHE
> Task :compileTestJava NO-SOURCE
> Task :processTestResources
> Task :testClasses

BUILD SUCCESSFUL in 1s
11 actionable tasks: 5 executed, 5 from cache, 1 up-to-date
real 2.40
user 1.26
sys 0.16

marcospereira · 2023-11-03T03:38:40Z

@casid, I'm trying to figure out how to add a test for this, though.

codecov · 2023-11-03T03:42:35Z

Codecov Report

All modified and coverable lines are covered by tests ✅

Comparison is base (241efe8) 91.21% compared to head (5170ff2) 91.15%.

Additional details and impacted files

@@             Coverage Diff              @@
##               main     #295      +/-   ##
============================================
- Coverage     91.21%   91.15%   -0.07%     
+ Complexity     1197     1196       -1     
============================================
  Files            76       76              
  Lines          3132     3132              
  Branches        484      484              
============================================
- Hits           2857     2855       -2     
- Misses          167      168       +1     
- Partials        108      109       +1

see 1 file with indirect coverage changes

☔ View full report in Codecov by Sentry.
📢 Have feedback on the report? Share it here.

edward3h · 2023-11-03T16:28:06Z

I don't have time to have a detailed look, but I skimmed through and it seems good.

casid · 2023-11-05T19:56:48Z

Unfortunately, I'm not really a gradle expert, so I can't help much with this. :-(

The tests run in another step already.

.github/workflows/maven.yml

jte-gradle-plugin/build.gradle

test/gradle-test-wrapper/build.gradle

marcospereira · 2023-11-08T17:31:39Z

@casid @edward3h build is green, and this is good to be reviewed. :)

casid

Thank you for your contribution @marcospereira!

I left a few notes that need to be changed, so that we don't miss platform specific bugs in the future.

@edward3h could you have a quick look at the changes to the Gradle plugin, if possible? You have a lot more expertise here as I do :-)

.github/workflows/maven.yml

test/gradle-test-wrapper/build.gradle

casid · 2023-11-10T06:18:58Z

test/gradle-test-wrapper/src/test/java/gg/jte/GradleMatrixTest.java

+
+    public static Stream<Arguments> checkBuildCache() {
+        return Stream
+                .of(Paths.get("../jte-runtime-cp-test-models-gradle"), Paths.get("../jte-runtime-test-gradle-convention"))


Hmm, what about the other test projects?

gg.jte.GradleMatrixTest#runGradleBuild()

We are only interested in testing that the tasks are cacheable and not checking every project (done in the other above).

marcospereira

Hey @casid, fixed most of the comments and added a reply to the test selection in GradleMatrixTest. See if this looks better for you now.

marcospereira · 2023-11-10T15:16:52Z

test/gradle-test-wrapper/src/test/java/gg/jte/GradleMatrixTest.java

+
+    public static Stream<Arguments> checkBuildCache() {
+        return Stream
+                .of(Paths.get("../jte-runtime-cp-test-models-gradle"), Paths.get("../jte-runtime-test-gradle-convention"))


We are only interested in testing that the tasks are cacheable and not checking every project (done in the other above).

casid · 2023-11-10T16:10:28Z

LGTM, would wait a few more days with the merge, so that @edward3h might have a chance to look at it.

edward3h · 2023-11-10T17:12:46Z

I'm busy this weekend, but I can take a look on Monday.

edward3h · 2023-11-14T13:31:59Z

I am happy with this change. Thanks @marcospereira !

marcospereira · 2023-11-14T14:49:01Z

I am happy with this change. Thanks @marcospereira !

Awesome! Thanks for taking the time to review, @edward3h.

@casid, I guess this is good to merge/release then. 🍾

casid · 2023-11-15T16:15:32Z

Thanks everyone!

marcospereira · 2023-11-20T15:15:59Z

Hey @casid, just a quick check about when we can have a release with these changes. 🚀

casid · 2023-11-21T07:00:01Z

@marcospereira I just released version 3.1.5 :-)

marcospereira force-pushed the gradle/cacheable-tasks branch from 297aa6d to 9aaeaeb Compare November 3, 2023 06:10

This comment was marked as resolved.

Sign in to view

This comment was marked as outdated.

Sign in to view

marcospereira marked this pull request as draft November 3, 2023 18:58

marcospereira added 5 commits November 7, 2023 11:11

Make Gradle plugin tasks cacheable

7941c4e

Log the process of gradle plugin tests

ae090a0

Add tests for the cacheable tasks

c3e2ccf

Favours mavenLocal to fastly resolve snapshots

48878b3

ci: Skip tests when building locally

49b09a1

The tests run in another step already.

marcospereira force-pushed the gradle/cacheable-tasks branch from 8494243 to 49b09a1 Compare November 7, 2023 16:18

marcospereira added 2 commits November 7, 2023 11:23

Check if directory exists before trying to delete

692678c

gradle: tests use a temporary dir for build cache

c998ea0

marcospereira marked this pull request as ready for review November 8, 2023 01:34

This comment was marked as duplicate.

Sign in to view

marcospereira commented Nov 8, 2023

View reviewed changes

.github/workflows/maven.yml Outdated Show resolved Hide resolved

.github/workflows/maven.yml Show resolved Hide resolved

jte-gradle-plugin/build.gradle Show resolved Hide resolved

test/gradle-test-wrapper/build.gradle Outdated Show resolved Hide resolved

casid requested changes Nov 10, 2023

View reviewed changes

marcospereira added 2 commits November 10, 2023 09:58

ci: run the tests for every supported OS

e0bb83e

gradle: Remove test-logger from gradle-test-wrapper

5170ff2

marcospereira commented Nov 10, 2023

View reviewed changes

casid approved these changes Nov 10, 2023

View reviewed changes

casid merged commit 827e4e7 into casid:main Nov 15, 2023
7 checks passed

marcospereira deleted the gradle/cacheable-tasks branch November 15, 2023 19:06

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Make Gradle tasks cacheable #295

Make Gradle tasks cacheable #295

marcospereira commented Nov 3, 2023

marcospereira commented Nov 3, 2023

codecov bot commented Nov 3, 2023 •

edited

Loading

edward3h commented Nov 3, 2023

This comment was marked as resolved.

This comment was marked as outdated.

casid commented Nov 5, 2023

This comment was marked as duplicate.

marcospereira commented Nov 8, 2023

casid left a comment

casid Nov 10, 2023

marcospereira Nov 10, 2023

marcospereira left a comment

marcospereira Nov 10, 2023

casid commented Nov 10, 2023

edward3h commented Nov 10, 2023

edward3h commented Nov 14, 2023

marcospereira commented Nov 14, 2023

casid commented Nov 15, 2023

marcospereira commented Nov 20, 2023

casid commented Nov 21, 2023

Make Gradle tasks cacheable #295

Make Gradle tasks cacheable #295

Conversation

marcospereira commented Nov 3, 2023

What?

Local validation

Before

After

marcospereira commented Nov 3, 2023

codecov bot commented Nov 3, 2023 • edited Loading

Codecov Report

edward3h commented Nov 3, 2023

This comment was marked as resolved.

This comment was marked as outdated.

casid commented Nov 5, 2023

This comment was marked as duplicate.

marcospereira commented Nov 8, 2023

casid left a comment

Choose a reason for hiding this comment

casid Nov 10, 2023

Choose a reason for hiding this comment

marcospereira Nov 10, 2023

Choose a reason for hiding this comment

marcospereira left a comment

Choose a reason for hiding this comment

marcospereira Nov 10, 2023

Choose a reason for hiding this comment

casid commented Nov 10, 2023

edward3h commented Nov 10, 2023

edward3h commented Nov 14, 2023

marcospereira commented Nov 14, 2023

casid commented Nov 15, 2023

marcospereira commented Nov 20, 2023

casid commented Nov 21, 2023

codecov bot commented Nov 3, 2023 •

edited

Loading