Add support for Gradle Configuration Cache

[nedtwigg]

Gradle 6.6 introduces the configuration cache. It looks useful, and it would be great to refactor Spotless to support it. I am okay with bumping our minimum required Gradle in order to support it. gradle/gradle#13490

[nedtwigg]

I took a quick look at this. The first stumbling block is that we use buildFinished:

spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/RegisterDependenciesTask.java

Lines 94 to 100 in 41ed32d

	getProject().getGradle().buildFinished(new Closure(null) {
	@SuppressFBWarnings("UMAC_UNCALLABLE_METHOD_OF_ANONYMOUS_CLASS")
	public Object doCall() {
	gitRatchet.close();
	return null;
	}
	});

Googling gradle configuration cache buildFinished surfaces runningcode/gradle-doctor#71, which links to several random commits, of which runningcode/gradle-doctor@120d920 is the most useful. Seems that we can use Gradle::getSharedServices coupled with BuildService to get around this. Probably the BuildService will be a generally-useful part of getting ourselves buildcache friendly...

[nedtwigg]

We're pretty close to being able to ship full support for this. But it's a bit nasty as-implemented, and it could be clean if there were some clarity about how BuildService and ConfigurationCache interact.

• Spotless as it ships today (5.6.1) doesn't interfere with other plugins that already support config cache (huzzah)
• If a Spotless task somehow gets configured, then it breaks config cache for everybody )
  • Fixed 5.15.1 by Configuration cache, part 1 (play nice with others) #720
• To make Spotless actually work with configuration cache is a little more work
  • PR Configuration cache (JVM-local support only, just an idea) #721 implements this, but we're blocked by Improve configuration cache support for Java Object Serialization gradle/gradle#13588, but we could workaround
  • We also use a questionable hack, which I'll describe below

Questionable hack

One of Spotless' features is that you can specify a function in your buildscript, and use that function as a formatter, e.g.:

spotless { format 'blah', {
  custom 'lowercase', { str -> str.toLowerCase() }

However, that lambda is not serializable, so IIUC we'll never be able to support that fully with the traditional configuration cache model of "serialize every task input, execute based on the deserialization of those inputs". However, if there is some kind of ConfigurationCacheService which allows me to take objects that executed during the initial caching, and keep them alive in the daemon for future invocations, then I have a workaround. As it happens, that already exists, by abusing static variables:

• I have this simple little cache in a static variable

• spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/SpotlessTaskService.java

  Line 35 in 3986fe6

  private Map<String, SpotlessTaskImpl> map = new HashMap<>();

• when configuration actually runs, I store the "original" task in that cache

• spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/SpotlessTaskImpl.java

  Lines 78 to 83 in 3986fe6

  	@javax.inject.Inject
  	public SpotlessTaskImpl(FileSystemOperations fileSystemOperations, ObjectFactory objectFactory) {
  	this.fileSystemOperations = fileSystemOperations;
  	this.objectFactory = objectFactory;
  	this.projectDir = getProject().getProjectDir();
  	SpotlessTaskService.instance().put(this);

• and when I'm executing, I grab objects from the original

• spotless/plugin-gradle/src/main/java/com/diffplug/gradle/spotless/SpotlessTaskImpl.java

  Lines 51 to 73 in 3986fe6

  	Formatter buildFormatter() {
  	// <sketchy configuration cache trick>
  	SpotlessTaskImpl original = SpotlessTaskService.instance().get(getPath());
  	if (original == this) {
  	// a SpotlessTask is registered with the SpotlessTaskService **only** if configuration ran
  	// so if we're in this block, it means that we were configured
  	return Formatter.builder()
  	.lineEndingsPolicy(lineEndingsPolicy)
  	.encoding(Charset.forName(encoding))
  	.rootDir(getProjectDir().toPath())
  	.steps(steps)
  	.exceptionPolicy(exceptionPolicy)
  	.build();
  	} else {
  	// if we're in this block, it means that configuration did not run, and this
  	// task was deserialized from disk. All of our fields are ".equals" to their
  	// originals, but their transient fields are missing, so we can't actually run
  	// them. Luckily, we saved the task from the original, just so that we could restore
  	// its formatter, whose transient fields are fully populated.
  	return original.buildFormatter();
  	}
  	// </sketchy configuration cache trick>
  	}

This nasty hack works, but it only work if the following holds true:

• when a configuration is cached, the classloader which was used for that caching will be reused for subsequent invocations (holds true right now)
• the classloader will not be used for anything else in-between the caching and subsequent invocations (probably not guaranteed)

IMO, the ability to specify arbitrary functions in a buildscript is a defining value-add of Gradle vs other systems. The configuration cache feature is fantastic, but if I have to give up those arbitrary functions (in this and other plugins), then the caching will always be a second-class feature. To be a first-class, always-works feature, it seems like there needs to be some way to take live objects from the caching, and store them for future invocations.

Interplay between Configuration Cache and BuildService

If you use BuildService as a property on a task (as is recommended), that service gets serialized and deserialized with the task, which is kind of surprising. I'm guessing (but haven't confirmed) that the lifecycle of the service doesn't get managed properly in the configuration cache scenario. The BuildService API is also ambiguous about what counts as "absent" to registerIfAbsent - is it unique per String name, or per String name, Class<T> class, or is the Action<T> init included in there somehow too?

I think it would be a lot clearer, for both BuildService and for the "Configuration Cache Store" that I'm asking for, to have something like this:

public <T> T ObjectFactory.onePerBuild(Class<T> clazz)
public <T> T ObjectFactory.onePerConfigCache(Class<T> clazz)

IMO it's a lot easier to test things when I'm not forced to make them abstract (not a fan of needing to leave methods abstract), and the contract of what Gradle is providing me with is clearer than BuildService.''

Overloaded "configuration"

I get confused by "configuration" the set of dependencies, versus "configuration" the process of executing build scripts and configuring tasks. Also configure is long to type. If I could ask for a breaking change in Gradle, it would be to rename executing build scripts to "setup" - "setup avoidance", "setup-cache". Maybe "wiring" or "init"? Just something unique, and shorter.

[nedtwigg]

@bigdaz are you guys looking for feedback on configuration cache? Super awesome feature, I got it mostly working in Spotless with some hacks, my experience is in the comment above.

[jbduncan]

Oh wow, the code in #721 is going to be... hard to maintain, to put it diplomatically. If there's anything we could do to simplify it, that would be a big win for me. So I'm tempted to wait until we either get some acknowledgement or feedback from the Gradle team or until the configuration cache feature has been developed some more.

In the meantime, I'm happy for #720 to be merged in. On a quick scan, it looks relatively simple.

[nedtwigg]

I agree we should wait for feedback from Gradle folks, and/or more clarity in the docs around it. Merging #720 would make us depend on @Incubating BuildService, and I think it's a not very good API, ObjectFactory.onePerBuild would be better. So if somebody is held-up by Spotless not supporting configuration cache, they can use our jitpack instructions to get either #720 or #721 and then they'll have whatever level of configuration cache support they want. Assuming configuration cache becomes mainstream (I hope it will!), they will both get merged eventually, but there is no rush.

[bigdaz]

I don't have much insight into the Configuration Cache, so I'm not going to be much help. I'll mention @eskatos here since he can likely help.

The best way to have a conversation about this would be on the Gradle Community Slack in the #configuration-cache channel.

[eskatos]

I left a comment about the usage of build services for your use case on #720.

About the formater as a function use case. The user function can be given as an explicit type, an anonymous class, a Java lambda, a Groovy closure, a Kotlin lambda etc... The configuration cache supports serializing some of those already but not all of them in all cases. It shouldn't be a plugin concern though. You can keep your existing API and let users that want to use the configuration cache pass the formater function in a way that is currently supported. Eventually the configuration cache will support more cases.

The nasty hack described above using static state won't fly. The configuration cache can be reused by fresh daemons that won't have the static state initialized.

[nedtwigg]

The configuration cache can be reused by fresh daemons that won't have the static state initialized

Roger. That's the nail in the coffin for our existing design, thanks for the clarification.

It seems like there's a design space of increasing complexity which looks like this:

1. Cache output artifacts from build-to-build on this machine - local build cache
2. Cache output artifacts from build-to-build on any machine - remote build cache
3. Cache configuration from build-to-build on this JVM - JVM-local configuration cache (unplanned)
4. Cache configuration from build-to-build on any JVM on this machine - configuration cache (as it exists)
5. Cache configuration from build-to-build on any machine - remote configuration cache (on the roadmap)

Did you explore option 3 and reject it? It seems like option 3 ought to be faster than 4 or 5 can be, and it's also much easier to implement in some cases because it doesn't require serialization support.

The improvements you guys are making are bringing Gradle's end-to-end runtime out of "build tool" territory and into "fully interactive", which I think is really exciting. The faster Gradle gets, the less important multiple daemons on a single project becomes. Even at its current speed, my usage model almost never requires a second daemon. If it's possible to make "configuration cache" faster for 90% of gradle invocations, at the cost that it only persists across invocations on a single daemon, that seems like a better tradeoff than making it a little slower, with the benefit that it can be serialized to disk and sent to other machines.

Especially given that local & remote buildcache already exist with broad support, it seems like JVM-local caching is the missing hole to allow next-level interactive performance from Gradle. If that hole is ever going to be filled, then I think Spotless will wait for that. And if it's not ever going to be filled, why not?

[eskatos]

Those are very good points. We indeed are considering keeping the cached state warm in the daemon for the next invocation. But we also want to be able to share it between machines. 3/ might not happen soon though.

That being said I don't see why you would be blocked until this happen.

[nedtwigg]

That being said I don't see why you would be blocked until this happen.

You are absolutely right that we can implement something which is compatible with option 4 and 5, and doing so would be nice. But long ago, back in Gradle 2.x, we made the choice to abuse serialization to implement equality. If we want to ignore a field for the purpose of equality (such as a local path), we mark it transient. Definitely a hack, but it has allowed us to build a lot features quickly, without doubling our codesize with .equals and .hashCode

To support option 3 is trivial for us. To support option 4 or 5 is pretty close to a rewrite, and at the end we lose functionality because we're constrained by which things are serializable, and we're a little bit slower than we are with option 3. If Spotless users had option 3 (plus the build cache they already have), I don't think they would mind very much if we didn't support 4 or 5.

• 90%+ of builds can reuse a daemon (as builds get faster, this percentage increases)
• with option 3 the first configure is 3 seconds, and every one after that is 0.2 seconds
• but instead of that, we can rewrite our codebase so that we can stream the configuration cache from the build server, so that every single configure is consistently 0.5 seconds (downside), even the first one (upside, if it happens to be cached)

The configuration cache is a brilliant idea, but it seems like it's being approached backwards, fighting the last battle (buildcache), and long-term restricts what plugins can do. My complaint isn't just that random legacy decisions make it hard in Spotless, it's that my favorite gradle feature is that plugins can accept a function as a build input. I use it all over the place, not just Spotless, it makes my plugins shorter to write, and more powerful and flexible to use.

I love it when other plugins provide function-based APIs too. Configuration cache 3 keeps these as first class citizens, and is faster, and puts a lower cognitive overhead on plugin developers. "Your task inputs stay in memory". If the per-build performance tradeoff of "your task inputs stay in memory" hasn't been compared against the current approach, then I would start there. It's also so simple to implement and document, relative to serializing random lambdas which might capture who knows what. I don't see how that can be done without huge provisos, so I don't see how Gradle doesn't lose "functions as parameters". I don't want "Bazel but with Groovy", I want Gradle!!

[eskatos]

Plugins can still accept functions. It's just that those functions must be serializable. We don't want to lose the convenience. It already works with Groovy closures and Kotlin lambdas. That covers most of the cases. Java lambdas can be problematic though, and this is a problem since forever, not only for the configuration cache, see e.g. gradle/gradle#10751

[nedtwigg]

Thanks for taking the time to dig up the unit test, I don't know how you did it but you did it! ;-)

I think there's a userland way we can hack option 3 ourselves (cleaner than the hack I proposed before, and we can make it opt-in). It allows us to do our rewrite piece-by-piece, rather than all at once. You've got more data on how people use Gradle than I do, and I'm on a small team so central caching is less important to me than to your customers. I'm always impressed by the Gradle team, thanks for helping us understand the APIs.

[eskatos]

Happy to help, and thank you for writing up your problems and concerns so clearly. It's so easy to help when things are crystal clear! I'm looking forward to a release of the spotless plugin that enables using the configuration cache.

[marcphilipp]

@nedtwigg Are there any updates on configuration cache support?

[nedtwigg]

Hoping to take a shot at it in a couple months. Happy to take a PR jf anybody beats me to it :)

[breskeby]

@nedtwigg we realised that spotless failed the build when configuration cache is enabled with the following message:

* What went wrong:
Configuration cache state could not be cached: field 'lineEndingsPolicy' from type 'com.diffplug.gradle.spotless.SpotlessTaskImpl': error writing value of type 'com.diffplug.spotless.extra.GitAttributesLineEndings$RelocatablePolicy'
> Configuration cache state could not be cached: field 'hasNonDefaultEnding' from type 'com.diffplug.spotless.extra.GitAttributesLineEndings$CachedEndings': error writing value of type 'com.googlecode.concurrenttrees.radix.ConcurrentRadixTree'
   > Configuration cache state could not be cached: field 'writeLock' from type 'com.googlecode.concurrenttrees.radix.ConcurrentRadixTree': error writing value of type 'java.util.concurrent.locks.ReentrantLock'
      > Unable to make field private final java.util.concurrent.locks.ReentrantLock$Sync java.util.concurrent.locks.ReentrantLock.sync accessible: module java.base does not "opens java.util.concurrent.locks" to unnamed module @31e3c90b

is that something you have on your agenda?

[nedtwigg]

I have been using configuration-cache for a few months now, and it is a nice feature. I use configuration cache for good performance on all testing and packaging tasks, and then I use --no-configuration-cache just for Spotless. For my builds, spotless is still far faster even without configuration cache than the rest of my day-to-day tasks are with configuration cache.

So I'm happy to merge a PR, but for the workflows I have, it is not a limiting factor.

In particular, imo the Gradle design has picked the wrong design point (more details here). If your design goal is a machine-independent cache, that comes with serialization constraints and serialization overhead. If your design goal is single-user low-latency interactivity, then a JVM-local cache removes the design and performance limitations imposed by serialization.

Personally, if I were going to put time into configuration cache, I would first build out JVM-local caching, which would be much faster, and also much easier to implement. Regardless, happy to merge a PR :)

[nedtwigg]

I've got a line of PRs setup which lets configuration-cache run.

• Resolve dependencies from "project" repositories, not buildscript #980
• SpotlessTaskService (towards configuration-cache) #982
• Resolve dependencies per-project #983

You quickly get cryptic errors, so I won't merge and release until we can safely flag which parts of Spotless do and don't support configuration cache. But at least the heavy lifting is now done. I expect 6.0 to get released sometime next week. I'm gonna let these PR's percolate in case anybody has comments before I merge them all.

[nedtwigg]

Spotless is working well with configuration-cache in my testing, but limited to a single daemon. Fair warning @eskatos, our users are going to see this error message anytime they use configuration-cache from a different daemon than the one that wrote the cache:

Spotless JVM-local cache is stale. Regenerate the cache with
  rm -rf .gradle/configuration-cache
To make this workaround obsolete, please upvote https://github.com/diffplug/spotless/issues/987

And in #987, I make the case I made above that configuration-cache ought to design for minimum time-to-iteration, and that serialization round-trip is an unnecessary cost, and especially that relocatability is going to be difficult.

But for common workflows, Spotless is soon to be configuration-cache happy (leaving the PR's to simmer for another day or two before releasing 6.0)

[nedtwigg]

Support for configuration-cache is now available in 6.0.0, see release notes for details. It's... really, really fast now :)

[Maragues]

Hi, even after updating to 6.0.3, Android studio reports spotless as not compatible with configuration cache

Any hint?

We use includeBuild, in case it makes any difference

Can i help by providing any logs when built from console?

This is reported when building assembleDebug, not even a spotless task.

Thanks a lot!

[nedtwigg]

I don't know anything about Android Studio, I just know that I'm using Spotless 6.0.3 with configuration-cache on :) Have you tried turning configuration cache on explicitly with org.gradle.unsafe.configuration-cache=true in your gradle.properties?

[Maragues]

You seem to be right, Gradle doesn't report any issue on spotless when run from console

./gradlew --configuration-cache help

Thank you very much and sorry for the noise

[nedtwigg]

No worries, probably others will have the same issue 👍

[eskatos]

@nedtwigg, we get lots of reports of users thrown off by the way this is currently working. The fact that it works differently than any other Gradle plugin is surprising. One needs to make a manual action on some "internal" files on disk to continue its workflow when a Gradle daemon expire or when starting a new daemon because the other one is busy or incompatible (e.g. cli vs IDE).

I understand keeping your static cache is important performance wise. I also understand that JVM-local only configuration cache would make this easier for you, but it isn't planned. Could there be a way for the Spotless plugin to automatically rebuild that cache on fresh daemons loading from the configuration cache?

[nedtwigg]

...static cache is important performance wise...

The real blocker for us is that we cannot "rehydrate" ourselves - we do not support round-trip serialization. The reason we don't is that we abused the Java serialization mechanism to quickly implement equality checks, using transient as a quick way to say "ignore this field for the sake of equality checks". Fixing this would be a large project for us, and it would generate a lot of serialization/equality boilerplate for us (unless we switched to Kotlin or Java records, which makes the project still large but at least more manageable).

I do think the Gradle configuration cache is designed to the wrong usecase (too slow for users, too many constraints for build developers), but I would happily submit to the ecosystem if I had time to rewrite our code. The workaround we are using right now is a good performance choice, but we aren't doing it for performance, it's just the easiest way we could implement support for round-trip serialization. It works well enough that no one has allocated enough of their time to submit a PR redoing all of the equality / serialization code for every formatter we support.

Could there be a way for the Spotless plugin to automatically rebuild that cache on fresh daemons loading from the configuration cache?

Yes! We want to work easily with the ecosystem, and we have an issue to automatically do the rm -rf <blah> and then retry the user's command. It's a risky approach though, and I doubt that anyone will actually implement it.

• Automatically clear gradle configuration cache and retry on stale-cache failure #1209

Even better for us would be if there was org.gradle.api.TriggerConfigurationError which we could throw during deserialization. If Gradle sees this error, it will trigger a rerun of the configuration phase and discard the existing cache.

TL;DR

1. we could adopt Kotlin or Java records and rewrite most of our codebase to properly separate serialization from equality
2. we could implement Automatically clear gradle configuration cache and retry on stale-cache failure #1209
3. you could implement TriggerConfigurationError or a JVM-local cache

IMO, a JVM-local cache "devcache" is the best long-term solution, but I understand that it is not planned. There are things we can do ourselves at Spotless, and we're happy to take a PR from anyone that accomplishes this, but it has apparently not been worth the cost/benefit to anyone so far.

FWIW, we just did some work to fix a warning about Task.usesService for Gradle 8.0, and I'm pretty confused by the design there too, but it was easy for us to meet Gradle where it is and fix the warning.

[eskatos]

Hi Ned,
Thanks for your reply.

Just to clarify, I was not suggesting to rehydrate when using a different daemon but to recalculate instead. In principle, make your static cache transient and recalculate when it is not available.

I don't think #1209 is a good idea. It would just automate something wrong from the Gradle pov: fiddling with internal state files and nuking all cache entries that are for task graphs unrelated to Spotless.

[nedtwigg]

If we can throw TriggerConfigurationError and ask Gradle to run us to recalculate, that would work for us, but obviously that's a lot of work for you just for us, and it probably doesn't actually make sense for you to implement. Here's an example of why we can't recalculate ourselves.

Here is the state that defines how to run the "Black" python formatter (note that it does not implement hashCode or equals).

spotless/lib/src/main/java/com/diffplug/spotless/python/BlackStep.java

Lines 73 to 79 in 3878d36

	static class State implements Serializable {
	private static final long serialVersionUID = -1825662356883926318L;
	// used for up-to-date checks and caching
	final String version;
	final transient ForeignExe exe;
	// used for executing
	private transient @Nullable String[] args;

We have transient fields which model how to call the black formatter, and the only non-transient field is String version. This means that for our up-to-date checks, including up-to-date w.r.t. build cache (classic), all we care about is the version of Black, we don't care about the path to the executable, etc.

When Spotless gets loaded in a new daemon, we know what version to use, but we don't have values for any of the transient fields. To fix this, we have to rewrite the "State" class, so that two states with different serialized forms will still .hashCode and .equals to the same thing as long as they have the String version. Decoupling serialization and equality adds a DOF to our system (expensive!). We have to do this for ~30 different FormatterState classes, and also for every piece of infrastructure that uses them which is also currently built on the assumption that "serialized form = equality-normalized form".

At the end of all this work, we'll fully support the Gradle cache model, but we'll be a little slower, and adding a new formatter will be a little bit more complicated than it used to be, because now each step has to support round-trip serialization, and we have to really audit the hashCode and equals because we're not wrapping that entirely through serialization anymore.

Round-trip serialization is a big constraint to put on a system. "Normalized form" is a nice trick for defining equality and serialization in one shot, if you're okay with giving-up roundtrip. It's a trick we lose when we implement round-trip serialization.

[eskatos]

I meant to make your whole cache transient wrt. the configuration cache. I'm not sure I recall correctly how you use Java serialization internally.

As a heads up, gradle/gradle#14624 will be implemented relatively soon. It means that the serialization roundtrip will be done on the first invocation. This might make the current Spotless workaround not to work at all anymore.

[EDIT] well, it might still work the same way, I'm not sure about your implementation. But you might want to consider the fact that this is the way Gradle will end up working.

[nedtwigg]

Our hack lets us do serialization roundtrips within a single JVM (by not actually doing the serialization). When Gradle requires multiple JVMs to do a single build, our approach will break, but until then we will keep working.

[eskatos]

Hi Ned,

Gradle does require multiple JVMs: the Gradle daemon can expire. There are many use cases for which it happens: not running a build for 3h, going back home to get some sleep and working again in the morning etc.. :)

The configuration cache should get promoted to a stable feature by the end of this year. It will eventually be made the default, tentatively a year after the promotion to stable. Many folks are already using the configuration cache and the current workflow with Spotless is creating friction (e.g. folks wondering if it should be recommended). The promotion to stable will bring a lot more users.

Static state in plugins has been discouraged for ages. IIUC, currently the static state of the Spotless plugin cannot be recreated by a new Gradle daemon with the configuration cache enabled. If the state is kept static, couldn't the cache population be moved at execution time to a task (that can be up-to-date)? Otherwise, Gradle offers means to declare logic that needs to happen even when the configuration phase is skipped: shared build services. This might help.

I guess what I'm after is an acknowledgment that Spotless currently doesn't fully support Gradle's configuration cache and a path forward to make it work for everyone.

We can think of less bold changes in Gradle that could help but changing the configuration cache base contract (in-jvm cache) or how the Gradle phases work (TriggerConfigurationError) are way too bold options.

[nedtwigg]

I wish we were in person so I could be sure to convey the goodwill that I mean to convey. Fwiw I feel goodwill from you :)

We can think of less bold changes in Gradle that could help but ... are way too bold

Gradle does require [round-trip serialization of all task state across] multiple JVMs.

I didn't suggest a bold change from you until you required one from me :). And I'm not requiring anything from you, I'm telling you the workaround I am using to get the fastest build that I can get. You can say "A plugin in good standing must serialize its tasks between JVMs", but as a user I can say "Ease of plugin development and interactive serial performance are more important to us than supporting Gradle's full feature set".

You can sell a toaster that says "You must have multiple toasters in case you want to move bread from one toaster to another", but I can still just have one toaster and not move bread. That will work for me, and so I will never put in the work to support moving bread because I don't actually have to ever move bread.

I do not want to be a thorn in your side, and if you want Spotless to abandon our workaround I am willing to merge a PR that does that. I have described what it would take - it's a huge job, and at the end Spotless is slower, and it's harder to add new formatters to Spotless, but I'm willing to merge it because I am not seeking to be a problem. I have outlined the work required informally in a few places, but I have added a more detailed description here

• Support Gradle configuration cache on multiple JVMs  #1274

Static state in plugins has been discouraged for ages

• We did not have static state.
• But our tasks did not support round-trip serialization.
• So, we added static state as a workaround, so that we could do round-trip serialization within a single JVM.

[eskatos]

Hi Ned,
Goodwill this is!
Thanks for the detailed description in the other issue.
FWIW, I personally won't be able to work on that in the near future.