Embed proguard/R8 rules in kotlin-reflect artifact jar #2893

ZacSweers · 2019-12-18T00:10:45Z

The Android build pipeline can extract embedded proguard configurations from dependencies and merge them automatically. This adds a conservative proguard configuration to the kotlin-reflect JVM artifact in support of that. This focuses mostly on just retaining what's necessary for kotlin-reflect's own functionality to operate, but could be expanded if community feedback discovers other good candidate rules.

With this in place - most Android projects using R8 or Proguard should Just Work™️ with kotlin-reflect.

ZacSweers · 2019-12-18T00:11:09Z

Following conversation from https://kotlinlang.slack.com/archives/C0AVAB92L/p1576528186001900?thread_ts=1516315390.000641&cid=C0AVAB92L

CC @udalov

The Android build pipeline can extract embedded proguard configurations from dependencies and merge them automatically. This adds a conservative proguard configuration to the kotlin-reflect JVM artifact in support of that. This focuses mostly on just retaining what's necessary for kotlin-reflect's own functionality to operate, but could be expanded if community feedback discovers other good candidate rules. With this in place - most Android projects using R8 or Proguard should Just Work™️ with kotlin-reflect.

jsjeon · 2019-12-18T23:23:00Z

core/reflection.jvm/resources/META-INF/proguard/kotlinreflect.pro

@@ -0,0 +1,14 @@
+##---------------Begin: Kotlin-reflect ----------
+# Keep Metadata annotations so they can be parsed at runtime.
+-keep class kotlin.Metadata { *; }


I wonder this is not actually needed. Recently, I started working on synthesizing metadata from scratch (for R8-processed Kotlin library), and kept adding more tests. During that, I found that, even without this rule, R8 will still keep metadata (annotation) associated to classes.

More conversion/details are here: https://r8-review.googlesource.com/c/r8/+/46240/1/src/test/java/com/android/tools/r8/kotlin/metadata/MetadataRenameInParametertypeTest.java#70

Ah, one caveat is, though, R8 with this fix (https://r8-review.googlesource.com/c/r8/+/44220) will keep metadata annotation without definition. For prior versions of R8, it may be the case that we still need this rule.

Yeah I think it's best to keep this as a conservative rule. We've had reports in Moshi that Metadata is stripped in android builds without a custom keep rule for some users, though unsure what version they're using or if they're just using proguard.

@jsjeon what version of R8 would that fix be in? I'm not sure how to check the latest release history or what's in them off-hand there

Ah, it's even difficult for me to check which version includes it. :\ Based on the timeline, I guess AGP 3.6 will include that.

However, that fix I mentioned was about keeping annotations associated with classes/members if the definition of annotations is missing. Therefore, if annotation class Metadata is given (via kotlin reflect artifact jar of course), i.e., not missing, it is still the case that R8 will take it away if it thinks it's not used.

I was about to mention the case like https://issuetracker.google.com/issues/146534384 where kotlin.Metadata itself is retrieved at runtime. In that case, yeah, this rule is necessary.

Sounds like it's safest to just keep this for now

jsjeon · 2019-12-18T23:23:51Z

core/reflection.jvm/resources/META-INF/proguard/kotlinreflect.pro

+
+# Keep generic signatures and annotations at runtime.
+# R8 requires InnerClasses and EnclosingMethod if you keepattributes Signature.
+-keepattributes InnerClasses,Signature,*Annotation*,EnclosingMethod


Do we need broad *Annotation* or just RuntimeVisibleAnnotation (because metadata is runtime visible) ?

I didn't know that RuntimeVisibleAnnotations was a thing! Historically any project reading annotations at runtime has always used *Annotation*, but maybe that's just a bespoke pattern that isn't needed. Let me try in our codebase

There are also at least RuntimeVisibleParameterAnnotations and RuntimeVisibleTypeAnnotations, so maybe the pattern should be RuntimeVisible*Annotations, if that works...

That seems to work b843558

jsjeon · 2019-12-18T23:24:55Z

core/reflection.jvm/resources/META-INF/proguard/kotlinreflect.pro

+-keep class kotlin.Metadata { *; }
+
+# Keep kotlin-reflect internals.
+-keep class kotlin.reflect.jvm.** { *; }


This seems way too broad. At least, we can try public APIs only.

Let me try it in our project. To be totally honest I'm not convinced this is truly needed, but it's frequently suggested as a fix in the community. The last comprehensive list we were using internally was based on this:

square/moshi#883

Though all of those are public classes as well, so it would work. @udalov any advice on what scope should be kept?

The list from the issue mentioned above is actually the combination of all services and implementations used in kotlin-reflect -- see META-INF/services in kotlin-reflect.jar. Shouldn't they be handled correctly in some other way by Proguard/R8? E.g. is it possible to "keep" the directory META-INF/services, and everything it references, to avoid enumerating classes used there?

Other than that, we do need to fully keep the contents of packages kotlin.reflect.full and kotlin.reflect.jvm, but not their subpackages (i.e. kotlin.reflect.jvm.internal).

Types declared inside META-INF/services/ are handled automatically provided that the code which calls into ServiceLoader with that service type is not being removed. You shouldn't need to do anything for them.

They're handled automatically by R8, but not proguard iirc. Can we have rules that are only picked up by one or the other?

Yes. See kotlinx.coroutines for an example that targets R8 1.6 and newer only.

Done in 08616fb

Also tweaked to just keep service-loaded APIs, only in proguard or R8 <1.6 c15aac4

Thanks for taking care of this thoroughly! Those specific rules that aim for only interface and services look great!

udalov · 2020-01-03T17:07:18Z

core/reflection.jvm/resources/META-INF/proguard/kotlinreflect.pro

@@ -0,0 +1,14 @@
+##---------------Begin: Kotlin-reflect ----------


Are these Begin/End tags required?

Also, please consider renaming the file itself to kotlin-reflect.pro

Are these Begin/End tags required?

Not needed. They will be added automatically by R8 now for blaming purposes.

https://issuetracker.google.com/issues/134372926

By R8 yes. Do we care about it in a proguard context? They're not strictly required but the blaming purpose is nice

Removed in c6f922b

ZacSweers

Took a significant pass at all the feedback and updated. The structure is now

/META-INF
  /com.android.tools
    /proguard/kotlin-reflect.pro       // Proguard-only, with custom serviceloader rules
    /r8-from-1.6.0/kotlin-reflect.pro  // R8 1.6+ with serviceloader handling
    /r8-upto-1.6.0/kotlin-reflect.pro  // R8 <1.6 with customservice loader rules
  /proguard/kotlin-reflect.pro         // Default proguard rules location, with custom serviceloader rules

Let me know what you think

jsjeon · 2020-01-08T20:54:37Z

core/reflection.jvm/resources/META-INF/com.android.tools/proguard/kotlin-reflect.pro

+-keep class kotlin.Metadata { *; }
+
+# Keep implementations of service loaded interfaces
+# R8 will automatically handle these these in 1.6+


Nit: this file is not quite related to R8. :) (and below too)

D'oh, fixed in 5091302

ZacSweers · 2020-01-14T04:57:45Z

@udalov anything else needed for this PR?

core/reflection.jvm/resources/META-INF/com.android.tools/proguard/kotlin-reflect.pro

udalov · 2020-01-17T19:14:01Z

@ZacSweers Thank you for the contribution!

Merged manually in 482874f. The changes will be available in 1.4-M1.

ZacSweers · 2020-01-17T20:54:45Z

Thanks!

ColtonIdle · 2021-04-20T19:01:34Z

@udalov any chance something changed here in 1.4.32?

We noticed this issue happening but it wasn't happening in the past (we've been using proguard since kotlin 1.4) but with 1.4.32 we need to manually add

-keep class kotlin.reflect.jvm.internal.** { *; }

for our app not to crash.

Edit: Filed on youtrack here as there was no github issue tab. Hope that's the right place. https://youtrack.jetbrains.com/issue/KT-46236

ZacSweers · 2021-04-20T20:22:11Z

File an issue with repro steps, commenting on an old PR isn't really the best venue

ZacSweers force-pushed the z/kotlinreflectProguardRules branch from 4c3c64a to 7bb0f84 Compare December 18, 2019 00:13

Fix wildcard

3cd4251

ZacSweers mentioned this pull request Dec 18, 2019

ProGuard configuration for Kotlin 1.2 square/moshi#402

Closed

udalov added the Reflection label Dec 18, 2019

udalov self-assigned this Dec 18, 2019

jsjeon reviewed Dec 18, 2019

View reviewed changes

udalov reviewed Jan 3, 2020

View reviewed changes

ZacSweers added 8 commits January 6, 2020 00:20

Just keep service loaded APIs

c15aac4

Use RuntimeVisible*Annotations

b843558

Nix headers

c6f922b

Rename to kotlin-reflect.pro

8af97da

Don't warn on relocated internal classes

82aa4e3

Split files across com.android.tools

08616fb

Remove service loader rules for r8 1.6+

a98b553

Generify wording

4e59c9f

ZacSweers commented Jan 6, 2020

View reviewed changes

jsjeon reviewed Jan 8, 2020

View reviewed changes

Nix R8 mention in proguard-only file

5091302

udalov reviewed Jan 15, 2020

View reviewed changes

core/reflection.jvm/resources/META-INF/com.android.tools/proguard/kotlin-reflect.pro Outdated Show resolved Hide resolved

Nix enclosing method and signature from proguard

a7a9f2b

udalov closed this Jan 17, 2020

ZacSweers mentioned this pull request Jan 17, 2020

Remove kotlin-reflect, replace with kotlinx-metadata-jvm square/moshi#1076

Closed

ganadist mentioned this pull request Aug 11, 2021

Provide proguard/r8 rules google/gson#1929

Open

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Embed proguard/R8 rules in kotlin-reflect artifact jar #2893

Embed proguard/R8 rules in kotlin-reflect artifact jar #2893

ZacSweers commented Dec 18, 2019

ZacSweers commented Dec 18, 2019

jsjeon Dec 18, 2019 •

edited

Loading

jsjeon Dec 18, 2019

ZacSweers Dec 23, 2019

ZacSweers Jan 6, 2020

jsjeon Jan 8, 2020

ZacSweers Jan 14, 2020

jsjeon Dec 18, 2019 •

edited

Loading

ZacSweers Dec 23, 2019

udalov Jan 3, 2020 •

edited

Loading

ZacSweers Jan 6, 2020

jsjeon Dec 18, 2019

ZacSweers Dec 23, 2019

udalov Jan 3, 2020

JakeWharton Jan 3, 2020

ZacSweers Jan 6, 2020

JakeWharton Jan 6, 2020

ZacSweers Jan 6, 2020

ZacSweers Jan 6, 2020

jsjeon Jan 8, 2020

udalov Jan 3, 2020

JakeWharton Jan 3, 2020

ZacSweers Jan 6, 2020

ZacSweers Jan 6, 2020

ZacSweers left a comment

jsjeon Jan 8, 2020

ZacSweers Jan 14, 2020

ZacSweers commented Jan 14, 2020

udalov commented Jan 17, 2020

ZacSweers commented Jan 17, 2020

ColtonIdle commented Apr 20, 2021 •

edited

Loading

ZacSweers commented Apr 20, 2021

		@@ -0,0 +1,14 @@
		##---------------Begin: Kotlin-reflect ----------

Embed proguard/R8 rules in kotlin-reflect artifact jar #2893

Embed proguard/R8 rules in kotlin-reflect artifact jar #2893

Conversation

ZacSweers commented Dec 18, 2019

ZacSweers commented Dec 18, 2019

jsjeon Dec 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

jsjeon Dec 18, 2019 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

udalov Jan 3, 2020 • edited Loading

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZacSweers left a comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

Choose a reason for hiding this comment

ZacSweers commented Jan 14, 2020

udalov commented Jan 17, 2020

ZacSweers commented Jan 17, 2020

ColtonIdle commented Apr 20, 2021 • edited Loading

ZacSweers commented Apr 20, 2021

jsjeon Dec 18, 2019 •

edited

Loading

jsjeon Dec 18, 2019 •

edited

Loading

udalov Jan 3, 2020 •

edited

Loading

ColtonIdle commented Apr 20, 2021 •

edited

Loading