Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[generator] Add --with-javadoc-xml=FILE support. #687

Merged
merged 1 commit into from
Jan 4, 2021

Conversation

jonpryor
Copy link
Member

@jonpryor jonpryor commented Jul 31, 2020

Fixes: #642

Context: dotnet/android#4789
Context: dotnet/android#5200

Commit 69e1b80 added tools/java-source-utils, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

    # API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
    # Skip the `.annotated.java` files
    $ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
            | grep -v '\.annotated\.' > sources.txt
    $ java -jar java-source-utils.jar -v \
            --source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
            --output-javadoc android-javadoc.xml \
            @sources.txt

What can we do with the generated android-javadoc.xml?

android-javadoc.xml contains parameter names, and thus can be used
with class-parse --docspath; see commit 806082f.

What we really want to do is make the Javadoc information useful
to consumers of the binding assembly. This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit C# XML Documentation Comments into the binding source code!

Add a new generator --with-javadoc-xml=FILE option. When specified,
FILE will be treated as an XML file containing the output from
java-source-utils.jar --output-javadoc (69e1b80), and all
<javadoc/> elements within the XML file will be associated with C#
types and members to emit, based on the //@jni-signature and
//@name attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to Java.Interop.Tools.JavaSource.dll,
in the new Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we also want to provide "upstream
information" in the form of:

  1. A URL to the corresponding online Javadoc HTML documentation.
  2. A copyright notice disclaimer.

Allow provision of this information by updating
java-source-utils.jar to support the new options:

    --doc-copyright FILE   Copyright information for Javadoc.  Should be in
                             mdoc(5) XML, to be held within <remarks/>.
                             Stored in //javadoc-metadata/copyright.
    --doc-url-prefix URL   Base URL for links to documentation.
                             Stored in //javadoc-metadata/link/@prefix.
    --doc-url-style STYLE  STYLE of URLs to generate for member links.
                             Stored in //javadoc-metadata/link/@style.
                             Supported styles include:
                             - developer.android.com/reference@2020-Nov

The new /api/javadoc-metadata/link@prefix and
/api/javadoc-metadata/link@style XML attributes, stored within
java-source-utils.jar --output-javadoc XML output, allow
construction of a URL to the Java member.

For example, given:

    java -jar java-source-utils.jar \
            --doc-url-prefix https://developer.android.com/reference \
            --doc-url-style developer.android.com/reference@2020-Nov

Then generator can emit the C# documentation comment for
java.lang.Object.equals(Object):

    /// <format type="text/html">
    ///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
    /// </format>

The copyright notice disclaimer is supported by
java-source-utils.jar --doc-copyright FILE; the contents of FILE
are inserted into the /api/javadoc-metadata/copyright element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

Unfortunately, converting Javadoc to C# XML Documentation Comments is
not a "zero cost" operation. Add a new
generator --doc-comment-verbosity=STYLE option to control how
"complete" the generated documentation comments are:

    --doc-comment-verbosity=STYLE
                           STYLE of C# documentation comments to emit.
                             Defaults to `full`.  STYLE may be:
                               * `intellisense`: emit <summary>, <param>,
                                 <returns>, <exception>.
                               * `full`: plus <remarks>, <altmember>, ...

Using --doc-comment-verbosity=full will most impact build times.

TODO:

  • SourceJavadocToXmldocParser doesn't support many constructs.

@jonpryor jonpryor requested a review from jpobst July 31, 2020 19:58
@jonpryor
Copy link
Member Author

@jonpryor
Copy link
Member Author

jonpryor commented Aug 3, 2020

TODO:

  • Emit docs for properties.
  • Double-check constructors
  • Unit tests!
  • use regular expressions for javadoc matching, as per 69e1b80:

    In some scenarios, types won't be resolvable. What should output be?

    docs.xml will output L.*UnresolvableType;.

@jonpryor
Copy link
Member Author

jonpryor commented Aug 3, 2020

What does an "unresolved type" really look like? Example:

$ java -jar bin/Debug/java-source-utils.jar ~/android-toolchain/sdk/platforms/_t/android/app/Activity.java -D a.xml

In this case, ~/android-toolchain/sdk/platforms/_t contains the extracted contents of platforms/android-29/android-stubs-src.jar.

java-source-util.jar -s ~/android-toolchain/sdk/platforms/_t*wasn't* used, so *all* types referenced byActivity.java` can't be resolved. The result?

<api api-source="java-source-utils">
  <package jni-name="android/app" name="android.app">
    <class jni-signature="Landroid/app/Activity;" name="Activity">
      <javadoc>
…

The /api/package/class/@jni-signature value is correct, as we know the package from the source code.

      <method jni-return="V" jni-signature="(.*android.os.CancellationSignal.*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;)V" name="onGetDirectActions" return="void">
        <parameter jni-type=".*android.os.CancellationSignal" name="cancellationSignal" type=".*android.os.CancellationSignal"/>
        <parameter jni-type=".*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;" name="callback" type=".*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;"/>
        <javadoc>
          <![CDATA[Returns the list of direct actions supported by the app.

<method/> (and potentially <field/>, if the field is for a reference type) is where the "unresolved type" scenario comes in.

Additionally -- and unexpectedly, as I hadn't thought to test this condition (😞), is that generic type parameters are present in the signature. These need to be removed in java-source-util.jar.

Additionally, the behavior doesn't match the commit message docs (double 😭), as it lacks the leading L and trailing ;. More fixes needed to java-source-util.jar.

The <method/> thus should be:

      <method jni-return="V" jni-signature="(L.*android.os.CancellationSignal;L.*java.util.function.Consumer;)V" name="onGetDirectActions" return="void">
        <parameter jni-type="L.*android.os.CancellationSignal;" name="cancellationSignal" type=".*android.os.CancellationSignal"/>
        <parameter jni-type="L.*java.util.function.Consume;" name="callback" type=".*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;"/>
        <javadoc>
          <![CDATA[Returns the list of direct actions supported by the app.

@jonpryor
Copy link
Member Author

jonpryor commented Aug 3, 2020

SourceJavadoc

@jonpryor
Copy link
Member Author

@jonpryor jonpryor force-pushed the jonp-generator-javadoc-xml branch 2 times, most recently from 740cce6 to 1ad9e10 Compare October 30, 2020 19:06
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Oct 30, 2020
Context: dotnet/java-interop#687
Context: dotnet#5200

What do we want?  Updated documentation!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`].
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus are woefully out of date.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained a `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to extract the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `anddroid-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
@jonpryor jonpryor force-pushed the jonp-generator-javadoc-xml branch 2 times, most recently from 5f641d4 to 22ac80e Compare November 2, 2020 21:28
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Nov 2, 2020
Context: dotnet/java-interop#687
Context: dotnet#5200

What do we want?  Updated documentation!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`].
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus are woefully out of date.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained a `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to extract the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `anddroid-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Nov 2, 2020
Context: dotnet/java-interop#687
Context: dotnet#5200

What do we want?  Updated documentation!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`].
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus are woefully out of date.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained a `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to extract the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `anddroid-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
@jonpryor
Copy link
Member Author

jonpryor commented Nov 5, 2020

Current Java.Lang.Object output: https://gist.github.com/jonpryor/b89999ad4f261eb561b27af921aa1cc7

Note that every member except getClass() contains Javadoc comments, as compared to previous iterations (which lacked Javadocs for e.g. Wait(), Notify()).

Properties aren't yet handled, so the lack of getClass() is expected.

@jonpryor jonpryor mentioned this pull request Nov 9, 2020
jonpryor added a commit to jonpryor/java.interop that referenced this pull request Nov 13, 2020
Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

	$ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java > sources.txt
	$ java -jar java-source-utils.jar -v \
		--source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
		--output-javadoc android-javadoc.xml \
		@sources.txt

What can we *do* with the generated `android-javadoc.xml`?

`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.

What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly.  This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!

Add a new `generator --with-javadoc-xml=FILE` option.  When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:

 1. A URL to the corresponding online Javadoc HTML documentation.
 2. A copyright notice disclaimer.

Allow provision of this information by updating
`java-source-utils.jar` to support the new options:

	--doc-copyright FILE   Copyright information for Javadoc.  Should be in
	                         mdoc(5) XML, to be held within <remarks/>.
	                         Stored in //javadoc-metadata/copyright.
	--doc-url-prefix URL   Base URL for links to documentation.
	                         Stored in //javadoc-metadata/link/@Prefix.
	--doc-url-style STYLE  STYLE of URLs to generate for member links.
	                         Stored in //javadoc-metadata/link/@Style.
	                         Supported styles include:
	                         - developer.android.com/reference@2020-Nov

The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.

For example, given:

	java -jar java-source-utils.jar \
		--doc-url-prefix https://developer.android.com/reference \
		--doc-url-style developer.android.com/reference@2020-Nov

Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:

	/// <format type="text/html">
	///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
	/// </format>

The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

  * <https://gist.github.com/jonpryor/b89999ad4f261eb561b27af921aa1cc7>

TODO:

  * Fix `java-source-utils.jar` & `//*/@jni-signature` so that
    [generics types are properly represented][1].
  * Properties don't yet emit C# XML documentation comments, e.g.
    `Java.Lang.Object.Class` has no documentation, even though
    `java.lang.Object.getClass()` is documented.
  * `SourceJavadocToXmldocParser` doesn't support many constructs.

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc
[1]: dotnet#687 (comment)
jonpryor added a commit to jonpryor/java.interop that referenced this pull request Nov 14, 2020
Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

	$ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
		| grep -v '\.annotated\.' > sources.txt
	$ java -jar java-source-utils.jar -v \
		--source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
		--output-javadoc android-javadoc.xml \
		@sources.txt

What can we *do* with the generated `android-javadoc.xml`?

`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.

What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly.  This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!

Add a new `generator --with-javadoc-xml=FILE` option.  When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:

 1. A URL to the corresponding online Javadoc HTML documentation.
 2. A copyright notice disclaimer.

Allow provision of this information by updating
`java-source-utils.jar` to support the new options:

	--doc-copyright FILE   Copyright information for Javadoc.  Should be in
	                         mdoc(5) XML, to be held within <remarks/>.
	                         Stored in //javadoc-metadata/copyright.
	--doc-url-prefix URL   Base URL for links to documentation.
	                         Stored in //javadoc-metadata/link/@Prefix.
	--doc-url-style STYLE  STYLE of URLs to generate for member links.
	                         Stored in //javadoc-metadata/link/@Style.
	                         Supported styles include:
	                         - developer.android.com/reference@2020-Nov

The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.

For example, given:

	java -jar java-source-utils.jar \
		--doc-url-prefix https://developer.android.com/reference \
		--doc-url-style developer.android.com/reference@2020-Nov

Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:

	/// <format type="text/html">
	///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
	/// </format>

The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

  * <https://gist.github.com/jonpryor/ee60a7385e908458bbe9ad48e30b2ecc>

TODO:

  * Fix `java-source-utils.jar` & `//*/@jni-signature` so that
    [generics types are properly represented][1].
  * Properties don't yet emit C# XML documentation comments, e.g.
    `Java.Lang.Object.Class` has no documentation, even though
    `java.lang.Object.getClass()` is documented.
  * `SourceJavadocToXmldocParser` doesn't support many constructs.

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc
[1]: dotnet#687 (comment)
@jonpryor
Copy link
Member Author

Current Java.Lang.Object output: https://gist.github.com/jonpryor/ee60a7385e908458bbe9ad48e30b2ecc

Notable differences from last time are:

  1. Inclusion of a Java documentation for… link

     	///       <a href="https://developer.android.com/reference/java/lang/Object">Java documentation for <tt>java.lang.Object</tt>.</a>
    
  2. A copyright notice/blurb for all imported documentation

  3. Missing member documentation for Clone(), Wait(long), and Wait(long, int). The likely cause for this is that previous runs imported the API-29 Java "stubs", while the current iteration imports the API-30 "sources" package, which java-source-utils doesn't "like".

In particular for (3), having comments between the Javadocs and the Java code appears to screw things up:

    /**
     * Creates and returns a copy of this object.  The precise meaning
     * …
     */
    // BEGIN Android-changed: Use native local helper for clone()
    // Checks whether cloning is allowed before calling native local helper.
    // protected native Object clone() throws CloneNotSupportedException;
    protected Object clone() throws CloneNotSupportedException {
…

jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Nov 14, 2020
Context: dotnet/java-interop#687
Context: dotnet#5200

What do we want?  Updated documentation!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`].
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus are woefully out of date.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained a `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to extract the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `anddroid-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
jonpryor added a commit that referenced this pull request Dec 10, 2020
Context: #687 (comment)

What happens when there's a "regular" Java comment in between a
Javadoc comment and a member?

	/* partial */ class Object {
	    /** Create and return a copy of this object… */
	    // BEGIN Android-changed: Use native local helper for clone()
	    // …
	    protected Object clone() throws CloneNotSupportedException {…}
	}

What happens is that the Javadoc becomes *orphaned*.

Commit 69e1b80 attempted to handle such orphaned Javadocs via
heuristic, using the first orphaned Javadoc comment in the parent
scope.  This didn't work reliably, as the parent scope could contain
multiple "*unrelated*" orphaned Javadoc comments:

	class Outer {
	    /** Orphaned #1 */
	    // cause orphaning
	    class Inner {}

	    void m() {}
	}

Because containing types are fully processed before contained types,
`Outer.m()` would grab the Javadoc for `Outer.Inner` before
`Outer.Inner` would have a chance to grab it.

Re-work the logic to associate orphaned Javadocs with their members,
by requiring that the Javadoc comment begin *before* the member of
interest, and *after* any preceding members.  This should prevent
incorrect correlation of orphaned Javadoc comment blocks.

Additionally, update gradle to use javaparser 3.18.0, from 3.16.1:

  * javaparser/javaparser@javaparser-parent-3.16.1...javaparser-parent-3.18.0
jonpryor added a commit to jonpryor/java.interop that referenced this pull request Dec 10, 2020
Context: dotnet#687 (comment)

Commit 69e1b80 stated:

> We don't want to *require* that everything be resolvable -- it's painful, and
> possibly impossible, e.g. w/ internal types -- so instead we should "demark"
> the unresolvable types.
>
> `.params.txt` output will use `.*` as a type prefix, e.g.
>
> 	method(.*UnresolvableType foo, int bar);
>
> `docs.xml` will output `L.*UnresolvableType;`.

…except this didn't actually happen:
`java-source.utils.jar --output-javadoc` didn't have an `L` prefix
and `;` suffix for the JNI type representation.  Additionally, it
would include generic type parameters!

Thus, a declaration such as:

	static UnknownType<String, Integer> method(AnotherUnknownType<Class> p);

would result in:

	<method name="method" jni-return=".*UnknownType&lt;String, Integer&gt;" jni-signature="(.*AnotherUnknownType&lt;Class&gt;).*UnkonwnType&lt;String, Integer&gt;">

which was not at all intended, and causes subsequent issued in PR dotnet#687
as the JNI signature parser would break when encountering `.`.

Update `java-source-utils.jar` & `//*/@jni-signature` so that
[unresolved types are properly represented][0]: `//*/@jni-type` and
`//*/@jni-signature` shouldn't contain generic type parameters, and
should also (somewhat) "conform" to JNI convention, with a leading `L`
and trailing `;`, emitting:

	<method name="method" jni-return="L.*UnknownType;" jni-signature="(L.*AnotherUnknownType;)L.*UnknownType;">

Additionally, update `JavaType.java` to cause it to have "orphaned"
Javadoc comments; this change was forgotten in c1bc5c8.

[0]: dotnet#687 (comment)
jonpryor added a commit that referenced this pull request Dec 10, 2020
Context: #687 (comment)

Commit 69e1b80 stated:

> We don't want to *require* that everything be resolvable -- it's painful, and
> possibly impossible, e.g. w/ internal types -- so instead we should "demark"
> the unresolvable types.
>
> `.params.txt` output will use `.*` as a type prefix, e.g.
>
>     method(.*UnresolvableType foo, int bar);
>
> `docs.xml` will output `L.*UnresolvableType;`.

…except this didn't actually happen:
`java-source.utils.jar --output-javadoc` didn't have an `L` prefix
and `;` suffix for the JNI type representation.  Additionally, it
would include generic type parameters!

Thus, a declaration such as:

	static UnknownType<String, Integer> method(AnotherUnknownType<Class> p);

would result in:

	<method name="method" jni-return=".*UnknownType&lt;String, Integer&gt;" jni-signature="(.*AnotherUnknownType&lt;Class&gt;).*UnkonwnType&lt;String, Integer&gt;">

which was not at all intended, and causes subsequent issued in PR #687
as the JNI signature parser would break when encountering `.`.

Update `java-source-utils.jar` & `//*/@jni-signature` so that
[unresolved types are properly represented][0]: `//*/@jni-type` and
`//*/@jni-signature` shouldn't contain generic type parameters, and
should also (somewhat) "conform" to JNI convention, with a leading `L`
and trailing `;`, emitting:

	<method name="method" jni-return="L.*UnknownType;" jni-signature="(L.*AnotherUnknownType;)L.*UnknownType;">

Additionally, update `JavaType.java` to cause it to have "orphaned"
Javadoc comments; this change was forgotten in c1bc5c8.

[0]: #687 (comment)
jonpryor added a commit to jonpryor/java.interop that referenced this pull request Dec 10, 2020
Context: dotnet#687 (comment)

Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

	# API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
	# Skip the `.annotated.java` files
	$ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
		| grep -v '\.annotated\.' > sources.txt
	$ java -jar java-source-utils.jar -v \
		--source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
		--output-javadoc android-javadoc.xml \
		@sources.txt

What can we *do* with the generated `android-javadoc.xml`?

`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.

What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly.  This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!

Add a new `generator --with-javadoc-xml=FILE` option.  When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:

 1. A URL to the corresponding online Javadoc HTML documentation.
 2. A copyright notice disclaimer.

Allow provision of this information by updating
`java-source-utils.jar` to support the new options:

	--doc-copyright FILE   Copyright information for Javadoc.  Should be in
	                         mdoc(5) XML, to be held within <remarks/>.
	                         Stored in //javadoc-metadata/copyright.
	--doc-url-prefix URL   Base URL for links to documentation.
	                         Stored in //javadoc-metadata/link/@Prefix.
	--doc-url-style STYLE  STYLE of URLs to generate for member links.
	                         Stored in //javadoc-metadata/link/@Style.
	                         Supported styles include:
	                         - developer.android.com/reference@2020-Nov

The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.

For example, given:

	java -jar java-source-utils.jar \
		--doc-url-prefix https://developer.android.com/reference \
		--doc-url-style developer.android.com/reference@2020-Nov

Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:

	/// <format type="text/html">
	///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
	/// </format>

The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

  * <https://gist.github.com/jonpryor/5802ec3b799f36791d8065a985d1d1d8>

TODO:

  * `SourceJavadocToXmldocParser` doesn't support many constructs.

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc
@jonpryor
Copy link
Member Author

jonpryor commented Dec 10, 2020

Current Java.Lang.Object.cs output, courtesy dotnet/android#5253: https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e

Additionally, this is after the integration/rebasing atop #757 and #758, which should address previous java-source-utils.jar issues mentioned in this PR.

As such:

  • The Object.Class property is not only documented, but contains the correct documentation this time! (Compare to last time.)
  • Object.Clone() has documentation! As do Wait(long) and Wait(long, int)!

Also, the build log is absolutely spammed with # Unsupported HTML element messages, containing 159454 of them.

I'm not convinced that should be fixed as part of this PR; it falls under the "THIS WILL BE INCOMPLETE" "warning" in the PR message.

jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Dec 11, 2020
Fixes: dotnet#4789

Context: dotnet/java-interop#687
Context: dotnet#5200

What do we want?  Updated documentation!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`].
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus our documentation is
woefully out of date.

We could have scraped developer.android.com/reference within that
time frame, but web scraping is annoying, and we'd still have to deal
with the pain of parsing HTML.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained an `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to install the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `android-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc>
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

Finally, commit 380e95e, which added support for JDK 11, *broke*
support for `@(JavaSourceJar)` when running under JDK 11, as JDK 11's
Javadoc HTML output differs from what we expect.  With the new
`java-source-utils.jar` and
`generator --with-javadoc-xml=FILE` infrastructure in place, we can
reasonably address this breakage under JDK 11, fixing Issue dotnet#4789:

Update `Xamarin.Android.Bindings.Core.targets` so that before
invoking `generator`, we first run `java-source-utils.jar` on the
`@(JavaSourceJar)` files, producing Javadoc XML files.
The `<BindingsGenerator/>` task in turn is updated to accept a new
`BindingsGenerator.JavadocXml` property, which is converted into a
`generator --with-javadoc-xml=FILE` option.

The `@(JavaSourceJar)` item group is "extended" to support the
following item metadata:

  * `%(CopyrightFile)`: A path to a file that contains copyright
    information for the Javadoc contents, which will be appended to
	all imported documentation.

  * `%(UrlPrefix)`: A URL prefix to support linking to online
    documentation within imported documentation.

  * `%(UrlStyle)`: The "style" of URLs to generate when linking to
    online documentation.  Only one style is currently supported:
    `developer.android.com/reference@2020-Nov`.

For .NET 6 ("One .NET") integration purposes, provide a default item
group of `@(JavaSourceJar)` which includes `**\*-source?.jar` and
`**\*-src.jar`.  This will allow `dotnet build` of an "Android Java
Library Binding" project (`dotnet new android-bindinglib`) to
automatically process `.java` source code for documentation
translation purposes.

Add `Java.Interop.Tools.Generator.dll` to the
`Microsoft.Android.Sdk*.nupkg` file, as it is an assembly dependency
of `generator.dll`.  (Not sure why this didn't previously fail…)

Finally, add a new `$(_UseLegacyJavadocImport)` MSBuild property
which *disables* use of `java-source-utils.jar` for Javadoc importing,
and instead use the previous, legacy, doesn't work on JDK 11,
approach of using `javadoc` and HTML parsing.  This shouldn't be
needed, but just in case it *is* needed…

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Dec 11, 2020
Fixes: dotnet#4789

Context: dotnet/java-interop#687
Context: dotnet#5200

What do we want?  Updated documentation!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via [`tools/javadoc2mdoc`].
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus our documentation is
woefully out of date.

We could have scraped developer.android.com/reference within that
time frame, but web scraping is annoying, and we'd still have to deal
with the pain of parsing HTML.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained an `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to install the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `android-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc>
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

Finally, commit 380e95e, which added support for JDK 11, *broke*
support for `@(JavaSourceJar)` when running under JDK 11, as JDK 11's
Javadoc HTML output differs from what we expect.  With the new
`java-source-utils.jar` and
`generator --with-javadoc-xml=FILE` infrastructure in place, we can
reasonably address this breakage under JDK 11, fixing Issue dotnet#4789:

Update `Xamarin.Android.Bindings.Core.targets` so that before
invoking `generator`, we first run `java-source-utils.jar` on the
`@(JavaSourceJar)` files, producing Javadoc XML files.
The `<BindingsGenerator/>` task in turn is updated to accept a new
`BindingsGenerator.JavadocXml` property, which is converted into a
`generator --with-javadoc-xml=FILE` option.

The `@(JavaSourceJar)` item group is "extended" to support the
following item metadata:

  * `%(CopyrightFile)`: A path to a file that contains copyright
    information for the Javadoc contents, which will be appended to
	all imported documentation.

  * `%(UrlPrefix)`: A URL prefix to support linking to online
    documentation within imported documentation.

  * `%(UrlStyle)`: The "style" of URLs to generate when linking to
    online documentation.  Only one style is currently supported:
    `developer.android.com/reference@2020-Nov`.

For .NET 6 ("One .NET") integration purposes, provide a default item
group of `@(JavaSourceJar)` which includes `**\*-source?.jar` and
`**\*-src.jar`.  This will allow `dotnet build` of an "Android Java
Library Binding" project (`dotnet new android-bindinglib`) to
automatically process `.java` source code for documentation
translation purposes.

Add `Java.Interop.Tools.Generator.dll` to the
`Microsoft.Android.Sdk*.nupkg` file, as it is an assembly dependency
of `generator.dll`.  (Not sure why this didn't previously fail…)

Finally, add a new `$(_UseLegacyJavadocImport)` MSBuild property
which *disables* use of `java-source-utils.jar` for Javadoc importing,
and instead use the previous, legacy, doesn't work on JDK 11,
approach of using `javadoc` and HTML parsing.  This shouldn't be
needed, but just in case it *is* needed…

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
@jpobst
Copy link
Contributor

jpobst commented Dec 11, 2020

I think it may be advantageous to allow users to specify their intended usage of XML documentation. That is, there are 2 purpose for XML docs I am aware of:

  • Intellisense
  • Using the generated XML file as an input to an HTML docs generator

I suspect the first one is the much more common scenario, and it supports a lot fewer XML elements, namely <summary>, <param>, and <return>. By ignoring <remarks> and others, we may be able to significantly cut down on the run time and resources usage required to run this process. Especially if your use-case is to ship the XML with your library to provide Intellisense, the file is likely to be an order of magnitude larger than needed.

I would suggest we run just the Intellisense version by default, and add a flag allowing the user to opt-in to full documentation.

jonpryor added a commit to jonpryor/java.interop that referenced this pull request Dec 16, 2020
Context: dotnet#687 (comment)
Context: dotnet/android#5253 (comment)

"Full Javadoc Import" is slow, causing CI `generator` invocation to
take ~12x longer, and `CoreCompile` to take ~7x longer.

What happens if we instead do a "Summary Javadoc Import", and only
emit `<summary/>`, `<param/>`, `<returns/>`, `<exception/>`?

Add a `generator --doc-comment-style=summary,remarks` option, so that
we can emit only "summary" docs instead of *all* docs, and see what
the build time changes are.
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Dec 17, 2020
Context: dotnet#5253 (comment)
Context: dotnet/java-interop#687 (comment)
Context: dotnet/java-interop@a65b8ab

It looks like only emitting `<summary/>`, `<param/>`, `<returns/>`,
and `<exception/>` is *much* faster than trying to do full
`<remarks/>`.

Update the `generator.exe` invocation to use
`generator --doc-comment-style=summary`, which will hopefully
speed things up.
jonpryor added a commit to jonpryor/java.interop that referenced this pull request Dec 17, 2020
Context: dotnet#687 (comment)

Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

	# API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
	# Skip the `.annotated.java` files
	$ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
		| grep -v '\.annotated\.' > sources.txt
	$ java -jar java-source-utils.jar -v \
		--source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
		--output-javadoc android-javadoc.xml \
		@sources.txt

What can we *do* with the generated `android-javadoc.xml`?

`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.

What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly.  This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!

Add a new `generator --with-javadoc-xml=FILE` option.  When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:

 1. A URL to the corresponding online Javadoc HTML documentation.
 2. A copyright notice disclaimer.

Allow provision of this information by updating
`java-source-utils.jar` to support the new options:

	--doc-copyright FILE   Copyright information for Javadoc.  Should be in
	                         mdoc(5) XML, to be held within <remarks/>.
	                         Stored in //javadoc-metadata/copyright.
	--doc-url-prefix URL   Base URL for links to documentation.
	                         Stored in //javadoc-metadata/link/@Prefix.
	--doc-url-style STYLE  STYLE of URLs to generate for member links.
	                         Stored in //javadoc-metadata/link/@Style.
	                         Supported styles include:
	                         - developer.android.com/reference@2020-Nov

The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.

For example, given:

	java -jar java-source-utils.jar \
		--doc-url-prefix https://developer.android.com/reference \
		--doc-url-style developer.android.com/reference@2020-Nov

Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:

	/// <format type="text/html">
	///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
	/// </format>

The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

  * <https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e>

Unfortunately, converting Javadoc to C# XML Documentation Comments is
not a "zero cost" operation.  Add a new
`generator --doc-comment-verbosity=STYLE` option to control how
"complete" the generated documentation comments are:

	--doc-comment-verbosity=STYLE
	                       STYLE of C# documentation comments to emit.
	                         Defaults to `full`.  STYLE may be:
	                           * `intellisense`: emit <summary>, <param>,
	                             <returns>, <exception>.
	                           * `full`: plus <remarks>, <altmember>, ...

Using `--doc-comment-verbosity=full` will *most* impact build times.

TODO:

  * `SourceJavadocToXmldocParser` doesn't support many constructs.

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc
jonpryor added a commit to jonpryor/java.interop that referenced this pull request Dec 17, 2020
Context: dotnet#687 (comment)

Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

	# API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
	# Skip the `.annotated.java` files
	$ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
		| grep -v '\.annotated\.' > sources.txt
	$ java -jar java-source-utils.jar -v \
		--source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
		--output-javadoc android-javadoc.xml \
		@sources.txt

What can we *do* with the generated `android-javadoc.xml`?

`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.

What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly.  This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!

Add a new `generator --with-javadoc-xml=FILE` option.  When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:

 1. A URL to the corresponding online Javadoc HTML documentation.
 2. A copyright notice disclaimer.

Allow provision of this information by updating
`java-source-utils.jar` to support the new options:

	--doc-copyright FILE   Copyright information for Javadoc.  Should be in
	                         mdoc(5) XML, to be held within <remarks/>.
	                         Stored in //javadoc-metadata/copyright.
	--doc-url-prefix URL   Base URL for links to documentation.
	                         Stored in //javadoc-metadata/link/@Prefix.
	--doc-url-style STYLE  STYLE of URLs to generate for member links.
	                         Stored in //javadoc-metadata/link/@Style.
	                         Supported styles include:
	                         - developer.android.com/reference@2020-Nov

The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.

For example, given:

	java -jar java-source-utils.jar \
		--doc-url-prefix https://developer.android.com/reference \
		--doc-url-style developer.android.com/reference@2020-Nov

Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:

	/// <format type="text/html">
	///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
	/// </format>

The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

  * <https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e>

Unfortunately, converting Javadoc to C# XML Documentation Comments is
not a "zero cost" operation.  Add a new
`generator --doc-comment-verbosity=STYLE` option to control how
"complete" the generated documentation comments are:

	--doc-comment-verbosity=STYLE
	                       STYLE of C# documentation comments to emit.
	                         Defaults to `full`.  STYLE may be:
	                           * `intellisense`: emit <summary>, <param>,
	                             <returns>, <exception>.
	                           * `full`: plus <remarks>, <seealso>, ...

Using `--doc-comment-verbosity=full` will *most* impact build times.

TODO:

  * `SourceJavadocToXmldocParser` doesn't support many constructs.

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Dec 17, 2020
Context: dotnet#5253 (comment)
Context: dotnet/java-interop#687 (comment)
Context: dotnet/java-interop@a65b8ab

It looks like only emitting `<summary/>`, `<param/>`, `<returns/>`,
and `<exception/>` is *much* faster than trying to do full
`<remarks/>`.

Update the `generator.exe` invocation to use
`generator --doc-comment-style=summary`, which will hopefully
speed things up.
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Dec 17, 2020
Context: dotnet#5253 (comment)
Context: dotnet/java-interop#687 (comment)
Context: dotnet/java-interop@a65b8ab

It looks like only emitting `<summary/>`, `<param/>`, `<returns/>`,
and `<exception/>` is *much* faster than trying to do full
`<remarks/>`.

Update the `generator.exe` invocation to use
`generator --doc-comment-style=summary`, which will hopefully
speed things up.
@jonpryor jonpryor marked this pull request as ready for review December 17, 2020 19:17
Context: dotnet#687 (comment)

Commit 69e1b80 added `tools/java-source-utils`, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

	# API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
	# Skip the `.annotated.java` files
	$ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
		| grep -v '\.annotated\.' > sources.txt
	$ java -jar java-source-utils.jar -v \
		--source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
		--output-javadoc android-javadoc.xml \
		@sources.txt

What can we *do* with the generated `android-javadoc.xml`?

`android-javadoc.xml` contains parameter names, and thus can be used
with `class-parse --docspath`; see commit 806082f.

What we *really* want to do is make the Javadoc information *useful*
to consumers of the binding assembly.  This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit [C# XML Documentation Comments][0] into the binding source code!

Add a new `generator --with-javadoc-xml=FILE` option.  When specified,
`FILE` will be treated as an XML file containing the output from
`java-source-utils.jar --output-javadoc` (69e1b80), and all
`<javadoc/>` elements within the XML file will be associated with C#
types and members to emit, based on the `//@jni-signature` and
`//@name` attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to `Java.Interop.Tools.JavaSource.dll`,
in the new `Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser`
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we *also* want to provide "upstream
information" in the form of:

 1. A URL to the corresponding online Javadoc HTML documentation.
 2. A copyright notice disclaimer.

Allow provision of this information by updating
`java-source-utils.jar` to support the new options:

	--doc-copyright FILE   Copyright information for Javadoc.  Should be in
	                         mdoc(5) XML, to be held within <remarks/>.
	                         Stored in //javadoc-metadata/copyright.
	--doc-url-prefix URL   Base URL for links to documentation.
	                         Stored in //javadoc-metadata/link/@Prefix.
	--doc-url-style STYLE  STYLE of URLs to generate for member links.
	                         Stored in //javadoc-metadata/link/@Style.
	                         Supported styles include:
	                         - developer.android.com/reference@2020-Nov

The new `/api/javadoc-metadata/link@prefix` and
`/api/javadoc-metadata/link@style` XML attributes, stored within
`java-source-utils.jar --output-javadoc` XML output, allow
construction of a URL to the Java member.

For example, given:

	java -jar java-source-utils.jar \
		--doc-url-prefix https://developer.android.com/reference \
		--doc-url-style developer.android.com/reference@2020-Nov

Then `generator` can emit the C# documentation comment for
`java.lang.Object.equals(Object)`:

	/// <format type="text/html">
	///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
	/// </format>

The copyright notice disclaimer is supported by
`java-source-utils.jar --doc-copyright FILE`; the contents of `FILE`
are inserted into the `/api/javadoc-metadata/copyright` element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

  * <https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e>

Unfortunately, converting Javadoc to C# XML Documentation Comments is
not a "zero cost" operation.  Add a new
`generator --doc-comment-verbosity=STYLE` option to control how
"complete" the generated documentation comments are:

	--doc-comment-verbosity=STYLE
	                       STYLE of C# documentation comments to emit.
	                         Defaults to `full`.  STYLE may be:
	                           * `intellisense`: emit <summary>, <param>,
	                             <returns>, <exception>.
	                           * `full`: plus <remarks>, <seealso>, ...

Using `--doc-comment-verbosity=full` will *most* impact build times.

TODO:

  * `SourceJavadocToXmldocParser` doesn't support many constructs.

[0]: https://docs.microsoft.com/en-us/dotnet/csharp/codedoc
@jonpryor jonpryor merged commit 7574f16 into dotnet:master Jan 4, 2021
jonpryor added a commit to jonpryor/xamarin-android that referenced this pull request Jan 4, 2021
Changes: dotnet/java-interop@2f62ffd...7574f16

  * dotnet/java-interop@7574f166: [generator] Add `generator --with-javadoc-xml=FILE` support. (dotnet#687)
  * dotnet/java-interop@7d197f17: Refactor Crc64 type (dotnet#769)
  * dotnet/java-interop@876442f4: [generator] Fix MSBuild warning/error format for Visual Studio (dotnet#765)
  * dotnet/java-interop@3f6cf72b: [.Localization, .Cecil, .Diagnostics, .Generator] $(Nullable)=enable (dotnet#746)

dotnet/java-interop#687 was merged.
@jpobst jpobst added this to the 11.2 (16.10 / 8.10) milestone Jan 5, 2021
jonpryor added a commit to dotnet/android that referenced this pull request Jan 6, 2021
…5253)

Fixes: #4789

Context: dotnet/java-interop@7574f16
Context: #5200

What do we want?  Updated documentation!  Better Bindings!

How do we get that?  Uh…

Historically, [Xamarin.Android API docs][0] were produced by parsing
Android's HTML documentation from `docs-24_r01.zip` and converting it
into [**mdoc**(5) documentation][1] via `tools/javadoc2mdoc`.
The problem is that Google hasn't released an updated `docs*.zip`
package since API-24 (~6 years ago), and thus our documentation is
woefully out of date.

We could have scraped developer.android.com/reference within that
time frame, but web scraping is annoying, and we'd still have to deal
with the pain of parsing HTML.

There is an alternative, though: with API-30, there is a new `sources`
package which contains the Java source code used for the API level,
which in turn contains Javadoc source code comments.  Previous API
levels similarly contained an `android-stubs-src.jar` file which
likewise contained Java source code containing Javadoc comments.

The new approach is to:

 1. Update `build-tools/xaprepare` to install the `sources` package.

 2. Use [`java-source-tool.jar`][2] to parse the Java source code,
    creating an `android-javadoc.xml` file.

 3. [Update `generator`][3] to consume `android-javadoc.xml` and
    convert the Javadocs into [C# XML documentation comments][4].

 4. Update `src/Mono.Android` so that `generator` will emit
    C# XML doc comments, and then produce a `Mono.Android.xml` file.

The result is a `Mono.Android.xml` file which contains imported
Android Javadoc documentation, e.g.

	<doc>
	  <assembly><name>Mono.Android</name></assembly>
	  <members>
	    <member name="T:Java.Lang.Object">
	      <summary>Class <c>Object</c> is the root of the class hierarchy.</summary>
	      <remarks>
	        <para>Class <c>Object</c> is the root of the class hierarchy.
	…

"Later", `Mono.Android.xml` can then be used alongside
[`mdoc update --import=Mono.Android.xml`][5] to update our published
API documentation.

Generation of `Mono.Android.xml` is disabled by default on CI PR
builds, as generating `Mono.Android.xml` increases `src/Mono.Android`
build times by an unacceptable amount.

With the inclusion of `java-source-tools.jar`, we can now fix the
TODO mentioned in commit 380e95e, and re-add support for
`@(JavaSourceJar)` when running under JDK 11.

Update `Xamarin.Android.Bindings.Core.targets` so that before
invoking `generator`, we first run `java-source-utils.jar` on the
`@(JavaSourceJar)` files, producing Javadoc XML files.
The `<BindingsGenerator/>` task in turn is updated to accept a new
`BindingsGenerator.JavadocXml` property, which is converted into a
`generator --with-javadoc-xml=FILE` option.

The `@(JavaSourceJar)` item group is "extended" to support the
following item metadata:

  * `%(CopyrightFile)`: A path to a file that contains copyright
    information for the Javadoc contents, which will be appended to
	all imported documentation.

  * `%(UrlPrefix)`: A URL prefix to support linking to online
    documentation within imported documentation.

  * `%(UrlStyle)`: The "style" of URLs to generate when linking to
    online documentation.  Only one style is currently supported:
    `developer.android.com/reference@2020-Nov`.

For .NET 6 ("One .NET") integration purposes, provide a default item
group of `@(JavaSourceJar)` which includes `**\*-source*.jar` and
`**\*-src.jar`.  This will allow `dotnet build` of an "Android Java
Library Binding" project (`dotnet new android-bindinglib`) to
automatically process `.java` source code for documentation
translation purposes.

Add a new `$(AndroidJavadocVerbosity)` MSBuild property, which
controls "how much" of the Javadoc comments are converted into C#
XML documentation.  Supported values are:

  * `full`: Convert as much Javadoc as possible.
  * `intellisense`: Only emit XML documentation for IDE-useful
    constructs: summary, parameters, returns, exceptions.

The difference between the two is *build time* impact: `full` takes
longer, and thus may not always be desirable.

Finally, add a new `$(_UseLegacyJavadocImport)` MSBuild property
which *disables* use of `java-source-utils.jar` for Javadoc importing,
and instead use the previous, legacy, doesn't work on JDK 11,
approach of using `javadoc` and HTML parsing.  This shouldn't be
needed, but just in case it *is* needed…

[0]: https://github.com/xamarin/android-api-docs
[1]: http://docs.go-mono.com/?link=man%3amdoc(5)
[2]: dotnet/java-interop@69e1b80
[3]: dotnet/java-interop#687
[4]: https://docs.microsoft.com/dotnet/csharp/codedoc
[5]: http://docs.go-mono.com/?link=man%3amdoc-update(1)
@github-actions github-actions bot locked and limited conversation to collaborators Apr 13, 2024
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

JavaDoc Integration?
2 participants