[generator] Add --with-javadoc-xml=FILE support.

[jonpryor]

Fixes: #642

Context: dotnet/android#4789
Context: dotnet/android#5200

Commit 69e1b80 added tools/java-source-utils, which along with
b588ef5, can parse Java source code and extract Javadoc comments
from Android API-30 sources into an XML file:
into an XML file:

    # API-30 `sources` package contains both `Object.java` and `Object.annotated.java`;
    # Skip the `.annotated.java` files
    $ find $HOME/android-toolchain/sdk/platforms/android-30/src/{android,java,javax,org} -iname \*.java \
            | grep -v '\.annotated\.' > sources.txt
    $ java -jar java-source-utils.jar -v \
            --source "$HOME/android-toolchain/sdk/platforms/android-30/src" \
            --output-javadoc android-javadoc.xml \
            @sources.txt

What can we do with the generated android-javadoc.xml?

android-javadoc.xml contains parameter names, and thus can be used
with class-parse --docspath; see commit 806082f.

What we really want to do is make the Javadoc information useful
to consumers of the binding assembly. This means that we want a
C# XML Documentation file for the binding assembly.

The most straightforward way to get a C# XML Documentation File is to
emit C# XML Documentation Comments into the binding source code!

Add a new generator --with-javadoc-xml=FILE option. When specified,
FILE will be treated as an XML file containing the output from
java-source-utils.jar --output-javadoc (69e1b80), and all
<javadoc/> elements within the XML file will be associated with C#
types and members to emit, based on the //@jni-signature and
//@name attributes, as appropriate.

When the bindings are written to disk, the Javadoc comments will be
translated to C# XML Documentation comments, in a "best effort" basis.
(THIS WILL BE INCOMPLETE.)

To perform the Javadoc-to-C# XML Documentation comments conversion,
add a new Irony-based grammar to Java.Interop.Tools.JavaSource.dll,
in the new Java.Interop.Tools.JavaSource.SourceJavadocToXmldocParser
type, which parses the Javadoc content and translates to XML.

In addition to transforming the Javadoc comments into C# XML
documentation comments, we also want to provide "upstream
information" in the form of:

1. A URL to the corresponding online Javadoc HTML documentation.
2. A copyright notice disclaimer.

Allow provision of this information by updating
java-source-utils.jar to support the new options:

    --doc-copyright FILE   Copyright information for Javadoc.  Should be in
                             mdoc(5) XML, to be held within <remarks/>.
                             Stored in //javadoc-metadata/copyright.
    --doc-url-prefix URL   Base URL for links to documentation.
                             Stored in //javadoc-metadata/link/@prefix.
    --doc-url-style STYLE  STYLE of URLs to generate for member links.
                             Stored in //javadoc-metadata/link/@style.
                             Supported styles include:
                             - developer.android.com/reference@2020-Nov

The new /api/javadoc-metadata/link@prefix and
/api/javadoc-metadata/link@style XML attributes, stored within
java-source-utils.jar --output-javadoc XML output, allow
construction of a URL to the Java member.

For example, given:

    java -jar java-source-utils.jar \
            --doc-url-prefix https://developer.android.com/reference \
            --doc-url-style developer.android.com/reference@2020-Nov

Then generator can emit the C# documentation comment for
java.lang.Object.equals(Object):

    /// <format type="text/html">
    ///   <a href="https://developer.android.com/reference/java/lang/Object#equals(java.lang.Object)">Java documentation for <tt>java.lang.Object.equals(java.lang.Object)</tt>.</a>
    /// </format>

The copyright notice disclaimer is supported by
java-source-utils.jar --doc-copyright FILE; the contents of FILE
are inserted into the /api/javadoc-metadata/copyright element, and
will be copied into the output of every C# XML documentation block.

Example output is at:

• https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e

Unfortunately, converting Javadoc to C# XML Documentation Comments is
not a "zero cost" operation. Add a new
generator --doc-comment-verbosity=STYLE option to control how
"complete" the generated documentation comments are:

    --doc-comment-verbosity=STYLE
                           STYLE of C# documentation comments to emit.
                             Defaults to `full`.  STYLE may be:
                               * `intellisense`: emit <summary>, <param>,
                                 <returns>, <exception>.
                               * `full`: plus <remarks>, <altmember>, ...

Using --doc-comment-verbosity=full will most impact build times.

TODO:

• SourceJavadocToXmldocParser doesn't support many constructs.

[jonpryor]

Example generated output: https://gist.github.com/jonpryor/027ca5f1ce9b20ca0a8f3ecf4ebafb4b

[jonpryor]

TODO:

• Emit docs for properties.
• Double-check constructors
• Unit tests!
• use regular expressions for javadoc matching, as per 69e1b80:

  In some scenarios, types won't be resolvable. What should output be?
  …
  docs.xml will output L.*UnresolvableType;.

[jonpryor]

What does an "unresolved type" really look like? Example:

$ java -jar bin/Debug/java-source-utils.jar ~/android-toolchain/sdk/platforms/_t/android/app/Activity.java -D a.xml

In this case, ~/android-toolchain/sdk/platforms/_t contains the extracted contents of platforms/android-29/android-stubs-src.jar.

java-source-util.jar -s ~/android-toolchain/sdk/platforms/_t*wasn't* used, so *all* types referenced byActivity.java` can't be resolved. The result?

<api api-source="java-source-utils">
  <package jni-name="android/app" name="android.app">
    <class jni-signature="Landroid/app/Activity;" name="Activity">
      <javadoc>
…

The /api/package/class/@jni-signature value is correct, as we know the package from the source code.

      <method jni-return="V" jni-signature="(.*android.os.CancellationSignal.*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;)V" name="onGetDirectActions" return="void">
        <parameter jni-type=".*android.os.CancellationSignal" name="cancellationSignal" type=".*android.os.CancellationSignal"/>
        <parameter jni-type=".*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;" name="callback" type=".*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;"/>
        <javadoc>
          <![CDATA[Returns the list of direct actions supported by the app.
…

<method/> (and potentially <field/>, if the field is for a reference type) is where the "unresolved type" scenario comes in.

Additionally -- and unexpectedly, as I hadn't thought to test this condition (😞), is that generic type parameters are present in the signature. These need to be removed in java-source-util.jar.

Additionally, the behavior doesn't match the commit message docs (double 😭), as it lacks the leading L and trailing ;. More fixes needed to java-source-util.jar.

The <method/> thus should be:

      <method jni-return="V" jni-signature="(L.*android.os.CancellationSignal;L.*java.util.function.Consumer;)V" name="onGetDirectActions" return="void">
        <parameter jni-type="L.*android.os.CancellationSignal;" name="cancellationSignal" type=".*android.os.CancellationSignal"/>
        <parameter jni-type="L.*java.util.function.Consume;" name="callback" type=".*java.util.function.Consumer&lt;java.util.List&lt;android.app.DirectAction&gt;&gt;"/>
        <javadoc>
          <![CDATA[Returns the list of direct actions supported by the app.
…

[jonpryor]

SourceJavadoc

[jonpryor]

Current output: https://gist.github.com/jonpryor/7cd9ccd152a918887f54d327ff6a2740

[jonpryor]

Current Java.Lang.Object output: https://gist.github.com/jonpryor/b89999ad4f261eb561b27af921aa1cc7

Note that every member except getClass() contains Javadoc comments, as compared to previous iterations (which lacked Javadocs for e.g. Wait(), Notify()).

Properties aren't yet handled, so the lack of getClass() is expected.

[jonpryor]

Current Java.Lang.Object output: https://gist.github.com/jonpryor/ee60a7385e908458bbe9ad48e30b2ecc

Notable differences from last time are:

1. Inclusion of a Java documentation for… link

    	///       <a href="https://developer.android.com/reference/java/lang/Object">Java documentation for <tt>java.lang.Object</tt>.</a>
   

2. A copyright notice/blurb for all imported documentation

3. Missing member documentation for Clone(), Wait(long), and Wait(long, int). The likely cause for this is that previous runs imported the API-29 Java "stubs", while the current iteration imports the API-30 "sources" package, which java-source-utils doesn't "like".

In particular for (3), having comments between the Javadocs and the Java code appears to screw things up:

    /**
     * Creates and returns a copy of this object.  The precise meaning
     * …
     */
    // BEGIN Android-changed: Use native local helper for clone()
    // Checks whether cloning is allowed before calling native local helper.
    // protected native Object clone() throws CloneNotSupportedException;
    protected Object clone() throws CloneNotSupportedException {
…

[jonpryor]

Current Java.Lang.Object.cs output, courtesy dotnet/android#5253: https://gist.github.com/jonpryor/004f01f4cd5ff32299ff590ba7a2fe0e

Additionally, this is after the integration/rebasing atop #757 and #758, which should address previous java-source-utils.jar issues mentioned in this PR.

As such:

• The Object.Class property is not only documented, but contains the correct documentation this time! (Compare to last time.)
• Object.Clone() has documentation! As do Wait(long) and Wait(long, int)!

Also, the build log is absolutely spammed with # Unsupported HTML element messages, containing 159454 of them.

I'm not convinced that should be fixed as part of this PR; it falls under the "THIS WILL BE INCOMPLETE" "warning" in the PR message.

[jpobst]

I think it may be advantageous to allow users to specify their intended usage of XML documentation. That is, there are 2 purpose for XML docs I am aware of:

• Intellisense
• Using the generated XML file as an input to an HTML docs generator

I suspect the first one is the much more common scenario, and it supports a lot fewer XML elements, namely <summary>, <param>, and <return>. By ignoring <remarks> and others, we may be able to significantly cut down on the run time and resources usage required to run this process. Especially if your use-case is to ship the XML with your library to provide Intellisense, the file is likely to be an order of magnitude larger than needed.

I would suggest we run just the Intellisense version by default, and add a flag allowing the user to opt-in to full documentation.