Website & API Doc site generator using DocFx script

.gitignore

@@ -51,6 +51,7 @@ release/
 # NUnit test result file produced by nunit3-console.exe
 [Tt]est[Rr]esult.xml
 apidocs/_site/*
-apidocs/api/*
+apidocs/api/*.yml
 apidocs/tools/docfx.zip
 apidocs/tools/docfx/*
+apidocs/api/.manifest

apidocs/api/index.md

@@ -1,13 +1,13 @@
-<p>Apache Lucene is a high-performance, full-featured text search engine library.
+<!-- TODO: Create the packages table of contents ... manually? Post processor?  -->
+
+
+Apache Lucene is a high-performance, full-featured text search engine library.
 Here's a simple example how to use Lucene for indexing and searching (using JUnit
-to check if the results are what we expect):</p>
-
-<!-- code comes from org.apache.lucene.TestDemo: -->
-<!-- ======================================================== -->
-<!-- = Java Sourcecode to HTML automatically converted code = -->
-<!-- =   Java2Html Converter 5.0 [2006-03-04] by Markus Gebhard  markus@jave.de   = -->
-<!-- =     Further information: http://www.java2html.de     = -->
-<pre class="prettyprint">
+to check if the results are what we expect):
+
+
+<!-- TODO: Convert this to .NET  -->
+```csharp
     Analyzer analyzer = new StandardAnalyzer(Version.LUCENE_CURRENT);
 
     // Store the index in memory:
@@ -21,7 +21,7 @@ to check if the results are what we expect):</p>
     doc.add(new Field("fieldname", text, TextField.TYPE_STORED));
     iwriter.addDocument(doc);
     iwriter.close();
-    
+
     // Now search the index:
     DirectoryReader ireader = DirectoryReader.open(directory);
     IndexSearcher isearcher = new IndexSearcher(ireader);
@@ -32,17 +32,16 @@ to check if the results are what we expect):</p>
     assertEquals(1, hits.length);
     // Iterate through the results:
     for (int i = 0; i < hits.length; i++) {
-      Document hitDoc = isearcher.doc(hits[i].doc);
-      assertEquals("This is the text to be indexed.", hitDoc.get("fieldname"));
+        Document hitDoc = isearcher.doc(hits[i].doc);
+        assertEquals("This is the text to be indexed.", hitDoc.get("fieldname"));
     }
     ireader.close();
-    directory.close();</pre>
-<!-- =       END of automatically generated HTML code       = -->
-<!-- ======================================================== -->
-
+    directory.close();
+```
 
+The Lucene API is divided into several packages:
 
-<p>The Lucene API is divided into several packages:</p>
+<!-- TODO: Fix links  -->
 
 <ul>
 <li>
@@ -129,6 +128,9 @@ index for all the files contained in a directory.</li>
 &nbsp;<a href="../demo/src-html/org/apache/lucene/demo/SearchFiles.html">SearchFiles.java</a> prompts for
 queries and searches an index.</li>
 </ul>
+
+<!-- TODO: Fix this  -->
+
 To demonstrate these, try something like:
 <blockquote><tt>> <b>java -cp lucene-core.jar:lucene-demo.jar:lucene-analyzers-common.jar org.apache.lucene.demo.IndexFiles -index index -docs rec.food.recipes/soups</b></tt>
 <br><tt>adding rec.food.recipes/soups/abalone-chowder</tt>

apidocs/docfx.json

@@ -29,8 +29,8 @@
       },
       {
         "files": [
-          "articles/**.md",
-          "articles/**/toc.yml",
+          "docs/**.md",
+          "docs/**/toc.yml",
           "toc.yml",
           "*.md"
         ],
@@ -63,7 +63,7 @@
       }
     ],
 	"globalMetadata": {
-      "_appTitle": "Lucene.Net API docs",
+      "_appTitle": "Apache Lucene.NET 4.8.0 Documentation",
       "_enableSearch": false,
       "_disableContribution": false
     },

apidocs/docs.ps1

@@ -39,7 +39,7 @@ $FileExists = Test-Path $DocFxExe
 If ($FileExists -eq $False) {
 	Write-Host "Retrieving docfx..."
 	$DocFxZip = "$ToolsFolder\docfx.zip"
-	$SourceDocFx = "https://github.com/dotnet/docfx/releases/download/v2.17.4/docfx.zip"
+	$SourceDocFx = "https://github.com/dotnet/docfx/releases/download/v2.17.7/docfx.zip"
 	Invoke-WebRequest $SourceDocFx -OutFile $DocFxZip
 	#unzip
 	Expand-Archive $DocFxZip -DestinationPath (Join-Path -Path $ToolsFolder -ChildPath "docfx")
@@ -49,10 +49,17 @@ If ($FileExists -eq $False) {
 if ($Clean -eq 1) {
 	Remove-Item (Join-Path -Path $ApiDocsFolder "obj\*") -recurse -force -ErrorAction SilentlyContinue
 	Remove-Item (Join-Path -Path $ApiDocsFolder "obj") -force -ErrorAction SilentlyContinue
-	Remove-Item (Join-Path -Path $ApiDocsFolder "api\*") -recurse -force -ErrorAction SilentlyContinue
-	Remove-Item (Join-Path -Path $ApiDocsFolder "api") -force -ErrorAction SilentlyContinue
+	Remove-Item (Join-Path -Path $ApiDocsFolder "api\*") -exclude "*.md" -recurse -force -ErrorAction SilentlyContinue
+	##Remove-Item (Join-Path -Path $ApiDocsFolder "api") -force -ErrorAction SilentlyContinue
 }
 
+# NOTE: There's a ton of Lucene docs that we want to copy and re-format. I'm not sure if we can really automate this 
+# in a great way since the docs seem to be in many places, for example:
+# Home page - 	https://github.com/apache/lucene-solr/blob/branch_4x/lucene/site/xsl/index.xsl
+# Wiki docs - 	https://wiki.apache.org/lucene-java/FrontPage?action=show&redirect=FrontPageEN - not sure where the source is for this
+# Html pages - 	Example: https://github.com/apache/lucene-solr/blob/releases/lucene-solr/4.8.0/lucene/highlighter/src/java/org/apache/lucene/search/highlight/package.html - these seem to be throughout the source
+#				For these ones, could we go fetch them and download all *.html files from Git?
+
 $DocFxJson = Join-Path -Path $RepoRoot "apidocs\docfx.json"
 
 Write-Host "Building metadata..."

apidocs/docs/index.md

@@ -0,0 +1,17 @@
+Apache Lucene.NET 4.8.0 Documentation
+===============
+
+Apache Lucene.NET™ 4.8.0 Documentation
+---------------
+
+Lucene is a Java full-text search engine. Lucene.NET is not a complete application, but rather a code library and API that can easily be used to add search capabilities to applications.
+
+This is the official documentation for Apache Lucene.NET 4.8.0. Additional documentation is available in the Wiki.
+
+## Getting Started
+
+The following section is intended as a "getting started" guide. It has three audiences: first-time users looking to install Apache Lucene in their application; developers looking to modify or base the applications they develop on Lucene; and developers looking to become involved in and contribute to the development of Lucene. The goal is to help you "get started". It does not go into great depth on some of the conceptual or inner details of Lucene:
+
+* Lucene demo, its usage, and sources: Tutorial and walk-through of the command-line Lucene demo.
+* Introduction to Lucene's APIs: High-level summary of the different Lucene packages.
+* Analysis overview: Introduction to Lucene's analysis API. See also the TokenStream consumer workflow.

apidocs/docs/search/highlight/index.md

@@ -0,0 +1,74 @@
+
+The highlight package contains classes to provide "keyword in context" features
+typically used to highlight search terms in the text of results pages.
+The Highlighter class is the central component and can be used to extract the
+most interesting sections of a piece of text and highlight them, with the help of
+Fragmenter, fragment Scorer, and Formatter classes.
+
+<h2>Example Usage</h2>
+
+<pre class="prettyprint">
+  //... Above, create documents with two fields, one with term vectors (tv) and one without (notv)
+  IndexSearcher searcher = new IndexSearcher(directory);
+  QueryParser parser = new QueryParser("notv", analyzer);
+  Query query = parser.parse("million");
+
+  TopDocs hits = searcher.search(query, 10);
+
+  SimpleHTMLFormatter htmlFormatter = new SimpleHTMLFormatter();
+  Highlighter highlighter = new Highlighter(htmlFormatter, new QueryScorer(query));
+  for (int i = 0; i < 10; i++) {
+    int id = hits.scoreDocs[i].doc;
+    Document doc = searcher.doc(id);
+    String text = doc.get("notv");
+    TokenStream tokenStream = TokenSources.getAnyTokenStream(searcher.getIndexReader(), id, "notv", analyzer);
+    TextFragment[] frag = highlighter.getBestTextFragments(tokenStream, text, false, 10);//highlighter.getBestFragments(tokenStream, text, 3, "...");
+    for (int j = 0; j < frag.length; j++) {
+      if ((frag[j] != null) && (frag[j].getScore() > 0)) {
+        System.out.println((frag[j].toString()));
+      }
+    }
+    //Term vector
+    text = doc.get("tv");
+    tokenStream = TokenSources.getAnyTokenStream(searcher.getIndexReader(), hits.scoreDocs[i].doc, "tv", analyzer);
+    frag = highlighter.getBestTextFragments(tokenStream, text, false, 10);
+    for (int j = 0; j < frag.length; j++) {
+      if ((frag[j] != null) && (frag[j].getScore() > 0)) {
+        System.out.println((frag[j].toString()));
+      }
+    }
+    System.out.println("-------------");
+  }
+</pre>
+
+<h2>New features 06/02/2005</h2>
+
+This release adds options for encoding (thanks to Nicko Cadell).
+An "Encoder" implementation such as the new SimpleHTMLEncoder class can be passed to the highlighter to encode
+all those non-xhtml standard characters such as &amp; into legal values. This simple class may not suffice for
+some languages -  Commons Lang has an implementation that could be used: escapeHtml(String) in
+http://svn.apache.org/viewcvs.cgi/jakarta/commons/proper/lang/trunk/src/java/org/apache/commons/lang/StringEscapeUtils.java?rev=137958&view=markup
+
+<h2>New features 22/12/2004</h2>
+
+This release adds some new capabilities:
+<ol>
+	<li>Faster highlighting using Term vector support</li>
+	<li>New formatting options to use color intensity to show informational value</li>
+	<li>Options for better summarization by using term IDF scores to influence fragment selection</li>
+</ol>
+
+<p>
+The highlighter takes a TokenStream as input. Until now these streams have typically been produced
+using an Analyzer but the new class TokenSources provides helper methods for obtaining TokenStreams from
+the new TermVector position support (see latest CVS version).</p>
+
+<p>The new class GradientFormatter can use a scale of colors to highlight terms according to their score.
+A subtle use of color can help emphasise the reasons for matching (useful when doing "MoreLikeThis" queries and
+you want to see what the basis of the similarities are).</p>
+
+<p>The QueryScorer class has a new constructor which can use an IndexReader to derive the IDF (inverse document frequency)
+for each term in order to influence the score. This is useful for helping to extracting the most significant sections
+of a document and in supplying scores used by the new GradientFormatter to color significant words more strongly.
+The QueryScorer.getMaxWeight method is useful when passed to the GradientFormatter constructor to define the top score
+which is associated with the top color.</p>

apidocs/docs/search/index.md

@@ -0,0 +1,8 @@
+Support for testing search components. 
+<p>
+The primary classes are:
+<ul>
+  <li>{@link org.apache.lucene.search.QueryUtils}: Useful methods for testing Query classes.
+  <li>{@link org.apache.lucene.search.ShardSearchingTestBase}: Base class for simulating distributed search.
+</ul>
+</p>

apidocs/docs/search/toc.yml

@@ -0,0 +1,3 @@
+- name: Highlight
+  href: highlight/index.md
+  topicHref: highlight/index.md

apidocs/docs/toc.yml

@@ -0,0 +1,6 @@
+- name: Getting Started
+  href: index.md
+  topicHref: index.md
+- name: "Search"
+  href: search/toc.yml
+  topicHref: search/index.md

apidocs/index.md

@@ -1,11 +1,117 @@
-# Lucene.Net API reference
+Apache Lucene.NET™ 4.8.0 Documentation
+===============
 
-Welcome to the [Lucene.Net API documentation](api/Lucene.Net.html)
+---------------
+<div>
+<a href="https://lucenenet.apache.org/">
+    <img src="https://raw.githubusercontent.com/apache/lucenenet/master/branding/logo/lucene-net-icon-128.png" title="Apache.NET Lucene Logo" alt="Lucene.NET">
+</a>
+</div>
 
-## Quick links
+<p>Lucene is a .NET full-text search engine. Lucene.NET is not a complete application, 
+        but rather a code library and API that can easily be used to add search capabilities
+        to applications.</p>
+<p>
+This is the official documentation for <b>Apache Lucene.NET 4.8.0</b>. Additional documentation is available <a href="docs/index.html">here</a>.
+</p>
 
-* [Lucene.Net API documentation](api/Lucene.Net.html)
-* [Lucene.NET homepage](https://lucenenet.apache.org/)
-* [Lucene.NET GitHub](https://github.com/apache/lucenenet)
-* [Lucene.NET Official Git repo](https://git-wip-us.apache.org/repos/asf/lucenenet.git)
-* [Lucene.NET Issue tracker](https://issues.apache.org/jira/browse/LUCENENET)
+## Getting Started
+
+<p>The following section is intended as a "getting started" guide. It has three
+        audiences: first-time users looking to install Apache Lucene in their
+        application; developers looking to modify or base the applications they develop
+        on Lucene; and developers looking to become involved in and contribute to the
+        development of Lucene. The goal is to help you "get started". It does not go into great depth
+        on some of the conceptual or inner details of Lucene:</p>
+<ul>
+<li>
+<a href="demo/overview-summary.html#overview_description">Lucene demo, its usage, and sources</a>:
+        Tutorial and walk-through of the command-line Lucene demo.</li>
+<li>
+<a href="core/overview-summary.html#overview_description">Introduction to Lucene's APIs</a>:
+        High-level summary of the different Lucene packages. </li>
+<li>
+<a href="core/org/apache/lucene/analysis/package-summary.html#package_description">Analysis overview</a>:
+        Introduction to Lucene's analysis API.  See also the
+        <a href="core/org/apache/lucene/analysis/TokenStream.html">TokenStream consumer workflow</a>.</li>
+</ul>
+
+## Reference Documents
+
+<ul>
+<li>
+<a href="changes/Changes.html">Changes</a>: List of changes in this release.</li>
+<li>
+<a href="SYSTEM_REQUIREMENTS.html">System Requirements</a>: Minimum and supported Java versions.</li>
+<li>
+<a href="MIGRATE.html">Migration Guide</a>: What changed in Lucene 4; how to migrate code from Lucene 3.x.</li>
+<li>
+<a href="JRE_VERSION_MIGRATION.html">JRE Version Migration</a>: Information about upgrading between major JRE versions.</li>
+<li>
+<a href="core/org/apache/lucene/codecs/lucene46/package-summary.html#package_description">File Formats</a>: Guide to the supported index format used by Lucene.  This can be customized by using <a href="core/org/apache/lucene/codecs/package-summary.html#package_description">an alternate codec</a>.</li>
+<li>
+<a href="core/org/apache/lucene/search/package-summary.html#package_description">Search and Scoring in Lucene</a>: Introduction to how Lucene scores documents.</li>
+<li>
+<a href="core/org/apache/lucene/search/similarities/TFIDFSimilarity.html">Classic Scoring Formula</a>: Formula of Lucene's classic <a href="http://en.wikipedia.org/wiki/Vector_Space_Model">Vector Space</a> implementation. (look <a href="core/org/apache/lucene/search/similarities/package-summary.html#package_description">here</a> for other models)</li>
+<li>
+<a href="queryparser/org/apache/lucene/queryparser/classic/package-summary.html#package_description">Classic QueryParser Syntax</a>: Overview of the Classic QueryParser's syntax and features.</li>
+</ul>
+
+## API .NET Docs
+
+<ul>
+<li style="font-size:larger; margin-bottom:.5em;">
+<b><a href="api/index.html">core</a>: </b>Lucene core library</li>
+<li>
+<b><a href="analyzers-common/index.html">analyzers-common</a>: </b>Analyzers for indexing content in different languages and domains.</li>
+<li>
+<b><a href="analyzers-icu/index.html">analyzers-icu</a>: </b>Analysis integration with ICU (International Components for Unicode).</li>
+<li>
+<b><a href="analyzers-kuromoji/index.html">analyzers-kuromoji</a>: </b>Japanese Morphological Analyzer</li>
+<li>
+<b><a href="analyzers-morfologik/index.html">analyzers-morfologik</a>: </b>Analyzer for indexing Polish</li>
+<li>
+<b><a href="analyzers-phonetic/index.html">analyzers-phonetic</a>: </b>Analyzer for indexing phonetic signatures (for sounds-alike search)</li>
+<li>
+<b><a href="analyzers-smartcn/index.html">analyzers-smartcn</a>: </b>Analyzer for indexing Chinese</li>
+<li>
+<b><a href="analyzers-stempel/index.html">analyzers-stempel</a>: </b>Analyzer for indexing Polish</li>
+<li>
+<b><a href="analyzers-uima/index.html">analyzers-uima</a>: </b>Analysis integration with Apache UIMA</li>
+<li>
+<b><a href="benchmark/index.html">benchmark</a>: </b>System for benchmarking Lucene</li>
+<li>
+<b><a href="classification/index.html">classification</a>: </b>Classification module for Lucene</li>
+<li>
+<b><a href="codecs/index.html">codecs</a>: </b>Lucene codecs and postings formats.</li>
+<li>
+<b><a href="demo/index.html">demo</a>: </b>Simple example code</li>
+<li>
+<b><a href="expressions/index.html">expressions</a>: </b>Dynamically computed values to sort/facet/search on based on a pluggable grammar.</li>
+<li>
+<b><a href="facet/index.html">facet</a>: </b>Faceted indexing and search capabilities</li>
+<li>
+<b><a href="grouping/index.html">grouping</a>: </b>Collectors for grouping search results.</li>
+<li>
+<b><a href="highlighter/index.html">highlighter</a>: </b>Highlights search keywords in results</li>
+<li>
+<b><a href="join/index.html">join</a>: </b>Index-time and Query-time joins for normalized content</li>
+<li>
+<b><a href="memory/index.html">memory</a>: </b>Single-document in-memory index implementation</li>
+<li>
+<b><a href="misc/index.html">misc</a>: </b>Index tools and other miscellaneous code</li>
+<li>
+<b><a href="queries/index.html">queries</a>: </b>Filters and Queries that add to core Lucene</li>
+<li>
+<b><a href="queryparser/index.html">queryparser</a>: </b>Query parsers and parsing framework</li>
+<li>
+<b><a href="replicator/index.html">replicator</a>: </b>Files replication utility</li>
+<li>
+<b><a href="sandbox/index.html">sandbox</a>: </b>Various third party contributions and new ideas</li>
+<li>
+<b><a href="spatial/index.html">spatial</a>: </b>Geospatial search</li>
+<li>
+<b><a href="suggest/index.html">suggest</a>: </b>Auto-suggest and Spellchecking support</li>
+<li>
+<b><a href="test-framework/index.html">test-framework</a>: </b>Framework for testing Lucene-based applications</li>
+</ul>

apidocs/toc.yml

@@ -1,6 +1,9 @@
+- name: Lucene.NET Docs
+  href: docs/
+  homepage: docs/index.md
 - name: Lucene.NET API Docs
   href: api/
-  homepage: api/Lucene.Net.yml
+  homepage: api/index.md
 - name: Lucene.NET homepage
   href: https://lucenenet.apache.org/
 - name: Lucene.NET GitHub