Automatically resolve includes in package readme file

Allows a package readme to simply reference other files (via relative path and optional
fragment specifier), like:

```
<!-- include ../../readme.md#content -->
<!-- include ../../docs/footer.md -->
```

Where the main project readme could contain a bunch of project-level info,
contrib, badges, etc., which you can skip by marking a specific area as t
he #content anchor:

```
# My Project
// bunch of badges
// building/contributing, etc.

<~-- #content -->
# Usage
// now the real stuff we want in a package readme
```

The `footer.md` could contain project sponsors and the like, and also be
reused across multiple packages.

The includes must be relative (to the including file) file paths, and the fragmnet
specifier is optional. If the fragment specifier does not have an "end" anchor in
the included file, the file is included from the anchor definition until EOF.

The includes are only resolved when the actual .nupkg is created, and via a temp
file, to avoid affecting the build's incrementality (which doesn't apply to the .nupkg
itself either).

Fixes #210

readme.md

@@ -46,6 +46,42 @@ in your package. This gives you ultimate control without having to understand an
 
 All [inference rules are laid out in a single .targets](src/NuGetizer.Tasks/NuGetizer.Inference.targets) file that's easy to inspect them to learn more, and the file is not imported at all when `EnablePackInference=false`.
 
+## Package Readme
+
+Since the introduction of [package readme on nuget.org](https://docs.microsoft.com/en-us/nuget/nuget-org/package-readme-on-nuget-org), 
+more and more packages are leveraging this feature to make a package more discoverable and user friendly. One common 
+need that arises is reusing existing documentation content that exists elsewhere in the project repository, such as 
+on the root readme for the project (which typically contains additional information beyond user facing documentation, 
+such as how to clone, build and contribute to the repository). In order to maximize reuse for these documentation files, 
+NuGetizer supports includes in the package readme, such as:
+
+```
+This is the package readme.
+<!-- include ../../../readme.md#usage -->
+
+<!-- include ../../../footer.md -->
+```
+
+This readme includes a specific section of the repository root readme (via `#usage`), which is defined as follows:
+
+```
+# Project Foo
+This is a general section on cloning, contributing, CI badges, etc.
+
+<!-- #usage -->
+# Usage
+Here we explain our awesome API...
+<!-- #usage -->
+...
+```
+
+By defining both starting and closing `#usage` markup, the package readme can include a specific section. 
+The footer, by contrast, is included wholesale. 
+
+When the `.nupkg` is created, these includes are resolved automatically so you keep content duplication to a 
+minimum. Nested includes are also supported (i.e. `footer.md` might in turn include a `sponsors.md` file or 
+a fragment of it).
+
 ## dotnet-nugetize
 
 Carefully tweaking your packages until they look exactly the way you want them should not be a tedious and slow process. Even requiring your project to be built between changes can be costly and reduce the speed at which you can iterate on the packaging aspects of the project. Also, generating the final `.nupkg`, opening it in a tool and inspecting its content, is also not ideal for rapid iteration.

src/NuGetizer.Tasks/CreatePackage.cs

@@ -197,6 +197,17 @@ void GeneratePackage(Stream output = null)
 
             // We don't use PopulateFiles because that performs search expansion, base path 
             // extraction and the like, which messes with our determined files to include.
+
+            if (!string.IsNullOrEmpty(manifest.Metadata.Readme) &&
+                manifest.Files.FirstOrDefault(f => Path.GetFileName(f.Target) == manifest.Metadata.Readme) is ManifestFile readmeFile &&
+                File.Exists(readmeFile.Source))
+            {
+                // replace readme with includes replaced.
+                var temp = Path.GetTempFileName();
+                File.WriteAllText(temp, IncludesResolver.Process(readmeFile.Source));
+                readmeFile.Source = temp;
+            }
+
             builder.Files.AddRange(manifest.Files.Select(file =>
                 new PhysicalPackageFile { SourcePath = file.Source, TargetPath = file.Target }));
 

src/NuGetizer.Tasks/IncludesResolver.cs

@@ -0,0 +1,72 @@
+using System;
+using System.Collections.Generic;
+using System.IO;
+using System.Text.RegularExpressions;
+
+namespace NuGetizer;
+
+public class IncludesResolver
+{
+    static readonly Regex IncludeRegex = new Regex(@"<!--\s?include\s(.*?)\s?-->", RegexOptions.Compiled);
+
+    public static string Process(string filePath)
+    {
+        var content = File.ReadAllText(filePath).Trim();
+        // Allow self-excluding files for processing. Could be useful if the file itself  
+        // documents the include/exclude mechanism, for example.
+        if (content.StartsWith("<!-- exclude -->") || content.EndsWith("<!-- exclude -->"))
+            return content;
+
+        var replacements = new Dictionary<Regex, string>();
+
+        foreach (Match match in IncludeRegex.Matches(content))
+        {
+            var includedPath = match.Groups[1].Value.Trim();
+            string fragment = default;
+            if (includedPath.Contains("#"))
+            {
+                fragment = "#" + includedPath.Split('#')[1];
+                includedPath = includedPath.Split('#')[0];
+            }
+
+            var includedFullPath = Path.Combine(Path.GetDirectoryName(filePath), includedPath).Replace(Path.DirectorySeparatorChar, Path.AltDirectorySeparatorChar);
+            if (File.Exists(includedFullPath))
+            {
+                // Resolve nested includes
+                var includedContent = Process(includedFullPath);
+                if (fragment != null)
+                {
+                    var anchor = $"<!-- {fragment} -->";
+                    var start = includedContent.IndexOf(anchor);
+                    if (start == -1)
+                        // Warn/error?
+                        continue;
+
+                    includedContent = includedContent.Substring(start);
+                    var end = includedContent.IndexOf(anchor, anchor.Length);
+                    if (end != -1)
+                        includedContent = includedContent.Substring(0, end + anchor.Length);
+                }
+
+                // see if we already have a section we previously replaced
+                var existingRegex = new Regex(@$"<!--\s?include {includedPath}{fragment}\s?-->[\s\S]*<!-- {includedPath}{fragment} -->");
+                var replacement = $"<!-- include {includedPath}{fragment} -->{Environment.NewLine}{includedContent}{Environment.NewLine}<!-- {includedPath}{fragment} -->";
+                if (existingRegex.IsMatch(content))
+                    replacements[existingRegex] = replacement;
+                else
+                    replacements[new Regex(@$"<!--\s?include {includedPath}{fragment}\s?-->")] = replacement;
+            }
+        }
+
+        if (replacements.Count > 0)
+        {
+            var updated = content;
+            foreach (var replacement in replacements)
+                updated = replacement.Key.Replace(updated, replacement.Value);
+
+            return updated.Trim();
+        }
+
+        return content;
+    }
+}

src/NuGetizer.Tasks/NuGetizer.Tasks.csproj

@@ -5,7 +5,6 @@
     <CopyLocalLockFileAssemblies>true</CopyLocalLockFileAssemblies>
     <RootNamespace>NuGetizer</RootNamespace>
     <AppendTargetFrameworkToOutputPath>false</AppendTargetFrameworkToOutputPath>
-    <PackageReadmeFile>readme.md</PackageReadmeFile>
   </PropertyGroup>
 
   <ItemGroup>

src/NuGetizer.Tests/Content/Common/footer.md

@@ -0,0 +1,2 @@
+the-footer
+<!-- include ../sections.md#copyright -->

src/NuGetizer.Tests/Content/Common/header.md

@@ -0,0 +1 @@
+the-header

src/NuGetizer.Tests/Content/readme.md

@@ -0,0 +1,7 @@
+<!-- include Common/header.md -->
+
+<!-- include sections.md#first -->
+
+<!-- include sections.md#third -->
+
+<!-- include Common/footer.md -->

src/NuGetizer.Tests/Content/sections.md

@@ -0,0 +1,15 @@
+<!-- #first -->
+section#1
+<!-- #first -->
+
+<!-- #second -->
+section#2
+<!-- #second -->
+
+<!-- #third -->
+section#3
+<!-- #third -->
+
+<!-- #copyright -->
+@kzu
+<!-- #copyright -->

src/NuGetizer.Tests/CreatePackageTests.cs

@@ -2,6 +2,7 @@
 using System.Collections.Generic;
 using System.Diagnostics;
 using System.IO;
+using System.IO.Compression;
 using System.Linq;
 using Microsoft.Build.Framework;
 using Microsoft.Build.Utilities;
@@ -579,6 +580,41 @@ public void when_creating_package_with_file_then_contains_file()
             Assert.Contains(manifest.Files, file => file.Target == "readme.txt");
         }
 
+        [Fact]
+        public void when_creating_package_with_readme_then_resolves_includes()
+        {
+            var readme = Path.GetTempFileName();
+            var footer = Path.GetTempFileName();
+            File.WriteAllText(footer, "footer");
+            File.WriteAllText(readme, $"<!-- include {Path.GetFileName(footer)} -->");
+
+            task.Manifest.SetMetadata("Readme", "readme.md");
+            task.Contents = new[]
+            {
+                new TaskItem(readme, new Metadata
+                {
+                    { MetadataName.PackageId, task.Manifest.GetMetadata("Id") },
+                    { MetadataName.PackFolder, PackFolderKind.None },
+                    { MetadataName.PackagePath, "readme.md" }
+                }),
+            };
+
+            var tmp = Path.GetTempFileName();
+            using (var file = File.Open(tmp, FileMode.OpenOrCreate))
+                task.Execute(file);
+
+            var zip = ZipFile.Open(tmp, ZipArchiveMode.Update);
+
+            Assert.Contains(zip.Entries, entry => entry.Name == "readme.md");
+
+            var entry = zip.Entries.Single(e => e.Name == "readme.md");
+            var updated = Path.GetTempFileName();
+            entry.ExtractToFile(updated, true);
+            var content = File.ReadAllText(updated);
+
+            Assert.Contains("footer", content);
+        }
+
         [Fact]
         public void when_creating_package_with_content_file_then_adds_as_content_file()
         {

src/NuGetizer.Tests/IncludesResolverTests.cs

@@ -0,0 +1,20 @@
+using Xunit;
+
+namespace NuGetizer;
+
+public class IncludesResolverTests
+{
+    [Fact]
+    public void ResolveIncludes()
+    {
+        var content = IncludesResolver.Process("Content/readme.md");
+
+        Assert.Contains("the-header", content);
+        Assert.Contains("the-footer", content);
+
+        Assert.Contains("section#1", content);
+        Assert.DoesNotContain("section#2", content);
+        Assert.Contains("section#3", content);
+        Assert.Contains("@kzu", content);
+    }
+}

src/NuGetizer.Tests/NuGetizer.Tests.csproj

@@ -32,8 +32,10 @@
 
   <ItemGroup>
     <None Include="Scenarios\**\*.*" Exclude="Scenarios\**\obj\**\*.*;Scenarios\**\bin\**\*.*" CopyToOutputDirectory="PreserveNewest" />
+    <None Update="Content\**\*.*" CopyToOutputDirectory="PreserveNewest" />
     <Content Include="..\NuGetizer.Tasks\bin\$(Configuration)\*.*" CopyToOutputDirectory="PreserveNewest" Visible="false" />
-    <Compile Remove="ModuleInitializerAttribute.cs" Condition="'$(TargetFramework)' == 'net6.0'" />  </ItemGroup>  
+    <Compile Remove="ModuleInitializerAttribute.cs" Condition="'$(TargetFramework)' == 'net6.0'" />
+  </ItemGroup>  
   <ItemGroup>
     <EmbeddedResource Include="..\NuGetizer.Tasks\Resources.resx" Link="Resources.resx" />
   </ItemGroup>

src/NuGetizer.Tests/Scenarios/given_a_library_with_content/footer.md

@@ -0,0 +1 @@
+

src/NuGetizer.Tests/Scenarios/given_a_library_with_content/readme.md

@@ -1 +1,3 @@
-# Library
+# Library
+
+<!-- include footer.md -->