Merging arrays transformations back into 'develop' branch (#1236)

* Fix up tests

* Conversion to files array. WIP. Core SARIF component build complete except for SarifLogger tail.

* Add fileIndex property to file object (#1186)

* Fix up tests

* PR feedback to improve schema comment

* Logical locations notes (#1185) (#1187)

* Respond to a small # of PR comments related to recent logical locations change.

* Fix visibility on helper

* Fix up v1 transformation with keys that collide

* Preserve decorated name data

* Rebaseline test for decorated name propagation

* Respond to PR feedback. Update tests to close test holes.

* Rebaseline updated test

* Test key collision in annotated code locations.

* Update baseline

* Reduced files array build (#1191)

* Sarif and Sarif.Converters now building

* Files array (#1188)

* Add fileIndex property to file object (#1186)

* Fix up tests

* PR feedback to improve schema comment

* Logical locations notes (#1185) (#1187)

* Respond to a small # of PR comments related to recent logical locations change.

* Fix visibility on helper

* Fix up v1 transformation with keys that collide

* Preserve decorated name data

* Rebaseline test for decorated name propagation

* Respond to PR feedback. Update tests to close test holes.

* Rebaseline updated test

* Test key collision in annotated code locations.

* Update baseline

* DRY out converters to isolate shared code.

* Restore essential change in schema that converts files dictionary to an array.

* Simplify ShouldSerialize logic

* Remove unused namespaces

* Respond to PR feedback.

* Respond to PR feedback

* End-to-end build works. Now we can author core transformation and fix tests. (#1192)

* Fix up merge from 'develop' branch.

* Update supporting test code for processing checked in files. (#1202)

* Update supporting test code for processing checked in files.

* Update nested files test to contain single file.

* Files array basic transform (#1205)

* Update supporting test code for processing checked in files.

* Update nested files test to contain single file.

* WIP. Furhter progress

* Fix up samples build

* Fix up merge from basic transform branch

* Mime type validation (#1206)

* Fix up merge from basic transform branch

* Fix up mime test

* Start work on v1 <-> v2 transformation (#1208)

* Restore TransformCommand and first (unaffected) unit test

* Restore "minimal prerelease v2 to current v2" test.

* estore "minimal v1 to current v2" test.

* Restore remaining TransformCommand unit tests.

* Uncomment v2 => v1 tests to observe failures.

* Uncomment 'transform' command.

* Restore MakeUrisAbsoluteVisitor tests (#1210)

This change updates the visitor that expands URIs in the presence of `originalUriBaseIds`. Turns out there was technical debt here, because our tests provided `originalUriBaseIds` equivalents in the property bag (because we had no official SARIF slot for them). I did not notice this gap when we made the schema change to add `originalUriBaseIds`.

* Get v2 -> v1 transform working with files array (#1211)

Test failure count is down to 32; will be 28 when you merge your fix.

There is not -- and never was -- a test case for fileLocations that use uriBaseId (never was one). I know for a fact that there is no code to support that case. You’ll see a comment to that effect in the code. I will take care of that next. Then I will move on to v1 -> v2 transform.

As part of this change, the `SarifCurrentToVersionOneVisitorTests` are now based on the `RunTest` helper method from the base class `FileDiffingTests`.

* Convert debug assert to exception to make test execution more deterministic (#1214)

* Update insert optional data tests and update indices visitor. (#1212)

* Update insert optional data tests and update indices visitor.

* Delete speculatively needed file

* Remove erroneous override of base visit method.

* Rework summary comment on DeletesOutputsDirectoryOnClassInitializationFixture.

* Update clang analyzer name. Flow converter log verification through JToken comparison. (#1213)

* The multiool, core sarif, and validation test binaries now all pass (#1215)

* The multiool, core sarif, and validation test binaries now all pass completely.

* Remove unwanted assert that may fire during unit testing.

* Merge from files-array

* PR feedback.

* PR feedback tweaks

* Accept PR feedback from previous change. (#1216)

Use LINQ IEnuemrable.Except in the unit test, which improves readability without compromising efficiency (because Except uses a Set to do its work in O(N) time).

* Fix Sarif.Driver and Sarif.Functional tests. (#1217)

* Fix Sarif.Driver and Sarif.Functional tests.

* Restore test file

* Fix Semmle tests and Fortify converter: all tests now pass. (#1218)

* Sarif converters fixups (#1219)

* Fix semmle tests and fority.

* Final test fixups

* Invoke appveyor for files-array branch.: (#1220)

* Update SarifVersionOneToCurrentVisitor for run.files array (#1221)

* Uncomment v1 -> v2 tests; 3/14 fail.

* Move test data to locations expected by FileDiffingTests.

* Fix up some IDE C#7 code cleanups.

* Use FileDiffingTests helper.

* Fix bug in FileDiffingTests that caused a test failure.

* Remove default-valued argument from a call to RunTest.

* Create basic files array

Does not yet have fileIndex, parentIndex, or response file handling.

* Revert incorrect change in FileDiffingTests.

* Fix one unit test with spot fix to "expected" file.

* Fix up some C#7 IDE warnings

* Force update in FileDiffing tests to avoid deserialization errors from out of date "expected" files.

* Fix missing "modified" flag sets in PreRelCompatTransformer.

* Populate fileIndex in run.files array.

* Fix unit test by fixing fileLocation creation.

* Restore response file handling.

* Populate fileIndex on fileLocations as appropriate.

* Fix last test failure by reworking response file handling.

* Feedback: Introduce transformer helper PopulatePropertyIfAbsent.

* Feedback: Tighten platform-independent string compare.

Also:
- Reformat unit test lines.

* Feedbakc: Revert FileDiffingTest change; downgrade affected test files to provoke transform

* Basic rules transformation (except for v1 <-> v2 conversion) (#1223)

* Basic rules transformation (except for v1 <-> v2 conversion)

* Respond to very excellent PR feedback.

* PR feedback

* Add files array tests for nested files and uriBaseIds (#1226)

* Add failing v1 -> v2 nested files test

* Fix existing mis-handling of analysisTarget and resultFile.

* Get nested files test case working.

* Add failing v1 => v2 uriBaseId test.

* Populate v2 uriBaseId.

* Fix up expected v2 fileLocation.properties: test passes.

* Enhance uriBaseIds test case.

* Implement v2->v1 conversion for rules dictionary (#1228)

* Notification rule index (#1229)

* Add notification.ruleIndex and increase flatten messages testing

* Notification message tests + add notification.ruleIndex to schema

* Notification rule index (#1230)

* Add notification.ruleIndex and increase flatten messages testing

* Notification message tests + add notification.ruleIndex to schema

* Missed feedback from previous PR (#1231)

* Implement v1->v2 conversion for rules dictionary (#1232)

* Partial implementation

* Get last test working.

* Somebody missed checking in a generated file.

* Schema changes from TC #30 (#1233)

* Add source language, fix rank default.

* Adjust rank minimum to accommoodate default.

* Fix broken test.

* Remove unnecessary None items from project file.

* PR feedback

* Files array results matching (#1234)

* WIP preliminary results matching

* Restore results matching for files array

* Add back autogenerated (unused) namespace directive

appveyor.yml

@@ -7,6 +7,7 @@ branches:
   only:
     - master
     - develop
+    - files-array
     - sarif-v1
 
 platform: Any CPU

scripts/Run-Tests.ps1

@@ -41,6 +41,7 @@ if ($AppVeyor) {
 
 $TestRunnerRootPath = "$NuGetPackageRoot\xunit.runner.console\2.3.1\tools\"
 
+$failedTestProjects = @()
 foreach ($project in $Projects.Tests) {
     foreach ($framework in $Frameworks.Application) {
         Write-Information "Running tests in ${project}: $framework..."
@@ -52,9 +53,12 @@ foreach ($project in $Projects.Tests) {
             & ${TestRunnerRootPath}net452\xunit.console.exe $dll $ReporterOption
         }
         if ($LASTEXITCODE -ne 0) {
-            Pop-Location
-            Exit-WithFailureMessage $ScriptName "${project}: tests failed."
+            $failedTestProjects += "${project}: $framework"
         }
         Pop-Location
     }
 }
+
+if ($failedTestProjects) {
+    Exit-WithFailureMessage $ScriptName "Tests failed in these projects `n    $($failedTestProjects -join "`n    ")"
+}

src/Samples/SarifDeferredSample/Program.cs

@@ -72,9 +72,8 @@ private static void Main(string[] args)
                 if (run.Files != null)
                 {
                     // Fastest: Enumerate
-                    foreach (var item in run.Files)
+                    foreach (FileData file in run.Files)
                     {
-                        FileData file = item.Value;
                         uriLengthTotal += file?.FileLocation?.Uri?.OriginalString?.Length ?? 0;
                         fileCount++;
                     }

src/Sarif.Converters.UnitTests/AndroidStudioConverterTests.cs

@@ -42,7 +42,7 @@ public void AndroidStudioConverter_Convert_NullOutput()
             Assert.Throws<ArgumentNullException>(() => _converter.Convert(new MemoryStream(), null, OptionallyEmittedData.None));
         }
 
-        private string emptyResult =
+        private readonly string emptyResult =
 @"{
   ""$schema"": """ + SarifUtilities.SarifSchemaUri + @""",
   ""version"": """ + SarifUtilities.SemanticVersion + @""",
@@ -61,23 +61,23 @@ public void AndroidStudioConverter_Convert_NoResults()
         {
             string androidStudioLog = @"<?xml version=""1.0"" encoding=""UTF-8""?><problems></problems>";
             string actualJson = Utilities.GetConverterJson(_converter, androidStudioLog);
-            actualJson.Should().BeCrossPlatformEquivalent(emptyResult);
+            actualJson.Should().BeCrossPlatformEquivalent<SarifLog>(emptyResult);
         }
 
         [Fact]
         public void AndroidStudioConverter_Convert_EmptyResult()
         {
             string androidStudioLog = @"<?xml version=""1.0"" encoding=""UTF-8""?><problems><problem></problem></problems>";
             string actualJson = Utilities.GetConverterJson(_converter, androidStudioLog);
-            actualJson.Should().BeCrossPlatformEquivalent(emptyResult);
+            actualJson.Should().BeCrossPlatformEquivalent<SarifLog>(emptyResult);
         }
 
         [Fact]
         public void AndroidStudioConverter_Convert_EmptyResultSingleTag()
         {
             string androidStudioLog = @"<?xml version=""1.0"" encoding=""UTF-8""?><problems><problem /></problems>";
             string actualJson = Utilities.GetConverterJson(_converter, androidStudioLog);
-            actualJson.Should().BeCrossPlatformEquivalent(emptyResult);
+            actualJson.Should().BeCrossPlatformEquivalent<SarifLog>(emptyResult);
         }
 
         [Fact]

src/Sarif.Converters.UnitTests/ClangAnalyzerConverterTests.cs

@@ -38,7 +38,7 @@ public void ClangAnalyzerConverter_Convert_NullLogTest()
   ""runs"": [
     {
       ""tool"": {
-        ""name"": ""Clang""
+        ""name"": ""Clang Analyzer""
       },
       ""columnKind"": ""utf16CodeUnits"",
       ""results"": []

src/Sarif.Converters.UnitTests/ConverterTestsBase.cs

@@ -4,16 +4,17 @@
 using FluentAssertions;
 
 using Microsoft.CodeAnalysis.Sarif.Converters;
-using Microsoft.CodeAnalysis.Sarif.Readers;
 using Microsoft.CodeAnalysis.Sarif.TestUtilities;
+using Microsoft.CodeAnalysis.Sarif.Writers;
 using Newtonsoft.Json;
 
 namespace Microsoft.CodeAnalysis.Sarif
 {
     public class ConverterTestsBase<T> where T : ToolFileConverterBase, new()
     {
-        public SarifLog RunTestCase(string inputData, string expectedResult, bool prettyPrint = true)
+        public SarifLog RunTestCase(string inputData, string expectedResult, bool prettyPrint = true, bool forceV2Transform = false)
         {
+            PrereleaseCompatibilityTransformer.UpdateToCurrentVersion(expectedResult, forceUpdate: forceV2Transform, formatting: Formatting.Indented, out expectedResult);
             var converter = new T();
 
             // First retrieve converter JSON. This code will raise appropriate exceptions 
@@ -38,7 +39,7 @@ public SarifLog RunTestCase(string inputData, string expectedResult, bool pretty
             // are fragile. It would be better for our testing to have a dedicated set of data-driven
             // tests that flag changes and for the unit-tests to work exclusively against the 
             // object model.
-            actualJson.Should().BeCrossPlatformEquivalent(expectedResult);
+            actualJson.Should().BeCrossPlatformEquivalent<SarifLog>(expectedResult);
 
             return log;
         }

src/Sarif.Converters.UnitTests/FortifyFprConverterTests.cs

@@ -1,85 +1,46 @@
 // Copyright (c) Microsoft. All rights reserved.
 // Licensed under the MIT license. See LICENSE file in the project root for full license information.
 
-using System;
-using System.IO;
-using System.Text;
-using Microsoft.CodeAnalysis.Sarif.Writers;
+using Microsoft.CodeAnalysis.Sarif.TestUtilities;
 using Xunit;
 using Xunit.Abstractions;
 
 namespace Microsoft.CodeAnalysis.Sarif.Converters
 {
-    public class FortifyFprConverterTests : FileDiffingTests
+    public class FortifyFprConverterTests : FileDiffingTests, IClassFixture<FortifyFprConverterTests.FortifyFprConverterTestsFixture>
     {
+        public class FortifyFprConverterTestsFixture : DeletesOutputsDirectoryOnClassInitializationFixture { }
+
         protected override string TestLogResourceNameRoot => "Microsoft.CodeAnalysis.Sarif.Converters.UnitTests.TestData." + TypeUnderTest;
 
         public FortifyFprConverterTests(ITestOutputHelper outputHelper) : base(outputHelper) { }
 
-        private void VerifyFortifyFprConversionFromResource(string fprResourceName, string expectedSarifResourceName = null)
-        {
-            expectedSarifResourceName = expectedSarifResourceName ?? fprResourceName;
-            fprResourceName += ".fpr";
-            expectedSarifResourceName += ".sarif";
+        protected override bool RebaselineExpectedResults => false;
 
-            byte[] fprData = GetResourceBytes(fprResourceName);
-            string expectedSarifText = GetResourceText(expectedSarifResourceName);
-            PrereleaseCompatibilityTransformer.UpdateToCurrentVersion(expectedSarifText, forceUpdate: false, formatting: Newtonsoft.Json.Formatting.Indented, out expectedSarifText);
+        protected override string ConstructTestOutputFromInputResource(string inputResource)
+        {
+            byte[] fprData = GetResourceBytes(inputResource);
 
             var converter = new FortifyFprConverter();
-            string actualSarifText = Utilities.GetConverterJson(converter, fprData);
-
-            var sb = new StringBuilder();
-
-            string expectedFilePath = null;
-            string actualFilePath = null;
-
-            if (!Utilities.RunningInAppVeyor)
-            {
-                expectedFilePath = GetOutputFilePath("expected", expectedSarifResourceName);
-                actualFilePath = GetOutputFilePath("actual", expectedSarifResourceName);
-            }
-
-            if (!AreEquivalentSarifLogs<SarifLog>(actualSarifText, expectedSarifText))
-            {
-                string errorMessage = string.Format(@"Conversion from Fortify FPR to SARIF produced unexpected diffs for test: '{0}'.", fprResourceName);
-                sb.AppendLine(errorMessage);
-
-                if (!Utilities.RunningInAppVeyor)
-                {
-                    string expectedRootDirectory = Path.GetDirectoryName(expectedFilePath);
-                    string actualRootDirectory = Path.GetDirectoryName(actualFilePath);
-
-                    Directory.CreateDirectory(expectedRootDirectory);
-                    Directory.CreateDirectory(actualRootDirectory);
-
-                    File.WriteAllText(expectedFilePath, expectedSarifText);
-                    File.WriteAllText(actualFilePath, actualSarifText);
-
-                    sb.AppendLine("To compare all difference for this test suite:");
-                    sb.AppendLine(GenerateDiffCommand("FortifyFprConverterTests", Path.GetDirectoryName(expectedFilePath), Path.GetDirectoryName(actualFilePath)) + Environment.NewLine);
-                }
-
-                ValidateResults(sb.ToString());
-            }
+            return Utilities.GetConverterJson(converter, fprData);
         }
 
         [Fact]
         public void FortifyFprConverter_Convert_OneResultBasic()
         {
-            VerifyFortifyFprConversionFromResource("OneResultBasic");
+            RunTest("OneResultBasic.fpr");
         }
 
         [Fact]
         public void FortifyFprConverter_Convert_OneResultWithTwoTraces()
         {
-            VerifyFortifyFprConversionFromResource("OneResultWithTwoTraces");
+            RunTest("OneResultWithTwoTraces.fpr");
         }
 
         [Fact]
         public void FortifyFprConverter_Convert_TwoResultsWithNodeRefs()
         {
-            VerifyFortifyFprConversionFromResource("TwoResultsWithNodeRefs");
+            RunTest("TwoResultsWithNodeRefs.fpr");
         }
     }
 }

src/Sarif.Converters.UnitTests/FxCopConverterTests.cs

@@ -103,7 +103,7 @@ public void FxCopLogReader_Context_RefineStackTrace()
             var context = TestHelper.CreateProjectContext();
 
             context.RefineStackTrace(@"trace\n trace");
-            context.StackTrace.Should().BeCrossPlatformEquivalent(@"trace\n trace");
+            context.StackTrace.Should().BeCrossPlatformEquivalentStrings(@"trace\n trace");
         }
 
         [Fact]

src/Sarif.Converters.UnitTests/PylintConverterTests.cs

@@ -96,7 +96,7 @@ public void PylintConverter_Convert_WhenInputIsValid_Passes()
 
             var mockWriter = new Mock<IResultLogWriter>();
             mockWriter.Setup(writer => writer.Initialize(It.IsAny<Run>()));
-            mockWriter.Setup(writer => writer.WriteFiles(It.IsAny<IDictionary<string, FileData>>()));
+            mockWriter.Setup(writer => writer.WriteFiles(It.IsAny<IList<FileData>>()));
             mockWriter.Setup(writer => writer.OpenResults());
             mockWriter.Setup(writer => writer.CloseResults());
             mockWriter.Setup(writer => writer.WriteResults(It.IsAny<List<Result>>()));
@@ -106,7 +106,7 @@ public void PylintConverter_Convert_WhenInputIsValid_Passes()
             converter.Convert(stream, mockWriter.Object, OptionallyEmittedData.None);
 
             mockWriter.Verify(writer => writer.Initialize(It.IsAny<Run>()), Times.Once);
-            mockWriter.Verify(writer => writer.WriteFiles(It.IsAny<IDictionary<string, FileData>>()), Times.Once);
+            mockWriter.Verify(writer => writer.WriteFiles(It.IsAny<IList<FileData>>()), Times.Once);
             mockWriter.Verify(writer => writer.OpenResults(), Times.Once);
             mockWriter.Verify(writer => writer.CloseResults(), Times.Once);
             mockWriter.Verify(writer => writer.WriteResults(It.IsAny<List<Result>>()), Times.Once);

src/Sarif.Converters.UnitTests/Sarif.Converters.UnitTests.csproj

@@ -14,6 +14,12 @@
   <Import Project="$([MSBuild]::GetDirectoryNameOfFileAbove($(MSBuildThisFileDirectory), build.props))\build.props" />
 
   <ItemGroup>
+    <None Remove="TestData\FortifyFprConverter\ExpectedOutputs\OneResultBasic.sarif" />
+    <None Remove="TestData\FortifyFprConverter\ExpectedOutputs\OneResultWithTwoTraces.sarif" />
+    <None Remove="TestData\FortifyFprConverter\ExpectedOutputs\TwoResultsWithNodeRefs.sarif" />
+    <None Remove="TestData\FortifyFprConverter\Inputs\OneResultBasic.fpr" />
+    <None Remove="TestData\FortifyFprConverter\Inputs\OneResultWithTwoTraces.fpr" />
+    <None Remove="TestData\FortifyFprConverter\Inputs\TwoResultsWithNodeRefs.fpr" />
     <None Remove="TestData\FortifyFprConverter\OneResultBasic.fpr" />
     <None Remove="TestData\FortifyFprConverter\OneResultBasic.sarif" />
     <None Remove="TestData\FortifyFprConverter\OneResultWithTwoTraces.fpr" />
@@ -33,12 +39,12 @@
   </ItemGroup>
 
   <ItemGroup>
-    <EmbeddedResource Include="TestData\FortifyFprConverter\OneResultBasic.fpr" />
-    <EmbeddedResource Include="TestData\FortifyFprConverter\OneResultBasic.sarif" />
-    <EmbeddedResource Include="TestData\FortifyFprConverter\OneResultWithTwoTraces.fpr" />
-    <EmbeddedResource Include="TestData\FortifyFprConverter\OneResultWithTwoTraces.sarif" />
-    <EmbeddedResource Include="TestData\FortifyFprConverter\TwoResultsWithNodeRefs.fpr" />
-    <EmbeddedResource Include="TestData\FortifyFprConverter\TwoResultsWithNodeRefs.sarif" />
+    <EmbeddedResource Include="TestData\FortifyFprConverter\ExpectedOutputs\OneResultBasic.sarif" />
+    <EmbeddedResource Include="TestData\FortifyFprConverter\ExpectedOutputs\OneResultWithTwoTraces.sarif" />
+    <EmbeddedResource Include="TestData\FortifyFprConverter\ExpectedOutputs\TwoResultsWithNodeRefs.sarif" />
+    <EmbeddedResource Include="TestData\FortifyFprConverter\Inputs\OneResultBasic.fpr" />
+    <EmbeddedResource Include="TestData\FortifyFprConverter\Inputs\OneResultWithTwoTraces.fpr" />
+    <EmbeddedResource Include="TestData\FortifyFprConverter\Inputs\TwoResultsWithNodeRefs.fpr" />
   </ItemGroup>
 
   <ItemGroup>

src/Sarif.Converters.UnitTests/SemmleQLConverterTests.cs

@@ -55,7 +55,7 @@ public void SemmleQLConverter_SimpleCsv()
     }
   ]
 }";
-            RunTestCase(semmleCsvInput, expected, prettyPrint: true);
+            RunTestCase(semmleCsvInput, expected, prettyPrint: true, forceV2Transform: true);
         }
 
         [Fact]
@@ -153,7 +153,7 @@ public void SemmleQLConvert_EmbeddedLocations()
   ]
 }";
 
-            RunTestCase(semmleCsvInput, expected, prettyPrint: true);
+            RunTestCase(semmleCsvInput, expected, prettyPrint: true, forceV2Transform: true);
         }
 
         private class SemmleCsvRecord

src/Sarif.Converters.UnitTests/TSLintConverterTests.cs

@@ -173,7 +173,7 @@ public void TSLintConverter_Convert_WhenInputIsValid_Passes()
 
             var mockWriter = new Mock<IResultLogWriter>();
             mockWriter.Setup(writer => writer.Initialize(It.IsAny<Run>()));
-            mockWriter.Setup(writer => writer.WriteFiles(It.IsAny<IDictionary<string, FileData>>()));
+            mockWriter.Setup(writer => writer.WriteFiles(It.IsAny<IList<FileData>>()));
             mockWriter.Setup(writer => writer.OpenResults());
             mockWriter.Setup(writer => writer.CloseResults());
             mockWriter.Setup(writer => writer.WriteResults(It.IsAny<List<Result>>()));
@@ -183,7 +183,7 @@ public void TSLintConverter_Convert_WhenInputIsValid_Passes()
             converter.Convert(stream, mockWriter.Object, OptionallyEmittedData.None);
 
             mockWriter.Verify(writer => writer.Initialize(It.IsAny<Run>()), Times.Once);
-            mockWriter.Verify(writer => writer.WriteFiles(It.IsAny<IDictionary<string, FileData>>()), Times.Once);
+            mockWriter.Verify(writer => writer.WriteFiles(It.IsAny<IList<FileData>>()), Times.Once);
             mockWriter.Verify(writer => writer.OpenResults(), Times.Once);
             mockWriter.Verify(writer => writer.CloseResults(), Times.Once);
             mockWriter.Verify(writer => writer.WriteResults(It.IsAny<List<Result>>()), Times.Once);